Internet2 Distributed Storage Infrastructure Update

1
Internet2 Distributed Storage Infrastructure
Update

• Micah Beck
• Univ. of Tennessee, Knoxville
• Bert Dempsey
• Univ. of North Carolina, Chapel Hill
• Web Caching Workshop BOF
• 31 March 1999, San Diego
• http//dsi.internet2.edu

2
I2-DSI Participants

• UT Knoxville / ICL
• Micah Beck
• Terry Moore
• Martin Swany
• Judi Talley
• UNC Chapel Hill /SILS
• Bert Dempsey
• Paul Jones (MetaLab)
• Debra Weiss
• Zhiwei Xiao

• GigaPOP and Campus Site Managers
• UCAID/Internet2
• Network Storage Working Group
• Ted HanssApplications Director
• NC Networking Initiative
• Digital Library Federation

3
A Word From Our Sponsors

• Cisco DNS redirection
• Ellemtel engineering effort
• IBM large storage DCE servers
• Novell storage directory servers
• Starburst reliable multicast software
• StorageTek large storage servers
• Sun design collaboration

4
Single Server Model

• High performance locally
• Unacceptable performance across commodity backbone

5
Relying on Wide Area QoS

• High performance access with reserved bandwidth
• Essential for real-time communication
• Technically difficult, expensive, not generally
  available

6
I2-DSI Model Replicated Services

• Clients access nearby server
• Everyone gets performance
• Local resources implement a global service

7
I2-DSI Service Architecture

• Replication
• Rsynch, Omnicast, AFS/DFSNovell Replication
• Resolution
• Sonar DNS, Distributed Director
• Delegation
• Cache prefetch

general users
8
Internet Content Channels

• A channel is a collection of content which can be
  transparently delivered to end user communities
  at a chosen (price,performance) point through a
  flexible, policy-based application of resources

9
Server Channel Examples

• Replicated Web Servers
• APIs Standard HTML, Active Server Pages
• Channels Web sites
• Streaming Media
• APIs MPEG-2, proprietary file formats
• Channels collections of multimedia presentations
• Executable content
• APIs Java byte code, Tcl, Perl
• Channels CGI programs

10
Current Server Deployment
11
IBM Web Cache Manager
RS/6000 AIX Server 1 GB RAM 72 GB Disk / 900 GB
Tape ADSM Heirarchical Storage Mgt.
12
I2-DSI Server Operations

• Project Operations Coordinator
• Judi Talley, University of Tennessee at Knoxville
• Site Managers
• Dave Vernon, Indiana University
• David Lassner, University of Hawaii at Manoa
• Mark Johnson, NC Networking Initiative
• Michael Rechtenbaugh, EROS Data Center

13
Infrastructure Expansion

• StorageTek
• 2 PC/Linux Servers
• 700GB disk, tape backup (hot!)
• Novell
• 6 PC/NetWare Servers
• 100GB disk
• Smaller institutions or departments

14
InfrastructureDevelopment Efforts

• Proximity Resolution
• Martin Swany SonarDNS
• Geoff Carpenter, German Goldszmidt Narwhal (IBM)
• Replication Mechanisms and Modeling
• Bert Dempsey students
• Debra Weiss Batch rsync multicast
• Zhiwei Xiao Network metrics and modeling
• Channel Representation and Server
• Leif Abrahamsson, Christophe Achouiantz, Patrik
  Johansson (Ellemtel)

15
I2-DSI Applications Workshop Chapel Hill, NC
March 4 5, 1999

• 10 applications
• Indiana Digital music and media library
• UNC-CH Instructional Management System
• San Jose State Art history images
• Vanderbilt zoomable medical images
• Viagenie Network docs database

• Columbia Earth sciences environment
• UNC-CH Virtual Laboratories
• Ohio Supercomputer Center High Volume Datasets
• CalTech Globally Interconnected Databases
• Univ. of Kent National Software Archive
• Red Hat pan-Linux source distribution

16
I2-DSI Applications Workshop Chapel Hill, NC
March 4 5, 1999

• 4 technologies
• Minnesota Scalable Video
• IBM Research Multicast, Filter and Store
• Moscow Ctr. for New Info. Tech. in Med. Ed.
  Semantic Text Analysis
• IBM Research Narwhal Resolution Proxy
• http//dsi.internet2.edu/apps99.html
• Special issue of the Journal of Network and
  Computer Applications (Academic Press)

17
Application Management Partner MetaLab.unc.edu

• The site formerly known as SunSITE.unc.edu
• Fearless Leader Paul Jones
• A cool, tall glass of sweet tea on a hot day.
• 2 M HTTP 1/3 M FTP file transfers daily
• Collections policy
• teaching, research, or public service
• use technology in innovative and unique ways
• non-commercial or not-for-profit

18
Application Strategy

• Chose initial applications
• Available or easily ported services
• Low update demands
• Port to an I2-DSI server
• Our development effort is limited
• App developers can have access to the servers
• Distribute to homogeneous core
• Derive service abstractions

19
The Need for Channel Representation Standards
locally interpreted files
replicated files
Origin Server
Replicated Server
Replicated Server
proxy
Web clients
Standard-based Web traffic
Replication of source files
20
Replication Performance and Scalability Issues

• Server placement
• Server resources
• Server description (metadata)
• Server Channel description (metadata)
• Object representation
• Characterization of replication mechanisms
• Channel-to-server mapping (subscription)

21
NetStore 99 Workshop

• Network Storage Technical Workshop
• Knoxville, TN, October 1999
• http//dsi.internet2.edu/netstore99
• Scope
• I2-DSI implementation
• I2-DSI applications
• Related networking projects
• Storage technology

22
Conclusions

• A server platform is in place
• Infrastructure development
• Service abstractions (search, computation)
• Publication and replication protocols
• Portable representation and API
• Heterogeneous servers
• Six months to show results from initial
  application development efforts