NeST: Network Storage Flexible Commodity Storage Appliances

1
NeST Network StorageFlexible Commodity Storage
Appliances
2
New worlds, new promises

• The grid has transformed dist comp
• Multiple LANS have merged
• Confederations of auto admin domains
• Possibilities for resource and data sharing abound

3
New worlds, new problems

• Interoperability between domains
• Management of resources more important
• Discovery of resources more difficult

4
Enter NeST

• Traditional file servers have not evolved
• NeST is a 2nd gen file server
• Flexible storage appliance for the grid
• Provides local and remote access to data
• Easy management of storage resources
• User level sw turns machines into storage apps
• Deployable and portable

5
Research meets production

• NeST exists at an exciting intersection
• Freedom to pursue academic curiosities
• Opportunities to discover real user concerns

6
Very exciting intersection
7
Very exciting intersection
8
Outline

• Introduction
• Interoperability
• Resource management
• Questions?

9
Multi-protocol support

• Interoperability between admin domains
• NeST currently speaks
• Grid FTP and FTP
• HTTP
• NFS (beta)
• Chirp
• Designed for integration of new protocols

10
Why not JBOS?

• Just a bunch of servers has limitations
• NeST advantages over JBOS
• Single config and admin interface
• Optimizations across multiple protocols
• e.g. cache aware scheduling
• Management and control of protocols
• e.g. prefer local users to remote users

11
Dangerous migrations
12
Wasted migrations

• Wasted migrations are more expensive
• Distant resources are a mixed blessing
• Great opportunities
• Possibility for great waste

13
Full migration support

• NeST makes storage schedulable and reservable
• NeST allows creation of dynamic users
• Too expensive to create all accounts everywhere
• Sharing is insecure and makes accounting
  difficult
• NeST has lots
• Reservations of time and space

14
Three way matching
Refers to NearestStorage.
Knows where NearestStorage is.
Job Ad
Machine Ad
Storage Ad
match
Machine
Job
NeST
15
Three way ClassAds
Type job TargetType machine Cmd
sim.exe Owner thain Requirements
(OpSyslinux) NearestStorage.HasCMSData
Type machine TargetType job OpSys
linux Requirements (Ownerthain) NearestSto
rage ( Name turkey) (TypeStorage)
Machine ClassAd
Job ClassAd
16
Conclusions

• User level sw turns machines into storage
  appliances
• Single admin interface across multiple protocols
• Participates in resource discovery
• Management of space and transfers

17
For more information

• http//www.cs.wisc.edu/condor/nest
• Version 0.9 now available (linux only, no NFS)
• Solaris and NFS coming soon
• Requests welcome
• People
• John Bent, Nick Leroy, Alain Roy, Joseph Stanley
• Miron Livny, Andrea Arpaci-Dusseau, Remzi
  Arpaci-Dusseau

18
Design structure
Physical network layer
Chirp
FTP
Grid ftp
NFS
HTTP
Common protocol layer
Storage Mgr
Physical storage layer
19
Lot management overhead
20
NeST v JBOS, solo
21
NeST v JBOS, team
22
NeST proportional sharing