The Oceanic Data Utility: OceanStore GlobalScale Persistent Storage

1
The Oceanic Data Utility(OceanStore)Global-Scal
e Persistent Storage

• John Kubiatowicz

2
Properties of the ODU

• Motivation
• Growing quantity of inconsistent data
• Widespread mobility of producers and consumers
• Simplicity subsume Web, email, filesystems,
  databases
• Nomadic Data Serverless, Homeless
• Sharing of information between anyone, anywhere
• Promiscuous caching of data enabled by tacit
  information (option 5/introspection)
• Efficient dissemination of information
  (multicast)
• Federation of many different companies, just like
  phone service or electric grid.
• Highly-available data always duplicated
• Higher-probability access
• Copies placed with low probability of correlated
  failure
• Shares technology with options 1,4,5, and 8

3
Technical Challenges

• Scalability performance easy to destroy
• vast number of entities billions
• cross-administrative domains
• Security is not optional data never cleartext
• Availability
• Should bootstrap redundancy available on global
  scale
• Economies of scale applied to achieving data
  reliability
• Maintainability
• Too large for human intervention in normal
  operation
• Naming How to maintain global namespace?
• Indexability
• Must enable efficient location/searching of data
• Consistency/Conflict resolution
• Multiple copies must have well-defined
  relationship

4
State of the Art?

• Remote file-system community NFS, AFS
• All have single points of failure
• Only caching at endpoints
• Mobile computing community Coda
• Small scale, fixed coherence mechanism
• Web caching community Inktomi, others?
• Specialized, incremental solutions
• Caching along client/server path, various
  bottlenecks
• Database Community Mariposa
• Still small scale, specialized types of queries
• Economic model not quite right but on right track
• Internet backup companies Medley
• Very limited in scope and flexibility
• PalmPilot inspired general conflict-resolution

5
Our Enabling Technologies

• Data Economy
• User pays monthly fee to a primary utility
  provider who is responsible for reliability of
  data
• Utilities buy and sell capacity (both data and
  bandwidth) prices set for quantity and
  reliability
• Authoritative naming servers paid per query?
• Underlying database organization
• User-visible structure (e.g. filesystem)
  synthesized
• Federation of overlapping data location
  structures (indices) Introspection
• Separate the absolute authority for data location
  from moment-to-moment hearsay authorities
• Partially consistent indices continually adapted
  to improve performance
• Conflict Resolution, not consistency
• policies set via domain specific language

6
3 Year Plan for Success

• Year1
• Initial design and refinement of four components
• naming security scheme (security based on name)
• fluid, partially coherent index structures
• introspection for intelligent migration of data
• initial take on economic models
• Begin prototype implementation with all
  components
• Year2
• Finish prototyping and refinement of
  first-generation
• Client implementation for Windows and/or UNIX
• Year 3
• Second-generation prototype on Millenium
  infrastructure
• formulate plan for large-scale test
• Final evaluation and usability results