Building a Reliable Mutable File System on Peertopeer storage

1
Building a Reliable Mutable File System on
Peer-to-peer storage

• Talk by Lex Stein
• About work with Mike Tucker and Margo Seltzer
• Harvard University

IEEE Workshop on RPPDS Oct 13, 2002
2
Separate filesystems from data storage.

• Filesystem management quotas, users, groups,
  access control. These are difficult to implement
  in a completely decentralized system.
• Data storage Simple get-put block interface.
  DHTs can be used to build decentralized storage
  systems.

3
Why use a content-addressed, block-immutable DHT
for datablock storage ?

• Distributes blocks evenly across nodes.
• Addresses verify data integrity.
• Facilitates sharing of underlying storage
  resources across filesystem partitions.
• No cache consistency issues.

4
Why is it difficult to build a conventional
Read-Write (mutable) filesystem on a DHT ?
Example Modification of file /etc/foo
Root directory datablock
etc
/etc directory inode
/etc/ directory datablock
foo
/etc/foo file inode
/etc/foo file datablock
X
5
Related Work

• Ivy - For small, distributed cooperative groups.
  Uses special mutable DHT blocks. Built on
  Chord/Dhash.
• SFSRO - Uses content-hashes to name blocks.
  Read-only, but full filesystems can be repackaged
  and injected.

6
Problems with special mutable blocks in DHTs.

• How does server know to trust clients update ?
• How does client know to trust data read from
  server ?
• Solutions use public-key crypto.
• But face private key distribution problems.

7
A distributed filesystem for the Real
WorldExample Harvard University Deployment
Comp. Sci.
Literature
Metadata paths

• 30 more departments

Linguistics
Faculty of Arts and Sciences
DHT
Data paths
Another small department
8
Client architecture choices
Sharing

• NFS

AFS
NFS bridge/loopback
Binary Interposition Library
Kernel-level VFS/Vnode
9
The Metadata Service (MS)

• Can be implemented in a number of ways
• A replicated database server.
• A fault-tolerant compute cluster.
• Clients insulated from design of the MS.

10
Reliability

• Applications will starve if DHT fails.
• Filesystem depends on Metadata service for
  integrity.

11
Conclusion

• Eliot is a hybrid system.
• Combines DHTs with classical fault-tolerance.
• Eliot
• Preserves immutability of DHT (no public-key
  crypto or cache consistency problems).
• Low-latency for common metadata operations.
• Flexible and describable sharing semantics.

12

• http//www.eecs.harvard.edu/stein/