Network File Systems

1
Network File Systems

• Victoria Krafft
• CS 614
• 10/4/05

2
General Idea

• People move around
• Machines may want to share data
• Want a system with
• No new interface for applications
• No need to copy all the data
• No space consuming version control

3
Network File Systems
Diagram from http//www.cs.binghamton.edu/kang/cs
552/note11.ppt
4
A Brief History

• Network File System (NFS) developed in 1984
• Simple client-server model
• Some problems
• Andrew File System (AFS) developed in 1985
• Better performance
• More caching client-side
• SFS 1999
• NFS can be run on untrusted networks

5
Lingering Issues

• Central server is a major bottleneck
• All choices still require lots of bandwidth
• LANs getting faster lower latency
• Remote memory faster than local disk
• ATM faster with more nodes sending data

6
Cooperative Caching

• Michael D. Dahlin, Randolph Y. Wang, Thomas E.
  Anderson, and David A. Patterson in 1994
• ATM, Myrinet provide faster, low-latency network
• This makes remote memory 10-20x faster than disk
• Want to get data from memory of other clients
  rather than server disk

7
Cooperative Caching

• Data can be found in
• Local memory
• Server memory
• Other client memory
• Server disk
• How should we distribute cache data?

8
Design Decisions
Private/Global Coop. Cache?
Private
Global
Coordinated Cache Entries?
Direct Client Cooperation
Coordination
No Coordination
Static/Dynamic Partition?
Greedy Forwarding
Dynamic
Static
Block Location?
Weighted LRU
N-Chance
Any Client
Fixed
Hash
Cent. Coord
9
Direct Client Cooperation

• Active clients use idle client memory as a
  backing store
• Simple
• Dont get info from other active clients

10
Greedy Forwarding

• Each client manages its local cache greedily
• Server stores contents of client caches
• Still potentially large amounts of data
  duplication
• No major cost for performance improvements

11
Centrally Coordinated Caching

• Client cache split into two parts local and
  global

12
N-Chance Forwarding

• Clients prefer to cache singlets, blocks stored
  in only one client cache.
• Instead of discarding a singlet, set
  recirculation count to n and pass on to a random
  other client.

13
Sensitivity
Variation in Response Time with Client Cache Size
Variation in Response Time with Network Latency
14
Simulation results
Average read response time
Sever load
15
Simulation results
Slowdown
Slowdown
16
Results

• N-Chance forwarding close to best possible
  performance
• Requires clients to trust each other
• Requires fast network

17
Serverless NFS

• Thomas E. Anderson, Michael D. Dahlin, Jeanna M.
  Neefe, David A. Patterson, Drew S. Roselli, and
  Randolph Y. Wang in 1995
• Eliminates central server
• Takes advantage of ATM and Myrinet

18
Starting points

• RAID Redundancy if nodes leave or fail
• LFS Recovery when system fails
• Zebra Combines LFS and RAID for distributed
  systems
• Multiprocessor Cache Consistency Invalidating
  stale cache info

19
To Eliminate Central Servers

• Scaleable distributed metadata, which can be
  reconfigured after a failure
• Scalable division into groups for efficient
  storage
• Scalable log cleaning

20
How it works

• Each machine has one or more roles
• Client
• Storage Server
• Manager
• Cleaner
• Management split among metadata managers
• Disks clustered into stripe groups for
  scalability
• Cooperative caching among clients

21
xFS

• xFS is a prototype of the serverless network file
  system
• Lacks a couple features
• Recovery not completed
• Doesnt calculate or distribute new manager or
  stripe group maps
• No distributed cleaner

22
File Read
23
File Write

• Buffered into segments in local memory
• Client commits to storage
• Client notifies managers of modified blocks
• Managers update index nodes imaps
• Periodically, managers log changes to stable
  storage

24
Distributing File Management

• First Writer management goes to whoever created
  the file

does not include all local hits
25
Cleaning

• Segment utilization maintained by segment writer
• Segment utilization stored in s-files
• Cleaning controlled by stripe group leader
• Optimistic Concurrency control resolves cleaning
  / writing conflicts

26
Recovery

• Several steps are O(N2), but can be run in
  parallel

Steps For Recovery
27
xFS Performance
NFS max with 2 clients
NFS max with 2 clients
AFS max with 32 clients
AFS max with 12 clients
Aggregate Bandwidth Writing 10MB files
Aggregate Bandwidth Reading 10MB files
28
xFS Performance
Average time to complete the Andrew benchmark,
varying the number of simultaneous clients
29
System Variables
Aggregate Large-Write Bandwidth with Different
Storage Server Configurations
Variation in Average Small File Creation Speed
with more Managers
30
Possible Problems

• System relies on secure network between machines,
  and trusted kernels on distributed nodes
• Testing done on Myrinet

31
Low-Bandwidth NFS

• Want efficient remote access over slow or wide
  area networks
• File systems better than CVS, copying all data
  over
• Want close-to-open consistency

32
LBFS

• Large client cache containing users working set
  of files
• Dont send all the data reconstitute files from
  previous data, and only send changes

33
File indexing

• Non-overlapping chunks between 2K and 64K
• Broken up using 48 byte Rabin fingerprints
• Identified by SHA-1 hash, indexing on first 64
  bits
• Stored in database, recomputed before use to
  avoid synchronization issues

34
Protocol

• Based on NFS, added GETHASH, MKTMPFILE, TMPWRITE,
  CONDWRITE, COMMITTMP
• Security infrastructure from SFS
• Whole file caching
• Retrieve from server on read unless valid copy in
  cache
• Write back to server when file closed

35
File Reads
36
File Writes
37
Implementation

• LBFS server accesses file system as an NFS client
• Server creates trash directory for temporary
  files
• Server inefficient when files overwritten or
  truncated, which could be fixed by lower-level
  access.
• Client uses xfs driver

38
Evaluation
39
Bandwidth consumption
Much higher bandwidth for first build
40
Application Performance
41
Bandwidth and Round Trip Time
42
Conclusions

• New technologies open up new possibilities for
  network file systems
• Cost of increased traffic over Ethernet may cause
  problems for xFS, cooperative caching.