Nache: Design and Implementation of a Caching Proxy for NFSv4

1
Nache Design and Implementation of a Caching
Proxy for NFSv4

• Ajay Gulati, Rice University
• Manoj Naik, IBM Almaden
• Renu Tewari, IBM Almaden

2
Talk Outline

• Federated File System
• NFSv4 Overview and New Features
• Nache Architecture
• Nache Implementation
• Experimental Results
• Conclusions and Discussion

3
Federated File System

• File system federation with unified namespace
• Flexible data management
• Replication, migration and read/write caching
  across distributed file servers
• What are the options?
• AFS, DCE/DFS provide distributed FS over WAN,
  but
• difficult to deploy, not widely used
• GPFS, Lustre are cluster filesystems, but
• good for controlled environments, no federation
• NFS is a standard, but
• v2/v3 have been designed mainly for LAN, are
  chatty and not suited for WAN, firewall issues
• NFSv4?

4
Why NFSv4?

• Open industry standard
• Optimizations for WAN
• Compound operations
• A single RPC request contains multiple ops,
  reduces round-trips
• Read-Write delegations
• Efficient cache consistency, no need to contact
  server once a delegation is obtained
• Delegations can be recalled on conflicting access
• Federation support
• Client redirection
• Server returns NFS4ERR_MOVED to redirect clients
• Client requests fs_locations attribute
• Client connects to new location

5
Filesystem Namespace Unification
Almaden
Houston
New York
Clients
Separate namespace across servers
6
Data Access using NFSv4 Redirection
/project
Server (New York)
Server (San Jose)
Clients (San Jose)

• Limitations
• High network latency when clients access remote
  servers
• Delegation are unsuited for data sharing, fewer
  awarded and frequently broken
• No unified caching, every client has to access
  remote server

7
What is a Proxy Cache (Nache)?

• An NFSv4 server acts as a proxy for another
  remote NFS server
• caches data for reads and writes using
  delegations
• becomes the temporary owner of the data
  servicing opens, lock requests and reads and
  writes
• Multiple clients can share a proxy cache using a
  single server-to-proxy delegation
• Extends beyond web-proxy caching

8
Data Access using a Proxy Cache
Server (New York)
Proxy Cache (San Jose)
/project
Clients (San Jose)

• Benefits with NFSv4 Proxy
• Reduced network traffic and delay
• Clients access files through local proxy cache
• Clients can share delegations fewer
  conflicts/recalls
• Improved performance for cached files
• Clients can collectively take advantage of
  aggressive readahead, write back and local
  locking at the proxy
• Both data and delegations are shared

9
Nache Architecture
(Linux nfsd module)
(Linux nfs module)
Nache Server
Nache Client
Remote Server (mount from)
Local Client (export to)
VFS Layer
cacheFS
Buffer Cache
Nache
10
Nache Implementation

• Cascaded mounts
• NFS operation forwarding
• Sub-operation RPC call handling

11
Cascaded Mounts

• Proxy exports a NFS mounted file system
• Added export functionality for NFS
• export_ops added, not available in vanilla NFS

ls /net at client shows contents of /watson at
server
12
NFS Operation Forwarding

• NFS ops are redirected from nfsd to nfs using vfs
  layer
• Issues certain operations are stateful
• OPEN, CREATE, LOCK, ULOCK, CLOSE

Application
VFS
VFS
VFS
NFSD
Nache Client
NFS Client
Nache Server
Server
Client
Nache
13
Sub-Operation RPC calls

• Each FS operation at Nache server translates to a
  separate RPC call from Nache client

• Solution
• Remove such redundant checks from code path ?

14
Experimental Methodology

• Setup
• 2-6 local NFS clients (San Jose)
• One local proxy with modified kernel modules (San
  Jose)
• One local NFS server (San Jose)
• One remote NFS server (New York)
• Benchmarks
• Filebench (developed by Sun)
• Compilation of various packages Linux kernel,
  emacs, gdb
• Our micro-benchmarks

15
Experiment Categories

• Delegation Benefits
• Nache Benefits
• Nache Overhead

16
Read Delegation Benefits
Client repeats open-read-close on 100
files Server ops reduced by 50 due to delegations
17
Write Delegation Benefits
A client repeats open-read/write-close on 100
files Server ops reduced by 4 times
18
Delegation Benefits (Server load)
Server ops reduced due to delegations
19
Delegation Benefits (Client performance)
Time taken is lower and ops/sec is higher due
to delegations
20
Experiment Categories

• Delegation Benefits
• Nache Benefits
• Nache Overhead

21
Nache Benefits - Filebench (webserver)
Ops at server reduces by 38 with 4 clients
22
Nache Benefits Kernel compile
Ops at server unaffected by increase in number of
clients
23
Nache Benefits Compile Time
Compile time unaffected by increase in number of
clients
24
Experiment Categories

• Delegation Benefits
• Nache Benefits
• Nache Overhead

25
Nache Overhead over WAN (Throughput)
Micro benchmark Ops/s (Nache) Ops/s (no Nache) Overhead()
Create-Files 9 16 43
Random-Reads 77 48 -60
Random-Writes 8.6 10 14
Random-Append 2.2 2.4 8
Nache induces an overhead of 8-43 Code can be
further optimized to reduce overheads
26
Nache Overhead over WAN (Latency)
Micro benchmark Latency (Nache) Latency (no Nache) Overhead ()
Create-Files 5217 3009 42
Random-Reads 12.9 20.7 -37
Random-Writes 116.3 98 18
Random-Append 883.4 821 8
Proxy induces an overhead of 8-42 Code can be
further optimized to reduce overheads
27
Result Summary

• Delegations provide better throughput, lower
  latency and reduced load at server
• Nache improves performance in presence of sharing
  among clients
• Overhead of Nache in absence of sharing is not
  too high except for few cases and can be further
  reduced

28
Conclusions

• NFSv4 is a good alternative for building
  federated file systems
• Delegations help in reducing the number of server
  operations and provide better caching
• Nache can be integrated with a federated file
  system architecture and improve performance in
  presence of sharing