Directory Delegations in NFSv4

1
Directory Delegations in NFSv4
Brian Wickman

• CITI
• University of Michigan
• Ann Arbor

2
NFS version 4

• NFSv4 introduces state!
• OPEN and CLOSE
• integrated locking
• file delegations
• Delegation Paradigm
• Server-driven optimizations visible to the client
  but transparent to applications.
• More client responsibility but better
  performance!

3
NFSv4 File Delegations

• Server-driven optimization Server has final
  authority over delegating ability to clients
• Transparent to applications No special
  application-level tooling required, otherwise
  delegation utility would be undermined.
• Better performance Local locking! Smart
  caching! Less latency! Less traffic!

4
NFS Directory Delegations

• Want to delegate some portion of directory state
  to client
• What to delegate?
• Potential state
• Filename to filehandle bindings (DNLC)
• File attributes
• Anything else?

5
How to Enable Directory Delegations?

• Introduce pair of ops, OPENDIR/CLOSEDIR?
• Better mirroring of file delegation semantics
• Same interoperability concerns as delegating v4.0
  servers speaking with v2/v3 clients.
• Introduce single-op? GIVE_ME_DELEGATION_PRETTY_PL
  EASE?
• Half as many ops to implement. Works, too.
• How information-rich? Notifications?
  Asynchronous, synchronous, whaaaa?

6
draft-ietf-nfsv4-directory-delegation-01.txt

• Saadias proposal
• New op (GET_DIR_DELEGATION)
• New callback (CB_NOTIFY)
• Delegations and notifications, all in one
• Read-only, as specified
• Information rich, fine-grained
• Lots of new functionality to code, especially for
  server implementer

7
draft-wickman-nfsv4-directory-delegations-00.txt

• My proposal
• New op (REQUEST_DELEGATION)
• No new callbacks
• No notifications, existing data structures
  unchanged
• Read and write delegations specified
• Minimal information, coarse-grained
• Implementation more straightforward?

8
Write directory delegations

• Server doesnt need to change its behavior,
  relies more on client implementer
• If the client is clever it can
• Issue directory updates asynchronously
• Perform whole-tree write delegation on new
  directories
• Effectively own file write delegations on all
  files within a write delegated directory

9
Write delegations and STATEIDs

• Overlooked in my draft, need errata
• Directory ops such as READDIR or MKDIR do not
  reference a delegation stateid
• Need PUT_STATEID function and current STATEID
  so that we can modify directories on which we
  hold delegations
• Discussed at Connectathon?

10
Evaluating the proposals

• For starters, how do you define better?
• My take how many ops can you service locally and
  at what extra cost, given common workloads?
• So far, Ive investigated this for read-only
  directory delegations and for two common(?)
  workloads.

11
Steps to Evaluating Delegation Behavior

• Passively capture traces from NFSv3 deployments
  with thousands of users (v4 deployments, anyone?)
• Build an NFS (v3) metadata simulator
• Instrument extra information into traces (e.g.,
  OPEN, CLOSE, REQUEST_DELEGATION)
• Reconstruct filesystem image on the fly
• Observe delegation behavior!
• Measure what you see

12
Traces Used

• From Dan Ellards PhD research
• CAMPUS trace
• Read dominated e-mail workload
• Good test of notification behavior? (taking pulse
  of mail spool)
• Digital UNIX server, one week, late October 2001,
  242M v3 ops
• EECS trace
• Write-dominated research workload, lots of small
  files
• NetApp filer, one week, late October 2001, 41M v3
  ops

13
Simple use of Directory Delegations

• For fun, I emulated the v3 cache algorithm using
  the REQUEST_DELEGATION op
• When first caching directory entries,
  REQUEST_DELEGATION
• If clock expires and no revocation, do nothing
• If revocation occurs, issue all LOOKUPs to
  server, synchronously until clock expiration
• Reacquire delegation at clock expiration

14
Simple use of Directory Delegations

• Use liberal server policy for granting directory
  delegations (i.e., always-grant)
• In both traces, gt90 of LOOKUPs are serviced
  locally and guaranteed consistent
• In both traces, gt95 of directory cache
  validation traffic is eliminated
• - with unbounded lookup caches

15
What about with notifications?

• I also implemented the same algorithm but with
  GET_DIR_DELEGATION
• We can service a bunch of extra LOOKUPs
  (practically all) because the server notifies us
  when our lookup cache changes and what is
  entailed in each change, rather than revoke
• How much can we service and at what cost?

16
Not much
CAMPUS EECS
Extra LOOKUPs serviced locally 162K 14K
Percentage of total LOOKUPs 2.1 0.4
Notifications received 9M 164K
Ratio of Notifications to extra LOOKUPs 56x 12x
17
FILE_ATTR notifications

• If FILE_ATTR notifications are added, the ratio
  goes from 56x to 500x for CAMPUS
• Millions of DIR_ATTR notifications to service
  tens of thousands of LOOKUPs
• Tens of millions of FILE_ATTR notifications (67M)
  to service millions of GETATTRs (2.3M)

18
Reducing Notification Traffic

• Can we win if using asynchronous notifications?
• If nothing else, we can reduce traffic.
• Because the client can specify a notification
  window, the server has the opportunity to
  eliminate self-nullifying updates (e.g., ADD fh
  REMOVE fh) within that window.
• Asynchronous notifications would still mean less
  LOOKUP consistency than no notifications at all,
  as long as revocations are synchronous

19
Reducing Notification Traffic
20
Another Drawback of Notifications

• How many notifications refer to directory objects
  whose data or metadata we are actively accessing?
• CAMPUS 4
• EECS lt1
• Are we taking sips from a fire-hose?
• Also, knowing the existence of a change may be
  sufficient more often than knowing the change
  itself.

21
Summary

• A simple directory delegation scheme without
  notifications gives immediate improvements in
  latency and caching, in the read-only case
• Notifications have the tendency to create lots of
  information that is otherwise unused by the
  client
• Still much more work to be done

22
TODO

• Start looking at v4 traces. Of course, I need v4
  traces, especially Windows v4 traces Anyone want
  to help?
• Simulating write directory delegations
• Looking at other lookup cache maintenance
  algorithms (instead of just clock-based)
• Adding more variables to the simulator (cache
  eviction, for example)
• Implementing directory delegations for real

23
Questions?