Speculative Execution in a Distributed File System

1
Speculative Execution in a Distributed File System

• - Sumanth Gelle

2
Outline

• Introduction
• Motivation Speculation in NFS
• Conditions For Success
• Interface for Speculation
• Implementing Speculation
• Multi process Speculation
• Using Speculation
• Evaluation

3
Motivation

• Why are distributed file systems slow(er)?
• Sync n/w messages provide consistency
• Sync disk writes provide safety
• Sacrifice guarantees for speed
• Can DFS can be safe, consistent and fast?
• Yes! With OS support for speculative execution

4
Big Idea Slow Way
Big Idea Speculator
Client
Server
1) Checkpoint
RPC Req
RPC Req
Block!
2) Speculate!
RPC Resp
RPC Resp
3) Correct?
Yes discard ckpt.
No restore process re-execute
RPC Req
RPC Resp

• Guarantees without blocking I/O!

5
Motivation Speculation in NFS

• To close a file NFSv3 client needs to
• Write RPC
• Commit RPC
• To open a file NFSv3 clinet needs to
• Getattr RPC

6
Example Scenario
7
Conditions for Success

• Operations are highly predictable
• Conflicts are rare
• Checkpoints are cheaper than network I/O
• 52 µs for small process
• Computers have resources to spare
• Need memory and CPU cycles for speculation

8
An Interface for Speculation

• To initiate speculative process calls create_
  speculation.
• This function returns the spec_id that uniquely
  identifies the particular speculation.
• Based on the Speculation result the process
  executes
• commit_speculation or fail_speculation.

9
Implementing Speculation

• Process Checkpoint and Rollback
• Copy-on-write fork of the currently running
  process.
• On failure mark the process as failed in its task
  structure .
• Check point process assumes the identity of
  failed process.
• Speculation
• Adds two data structures to the kernel
• Speculation structure created during
  create_speculation. Contains set of kernel
  objects that depend on the new speculation.
• Undo log - associated with each kernel object
  that has speculative state
• Sharing of a checkpoint for those speculations
  which do not change state.
• Ensuring Correct Speculation Execution.

10
Implementing Speculation
1) System call
2) Create speculation
Time
Process
Checkpoint
11
Speculation Success
1) System call
2) Create speculation
3) Commit speculation
Time
Process
Checkpoint
12
Speculation Failure
2) Create speculation
1) System call
3) Fail speculation
Time
Process
Process
Checkpoint
13
Ensuring Correctness

• Spec processes often affect external state
• Three ways to ensure correct execution
• Block
• Buffer
• Propagate speculations (dependencies)
• New system call Jump table.
• sys_spec_deny a function to block the calling
  process until the speculations are resolved.

14
Multi Process Speculation

• Speculators strategy for IPC allows selected
  data structures within the kernel to have
  speculative state.
• Speculator propagates dependencies from a process
  P to an object X whenever P modifies X and P
  depends on speculations that X does not.
• Speculator propagates dependencies from an object
  X to a process P whenever P observes Xs state
  and X depends on speculations that P does not.

15
Multi-Process Speculation
Checkpoint
Checkpoint
Checkpoint
Checkpoint
Checkpoint
pid 8001
pid 8000
Chown-1
Chown-1
Write-1
Write-1
inode 3456
16
(No Transcript)
17
Using Speculation

• Modifying a distributed file system to use
  Speculator typically involves changes to the
  client, server, and network protocol.
• The client calls commit speculation if it
  correctly predicted the result of the RPC and
  fail speculation if it did not.

18
Example NFSv3 Linux
Client 1
Client 2
Server
Modify B
Write
Commit
Open B
Getattr
19
Example SpecNFS
Client 1
Client 2
Server
WriteCommit
Modify B
speculate
Getattr
Open B
speculate
Getattr
Open B
speculate
20
Problem Mutating Operations
Client 1 1. cat foo gt bar
Client 2 2. cat bar

• bar depends on cat foo
• What does client 2 view in bar?

21
Solution Mutating Operations

• Server determines speculation success/failure
• State at server never speculative
• Send server hypothesis speculation based on
• List of speculations an operation depends on
• Requires server to track failed speculations
• Requires in-order processing of messages

22
Group Commit

• Previously sequential ops now concurrent
• Sync ops usually committed to disk
• Speculator makes group commit possible

Client
Client
Server
Server
write
commit
write
commit
23
Putting it all Together SpecNFS

• Apply Speculator to an existing file system
• Modified NFSv3 in Linux 2.4 kernel
• Same RPCs issued (but many now asynchronous)
• SpecNFS has same consistency, safety as NFS
• Getattr, lookup, access speculate if data in
  cache
• Create, mkdir, commit, etc. always speculate

24
Putting it all Together BlueFS

• Design a new file system for Speculator
• Single copy semantics
• Synchronous I/O
• Each file, directory, etc. has version number
• Incremented on each mutating op (e.g. on write)
• Checked prior to all operations.
• Many ops speculate and check version async

25
Apache Benchmark

• SpecNFS up to 14 times faster

26
The Cost of Rollback

• All files out of date SpecNFS up to 11x faster

27
Conclusion

• Speculator greatly improves performance of
  existing distributed file systems
• Speculator enables new file systems to be safe,
  consistent and fast

28
References

• http//www.eecs.umich.edu/pmchen/papers/nightinga
  le05.slides.ppt
• Speculative Execution in a Distributed File
  System EDMUND B. NIGHTINGALE, PETER M. CHEN, and
  JASON FLINN University of Michigan