Speculative Execution in a Distributed File System

1
Speculative Execution in a Distributed File System

• Ed Nightingale
• Peter Chen
• Jason Flinn
• University of Michigan
• Best Paper at SOSP 2005
• Modified for CS739

2
Motivation

• Why are distributed file systems slow(er)?
• Sync network messages provide consistency
• Sync disk writes provide safety
• Sacrifice guarantees for speed
• Can DFS can be safe, consistent and fast?
• Yes! With OS support for speculative execution

3
Big Idea Slow Way
Big Idea Speculator
Client
Server
1) Checkpoint
RPC Req
RPC Req
Block!
2) Speculate!
RPC Resp
RPC Resp
3) Correct?
Yes discard ckpt.
No restore process re-execute
RPC Req
RPC Resp

• Guarantees without blocking I/O!

4
Conditions for Success

• Operations are highly predictable
• Conflicts are rare
• Checkpoints are cheaper than network I/O
• 52 µs for small process (6.3 ms for 64 MB
  process)
• Computers have resources to spare
• Need memory and CPU cycles for speculation

5
Assumptions --gt Experiments

• Operations are highly predictable
• What happens if not (lots of sharing)?
• Will cost of checkpointing and rolling back ever
  harm performance?
• Checkpoints are cheaper than network I/O
• What happens with large memory processes?
• Computers have resources to spare (Clients)
• What happens when running multiple processes?
• Can network become bottleneck?
• Can server become bottleneck?

6
Outline

• Motivation
• Implementing speculation
• Multi-process speculation
• Using Speculator
• Evaluation

7
Implementing Speculation

• Implementation
• All within OS
• No changes needed to applications
• Three new interfaces
• Create_speculation()
• Commit_speculation()
• Fail_speculation()
• Goal Ensure speculative state is
• Never externalized (sent to terminal, disk,
  network)
• Never directly observed by non-speculative
  processes

8
Implementing Speculation
1) System call
2) Create speculation
Time
Process
9
Speculation Success
1) System call
2) Create speculation
3) Commit speculation
Time
Process
Checkpoint
10
Speculation Failure
2) Create speculation
1) System call
3) Fail speculation
Time
Process
Process
Checkpoint
11
Replay after Failure

• Does code after checkpoint need to be
  deterministic for correct results?

12
Ensuring Correctness

• Spec processes often affect external state
• Speculative state should never be visible to user
  or any external device
• Process should never view speculative state
  unless it is already speculatively dependent on
  that state
• Three ways to ensure correct execution
• Block
• Buffer
• Propagate speculations (dependencies)

13
Systems Calls

• Modify system call jump table
• Block calls that externalize state
• Allow read-only calls (e.g. getpid)
• Allow calls that modify only task state (e.g.
  dup2)
• File system calls -- need to dig deeper
• Mark file systems that support Speculator

getpid
Call sys_getpid()
reboot
Block until specs resolved
mkdir
Allow only if fs supports Speculator
14
Output Commits Buffer
1) sys_stat
2) sys_mkdir
3) Commit spec 1
Time
Process
stat worked
Checkpoint
Checkpoint
mkdir worked
15
Multi-Process Speculation

• Processes often cooperate
• Example make forks children to compile, link,
  etc.
• Would block if speculation limited to one task
• Allow kernel objects to have speculative state
• Examples inodes, signals, pipes, Unix sockets,
  etc.
• Propagate dependencies among objects
• Objects rolled back to prior states when specs
  fail

16
Multi-Process Speculation
Stat A
Stat B
Checkpoint
Checkpoint
Checkpoint
Checkpoint
Checkpoint
pid 8001
pid 8000
Chown-1
Chown-1
Write-1
Write-1
inode 3456
17
Multi-Process Speculation

• What we handle
• DFS objects, RAMFS, Ext3, Pipes FIFOs
• Unix Sockets, Signals, Fork Exit
• What we dont (i.e. we block)
• System V IPC
• Multi-process write-shared memory

18
Outline

• Motivation
• Implementing speculation
• Multi-process speculation
• Using Speculator
• Evaluation

19
Example NFSv3 Linux
Client 1
Client 2
Server
Modify B
Write
Commit
Open B
Getattr
Why were asynchronous writes added to NFSv3? Is
this situation of no overlap expected to be the
common case? When can the commit return from the
server?
20
Example SpecNFS
Client 1
Client 2
Server
WriteCommit
Modify B
speculate
Getattr
Open B
speculate
Getattr
Open B
speculate
When can the commit return from the server?
21
Problem Mutating Operations
Client 1 1. cat foo gt bar
Client 2 2. cat bar

• bar depends on cat foo
• What does client 2 view in bar?

22
Solution Mutating Operations

• Server determines speculation success/failure
• State at server never speculative
• Send server hypothesis speculation based on
• List of speculations an operation depends on
• Requires server to track failed speculations
• Would like to be convinced OK when server
  crashes!
• Requires in-order processing of messages

23
Group Commit

• Previously sequential ops now concurrent
• Sync ops usually committed to disk
• Speculator makes group commit possible

Updating different files
Client
Client
Server
Server
write
commit
write
commit
Can significantly improve disk throughput
24
Putting it all Together SpecNFS

• Apply Speculator to an existing file system
• Modified NFSv3 in Linux 2.4 kernel
• Same RPCs issued (but many now asynchronous)
• SpecNFS has same consistency, safety as NFS
• Getattr, lookup, access speculate if data in
  cache
• Create, mkdir, commit, etc. always speculate
• Choose aliases for unknown file handles

25
Putting it all Together BlueFS

• Design a new file system for Speculator
• Single copy semantics
• Synchronous I/O
• Each file, directory, etc. has version number
• Incremented on each mutating op (e.g. on write)
• Checked prior to all operations.
• Many ops speculate and check version async

26
Outline

• Motivation
• Implementing speculation
• Multi-process speculation
• Using Speculator
• Evaluation

27
Apache Benchmark

• SpecNFS up to 14 times faster

28
Apache Benchmark
29
The Cost of Rollback (Q1)

• All files out of date SpecNFS up to 11x faster

30
Evaluation

• Other experiments you would like to see?

31
Evaluation

• Other experiments you would like to see?
• Impact of checkpoint size
• Resource constraints Scalability
• How many clients can server handle?
• Is network a constraint?
• Server must still handle poll requests, even if
  in bg
• Fault tolerance
• Server now maintains state per-client list of
  failed speculations

32
Conclusion

• Speculator greatly improves performance of
  existing distributed file systems
• Especially in wide-area
• Speculator enables new file systems to be safe,
  consistent and fast
• How general do you think Speculator is?
• Speculation clearly improves poll-based
  consistency (NFSv3, BlueFS)
• How much improvement with callbacks and leases?
  (AFS, Coda, NFSv4)?

33
Group Commit Sharing State
34
Related Work

• Chang Gibson, Fraser Chang
• Speculative pre-fetching
• Time Warp
• Virtual Time distributed simulations
• Hardware branch prediction
• Transactional file systems