NFS Failure Policy Fingerprinting

1
NFS Failure Policy Fingerprinting

• Swetha Krishnan Giridhar Ravipati
• CS739 Distributed Systems

2
Outline

• Goals
• Current Progress
• Next steps

3
Goals

• Analyze how the client/server react to failures
  network failures, local disk failures
• Find any inconsistencies by injecting errors
• Possible recovery mechanisms

4
Goals2

• Drop packets before being received by the NFS
  client
• Corruption at several layers
• IP, RPC, NFS
• Server/client crashes
• Graceful reboots and unexpected crashes

5
Tools

• NFSDump, Nistnet
• Customized Nistnet to recognize the NFS packets
  and selectively filter the ones to drop

6
Current Progress

• Examined packet drops at client
• Focus on Non-Idempotent requests
• Operations that may return an error when
  performed more than once (REMOVE, CREAT, SYMLINK
  etc)
• Modified client kernel to extract request
  information

7
Duplicate Request Cache

• Servers cache the requests-responses in the
  duplicate request cache
• Purpose avoid incorrect semantics for
  non-idempotent operations
• Found inconsistencies by varying the number of
  packet drops
• Returns an error when in fact the operation was
  performed successfully

8
Results

• Packet drops - 15
• Experiments conducted by changing the cache value
  expiry period
• 120HZ - Server tries to do it twice
• 240HZ All requests are satisfied from the cache

9
Next steps

• Server crashes
• Kill certain processes
• Reboot server at crucial points
• Inject panics through fault injection driver
• Packet corruption
• Data
• Metadata filehandle, xid ,timestamps
• Possible recovery means