Recovering Device Drivers

1
Recovering Device Drivers

• Michael M. Swift, ... Henry M. Levy
• DCSE, University of Washington

2
Motivation

• Improving reliability is the need of the era
• Most System failures caused by device drivers
• Earlier failure-isolation systems can prevent the
  kernel from drivers crash
• But applications using failed driver can still
  crash

3
Shadow driver

• During normal operation, the shadow tracks the
  state of the real driver by monitoring all
  communication between the kernel and the driver
• When a failure occurs, the shadow inserts itself
  temporarily in place of the failed driver,
  servicing requests on its behalf.
• While shielding the kernel and applications from
  the failure, the shadow driver restores the
  failed driver to a state where it can resume
  processing requests.

4
Design Principles

• Device driver failures should be concealed from
  the drivers clients.
• Recovery logic should be centralized in a single
  subsystem.
• Driver recovery logic should be generic.
• Recovery services should have low overhead when
  not needed.

5
Drivers structure
6
Driver Faults

• Deterministic failures
• Transient failures
• Fail-stop
• Shadow drivers can recover only from failures
  that are both transient and fail-stop

7
Shadow Drivers Functions

• Shadow driver monitors communication between a
  functioning driver and the kernel and
  impersonates one component to the other during
  failure and recovery
• Passive Mode
• Active Mode
• A shadow driver is a class driver
• Taps

8
Passive Mode

• Tap invokes the original driver
• Then tap invokes the shadow driver with the
  parameters and results of the call

9
Active Mode

• Tap terminates all communication between the
  driver and kernel
• redirects all invocations to their corresponding
  interface in the shadow
• In active mode, both the kernel and the
  recovering device driver interact only with the
  shadow driver
• Following recovery, the tap returns to its
  passive-mode state

10
The Shadow Manager

• Recovery is supervised by the shadow manager,
  which is a kernel agent that interfaces with and
  controls all shadow drivers
• SD instantiates
• State transitions
• Driver restore

11
Generic Infrastructure relied by SD

• isolation service
• transparent redirection mechanism
• object tracking service
• Nooks

12
Shadow Driver Architecture
13
Passive-Mode Monitoring

• Tracks requests made to the driver
• Records configuration and driver parameters
• Tracks all kernel objects that the driver
  allocated or received from the kernel
• Example
• Sound card driver just return
• Ioctl logs commands and data
• IDE does little or no work
• Network keeps references to outstanding packets

14
Active-Mode Recovery

• Stopping the failed driver
• Reinitializing the driver from a clean state
• Transferring relevant shadow driver state into
  the new driver

15
Stopping the failed driver
Shadow Manager
Failure detected
Garbage collects resources except kernel object
Disable failed driver
Sound Card
Disable hardware device (Disable IRQ line Remove
I/O Mapping, etc)
16
Reinitializing the Driver

• Initializes the drivers state and then repeats
  the kernels driver initialization sequence
• Reattaches the driver to its pre-failure kernel
  resources
• Re-enables hardware resources such as interrupt
  line
• Shadow driver responds as kernel

17
Transferring State to the New Driver

• Restores any configuration downloaded to the
  driver
• Handles requests that were either outstanding
  when the driver crashed or arrived while the
  driver was recovering
• Notifies shadow manager switching taps state

18
Active-Mode Proxying of kernel Requests

• respond with information that it has recorded
• silently drop the request
• queue the request for later processing
• block the request until the driver recovers
• report that the driver is busy and the kernel or
  application should try again later

19
Limitations

• Shadow drivers rely on dynamic unloading and
  reloading of device drivers
• Shadow drivers rely on explicit communication
  between the device driver and kernel
• Shadow drivers assume that driver failure does
  not cause irreversible side effects
• Effectiveness is limited by the abilities of the
  isolation and failure-detection subsystem
• Shadow drivers may not be suitable for
  applications with real-time demands

20
Evaluation

• Performance
• Fault-Tolerance
• Limitations
• Code size

21
Experimental Environment
22
Performance
23
Performance CPU
24
Fault Tolerance
25
Limitation
26
Code Size
27
Conclusion

• SDs mask device driver failures from
  applications, allowing applications to run
  normally during and after a driver failure
• SDs impose minimal performance overhead
• SDs require no changes to existing applications
  and device drivers
• SDs integrate easily into an existing operating
  system

28
Sound Card Example

• Passive-Mode Monitoring
• For read and write calls, sd implemented as
  no-ops
• For an ioctl call, sd logs the command and data
  for connection
• Active-Mode Recovery
• Resets the sound driver and all its open
  connection back to pre-failures state. Scans
  lists of open connections and calls open
  function.
• Walks sds log of configuration commands and
  replays any commands to set driver properties.
• For some properties without interface to set, sd
  intercepts calls and inserts the right value into
  return argument

29
Sound Card Example

• Active-Mode Proxying of Kernel Requests
• Uses a mix of all five strategies for emulating
  functions in its service interface
• Blocks kernel read and write requests
• Processes ioctl calls itself, either by
  responding with information it captured or by
  logging the request to be processed later.
• For idempotent ioctl commands, silently drops
  duplicate requests
• When apps query for buffer space, responds that
  buffers are full

30
Nooks
31
Nooks Layer Inside Linux
32
Protection of the kernel space
33
Control flow of extension and kernel wrappers