Using Model-Checking to Debug Device Firmware

1
Using Model-Checking to Debug Device Firmware

• Sanjeev Kumar
• Microprocessor Research Labs, Intel
• Kai Li
• Princeton University

2
Programmable Devices
Main CPU
Main Memory
Bus
Network Card
Disk
Network
Move functionality from main CPUs to
devices Device firmware is more complex
3
Firmware for Programmable Devices

• Difficult to write and debug
• Use concurrency
• Inherently difficult to program correctly
• Optimized for high performance
• Tradeoff program simplicity for performance
• Limited debugging support available
• Firmware reliability is important
• Firmware is trusted by the OS
• Bugs in firmware can crash the entire machine
• Model checking is a promising approach

4
Model Checking

• Systematically verify properties of concurrent
  systems
• Using State-space exploration
• Try all possible scheduling options

• Advantages
• Automatic
• Produces counter example
• Disadvantages
• Computationally expensive
• Exponential search
• Models cannot be too big

5
Using Model Checking to Debug Firmware

• Extracting models from programs
• Manually (by the programmer)
• Automatic (using the compiler)
• Reduces programmer effort
• Reduces mismatch between program and model
• Extracting smaller models models
• Support for abstraction
• Discard irrelevant details in the program

6
Our Work

• Extract abstract models using a compiler
• General compiler techniques
• Programmer controls the abstraction process
• Specifies what needs to be abstracted
• Compiler performs the abstractions conservatively
• Extract models for Spin model checker from
  programs written in the ESP language
• Make practical choices
• Goal Debugging and not Verification
• Used to debug VMMC firmware for a network card
• Found 7 bugs that can cause the firmware to
  deadlock
• Could not find these bugs without support for
  abstraction

7
Related Work

• Manual Model Extraction
• Harmony, RUBIS, Plan 9, Fluke OS
• Used model checking to debug a subsystem
• Automatic Model Extraction
• Teapot, Promela, Esterel, Java Pathfinder
• Domain-specific and general-purpose languages
• Automatic Extraction Support for abstraction
• Feaver, Lie et. al., Bandera

8
Outline

• Background
• Extracting Abstract Models using a Compiler
• From ESP Language
• For Spin Model Checker
• Evaluation Debugging VMMC Firmware
• Conclusion and Future Work

9
ESP A Language for Programmable Devices

• Goals
• Easy to program
• Allow extensive testing
• Performance

10
The ESP Language

• Concurrent language Processes Channels
• Pure message-passing communication
• in, out, alt operations on channels
• Channels are synchronous or unbuffered
• Processes and channels are static
• A number of interesting features
• Explicit memory management scheme that uses
  model-checking to ensure safety
• Supports dispatch on channels
• Efficient and powerful interface to C

11
Extracting Models for Spin

• Step 1 Detailed models PLDI01
• Translate each language construct into Spin
• Simple translation
• int, bool, records, arrays, unions
• If-then-else, while-loops
• process, channel
• Spin does not support
• Dynamic memory allocation Pointers
• Additional bookkeeping necessary to support these
• Can be used to check local properties
• Debug subsystems (1-2 processes) separately
• Too big to debug the entire system

12
Extracting Abstract Models Conservatively

• Step 2 Abstract models
• Necessary to check global properties (like
  deadlocks)
• Drop unnecessary details
• Depending on the property being verified
• Programmer controls the abstraction
• Abstraction specified by the programmer
• Drop variables
• Drop fields from records and unions
• Compiler used the abstraction specified
  conservatively
• Could introduce fast-positive bugs
• All bugs in the programs will be present in the
  extracted model
• Involves dealing with a number of tricky cases

13
Examples
Conservative Use nondeterminism to broaden the
state-space searched
b2 boolean true ... b1 boolean b2
14
Outline

• Background
• Extracting Abstract Models using a Compiler
• Evaluation Debugging VMMC Firmware
• Conclusion and Future Work

15
VMMC

• High-performance communication
• Bypass OS for data transfers
• Used Myrinet network cards
• Gigabit network
• 33 MHz CPU, 1 MB memory
• Original VMMC firmware
• Implemented in C
• Several man-years of debugging
• Still encounter bugs
• Some involve complex race conditions that are
  triggered only occasionally

Application
Data
OS
Network Card
Network
16
Debugging VMMC Firmware

• Reimplemented VMMC firmware using ESP
• Used model checking to debug
• Global property of program (deadlocks)
• Hard-to-find bugs
• Found 7 bugs using abstract models
• 4 Bugs would cause deadlock during normal
  operations
• 3 Bugs would be triggered only by a malicious
  machine
• Could not find these bugs without abstractions
• No firmware bugs encountered on device
• Microbenchmarks
• SPLASH2 parallel application suite
• On a 16-processor SMP cluster

17
Resource used for Model Checking
VMMC Firmware
Spin Mode States Searched (in Millions) CPU Time (in seconds) Memory (in Mbytes)
Exhaustive 0.38 84.0 268.35
Partial mode 99.7 14250.0 167.92
Limiting Resource

• Only partial search was possible
• Even partial searches were effective

18
Model extracted from VMMC Firmware
File Lines of Code
ESP Program 453
Abstraction Specification 108
Abstract Model Extracted 2202
Test Code 128

• Programmer only write a small amount of Spin Code
• Program can be rechecked with little effort

19
Outline

• Background
• Extracting Abstract Models using a Compiler
• Evaluation Debugging VMMC Firmware
• Conclusion and Future Work

20
Conclusions

• Use compiler to extract abstract models
• Evaluation Debugged VMMC firmware
• Using compiler to extract models is good
• Significantly reduces effort required to model
  check
• Abstraction is required
• To check global properties like deadlocks
• Programmer can control the abstraction
• Compiler is conservative
• Does not require the programmer to be correct
• Only partial search was possible
• Still effective in finding bugs

21
Future Work

• Optimizations to reduce size of state space
• Eliminating more redundancies
• Quantify the effectiveness of a partial seach
• Estimate the fraction of state-space searched
• Use type systems to reduce the size of state
  space that has to be searched

22
Questions?
To find out more, Visit http//www.cs.princeton.ed
u/skumar
23
Debug not Verify

• Several sources of incompleteness and unsoundness
  remain
• Programmer supplied Spin code
• Partial model checking
• The goal is to isolate/reduce the unsound
  portions of the code

24
ABCDEF
Abcdef Ghijk
ABCDEF
Abcdef Ghijk
Abcdef Ghijk
Abcdef Ghijk
Abcdef Ghijk
ABCDEF
Abcdef Ghijk