DART: Directed Automated Random Testing

1
DART Directed Automated Random Testing

• Patrice Godefroid, Nils Klarlund, and Koushik Sen
• PLDI05

Presented by Jaymie Strecker Software
Chat September 16, 2005
2
Unit testing

• Code coverage
• Checking of corner cases
• Requires test drivers and stubs
• Expensive to do manually
• Done poorly or skipped

3
DART automates unit testing

• Automated interface extraction
• Random testing by test driver
• Directs program along unexplored paths
• Generates input to satisfy path constraints

p
a
?a
Path constraint ? ?a, ?b ?
q
r
b
?b
b
s
t
4
DART overview
DART
Instrumented program
Test driver
Compilable program
Program crashes
Non-termination
Assertions
Assertion violations
5
Introductory example
int f(int x) return 2 x int h(int x, int
y) if (x ! y) if (f(x) x 10)
abort() / error / return
0
First try random testing
6
Introductory example
int f(int x) return 2 x int h(int x, int
y) if (x ! y) if (f(x) x 10)
abort() / error / return
0
First try random testing
7
Introductory example
int f(int x) return 2 x int h(int x, int
y) if (x ! y) if (f(x) x 10)
abort() / error / return
0
Second try DART
8
Introductory example
int f(int x) return 2 x int h(int x, int
y) if (x ! y) if (f(x) x 10)
abort() / error / return
0
Second try DART
9
The rest of this talk

• DART algorithm
• Program instrumentation
• Test driver
• Soundness and completeness
• DART for C
• Experimental results

10
Program instrumentation

• Two kinds of computation
• Concrete
• Regular execution
• Symbolic
• Writes expressions in terms of input parameters
• Used to calculate path constraints

11
Concrete and symbolic execution

• Two kinds of memory
• Concrete memory M maps addresses to values
• Symbolic memory S maps addresses to expressions
  (in terms of input parameters)

12
Concrete and symbolic execution

• Each program statement replaced with concrete and
  symbolic computation
• Assignments (m lt- e)
• Update M at m with evaluate_concrete(e,M)
• Update S at m with evaluate_symbolic(e,M,S)
• Conditionals (if (e) then goto l)
• Let b evaluate_concrete(e,M)
• Let c evaluate_symbolic(e,M,S)
• Append c (?c) to path constraint if b (?b)

13
Symbolic execution

• What if it gets stuck?
• Falls back on result of concrete execution
• Sets a completeness flag to signal potential
  incompleteness
• Non-linear arithmetic
• Sets all_linear to 0
• Dereference of non-constant pointer
• Sets all_locs_definite to 0

14
Test driver

• Tries to explore all execution paths
• Random search
• Directed search

15
Random search

• Initiates directed search with random input
  vector
• Loops until
• All execution paths explored
• Completeness flags
• Or bug found

16
Directed search

• Executes instrumented program
• Updates input vector to try to force program down
  unexplored paths
• Constraint solving
• Loops until
• All execution paths explored
• Or input vector fails to force program down
  desired path
• Impossible if completeness flags both 1
• Or bug found

17
Explored and unexplored paths

• Record conditionals executed in previous run
• stack
• ith entry corresponds to i1st conditional
  (constraint)
• 2 flags in each entry
• branch
• done

18
Explored and unexplored paths

• Force execution down unexplored path in next run
• Update I (input vector)
• Negate constraint for last unexplored conditional
  in stack
• Solve constraints to get I
• Update I with I, leaving I unchanged for any
  wildcard parameters in I

19
Soundness and completeness

• With respect to errors found!
• Sound
• No false positives in bug reports
• Complete (in a way)
• If test driver terminates without reporting a
  bug, no bug exists
• All paths exercised

20
Example pointer arithmetic
struct foo int I char c bar (struct foo a)
if (a-gtc 0) ((char )a
sizeof(int)) 1 if (a-gtc ! 0)
abort()

• Sound static analysis bug may exist
• Unsound static analysis (SLAM, BLAST) no bug
  found
• DART tests with a-gtc 0

21
Example non-linear constraints
foobar(int x, int y) if (xxx gt 0)
if (x gt 0 y 10) abort() else
if (x gt 0 y 20) abort()

• Static analysis using predicate abstraction
  (SLAM) both aborts may be reachable
• Test-generation using symbolic execution (Java
  PathFinder) stuck at first conditional
• DART directed search finds abort with
  probability 0.5 (x gt 0)

22
DART for C

• Issues
• Automated interface extraction
• Test driver
• Directed search

23
Automated interface extraction

• Program interface (non-deterministic)
• External variables and functions
• to compiler, undefined references
• Arguments to top-level function
• Statically parsed
• Deterministic, unknown library functions OK!
• Executed dynamically

24
Test driver

• Initialize with random values
• External variables
• Arguments to top-level function
• Stubs for external functions
• Return random values
• depth
• number of iterations of random search

25
Directed search

• Instrumentation CIL
• Parses and analyzes C code
• Constraint solving lp_solve

26
Experimental results

• Toy problem
• DART vs. random search
• Needham-Schroeder protocol
• DART vs. random and systematic searches
• oSIP
• DART vs. really buggy software?

27
Toy problem

• Program with input-filtering code
• State only changes when 0 lt input lt 3
• Obviously, DART beats random search

28
Needham-Schroeder protocol

• Public key authentication protocol
• Lowes attack
• Intruder impersonates an initiator to set up a
  false session with a responder
• Sequence of 6 message exchanges

29
Needham-Schroeder protocol

• Other state space searches failed to find attack
  (in 8 hours)
• random search
• systematic search (up to a certain depth)
• DART found Lowes attack and a bug in its fix

30
oSIP

• Open source implementation of the Session
  Initiation Protocol
• Telephony over IP
• 30,000 LOC
• 600 externally visible functions
• Largely unspecified

31
oSIP

• Found bugs in 65 of functions
• Mostly null pointer dereferences
• Checking inconsistent
• Found significant security vulnerability
• Send a very large message
• alloca(size) for message returns null
• Null pointer passed to another function and
  dereferenced
• DART vs. other algorithms?

32
DART implementations

• Version in this paper (Bell Labs)
• Not publicly available
• Cute (UIUC)
• Probably available soon
• Execution-generated tests (Stanford)
• Developed independently
• Available someday?

33
Summary

• DART automates unit testing
• Given a compilable program, automatically
• Extracts interface
• Writes test driver and stubs
• Test driver tries to explore all execution paths
• Random search
• Directed search
• DART reports only sound bugs

34
DART Directed Automated Random Testing

• Patrice Godefroid, Nils Klarlund, and Koushik Sen
• PLDI05

Presented by Jaymie Strecker Software
Chat September 16, 2005