Efficient Instrumentation for Code Coverage Testing

1
Efficient Instrumentation forCode Coverage
Testing

• Mustafa M. Tikir
• Jeffrey K. Hollingsworth

2
Evaluation of Code Coverage

• Measuring code coverage is important to
• Identify unexecuted program statements
• Verify that each path is taken at least once
• Requires extensive instrumentation
• Must determine if every statement is executed

3
Instrumentation for Code Coverage

• Traditional Approach
• Static instrumentation using counters
• Instrumentation code remains for entire execution
• Conservative instrumentation of all possibly
  needed instrumentation code for all functions
• Useless instrumentation wastes time
• especially for long running programs such as
  servers and enterprise software
• Full instrumentation increases setup time

4
Our Approach

• Insert instrumentation code dynamically
• Pre-Instrument all functions at program start
• On-Demand instrument at first function call
• Periodically remove instrumentation
• Use dominator trees
• Reduces instrumentation within a function
• Use incremental function instrumentation
• Insertion of instrumentation on first call
• Reduced which function gets instrumented

5
Dyninst API

• Modify code in a running program
• Implementations available for
• Alpha,Sparc,Power,Mips and x86 architectures
• A mutator program
• Generates machine code from high-level code
• Transfers machine code to a running mutatee
  program

Base Tramp
Mini Tramp
Program

• Mutatee is the application being instrumented
• Base-trampoline contains
• relocated instructions
• slots for calling code
• Mini-trampoline stores the inserted code

Save Registers
Pre
Setup Args
Relocated Instruction
Code Snippet
Restore Registers
Post
6
Using Dominator Trees

• Definitions
• A dom B if all paths from entry to basic block B
  goes through basic block A
• A idom B if, for all C, (C ! A) and (C dom B)
  implies (C dom A)
• Fact
• If a basic block, B, is executed
• all basic blocks along the path from B to the
  root of dominator tree also execute

7
Leaf Node Instrumentation

• Leaf nodes in dominator tree are instrumented
• Coverage of internal nodes will be inferred
• Coverage information propagated from leaf nodes
  to entry

Dominator Tree
Control Flow Graph
Entry
Entry
1
1
3
2
3
2
5
4
4
5
Exit
Exit
8
Non-Leaf Node Instrumentation

• Leaf node instrumentation is necessary but not
  sufficient
• Control flow might cause cross edges in dominator
  tree

Control Flow Graph
Dominator Tree
Entry
Entry
2
1
1
3
2
3
Exit
Exit

• We also instrument basic block, A, if
• A has at least one outgoing edge to basic block,
  B, and
• A does not dominate B

9
Code Coverage Algorithm
Pre-Instrumentation
On-Demand Instrumentation

• At Start
• Create CFG and dominator trees for all functions
• Instrument the basic blocks selected
• During Execution
• Stop the execution at fixed time intervals
• Delete the executed instrumentation code
• At Termination
• Propagate and collect the coverage information

• At Start
• Insert breakpoints at each function entry
• During Execution
• On breakpoint
• Identify the function
• Create CFG and dominator tree
• Instrument the basic blocks selected
• At fixed time intervals delete the executed code
• At Termination
• Propagate and collect the coverage information

10
Reduction In Instrumentation Points

• 34-49 with pre-instrumentation
• 42-79 with on-demand instrumentation

11
SPEC/compress Coverage Curve

• Covers 76 in the first 18 of the execution
• Most of the basic blocks that will execute are
  covered at the beginning of the program
• Rest of the run is re-executions

12
SPEC/compress Execution Time

• Purecov is a commercial state of art code
  coverage tool
• Our code coverage tools outperform purecov
• Significant reduction when dynamic code deletion
  is enabled
• Most of the instrumentation code is deleted at
  the beginning
• Setup time and deletion overhead is insignificant
• A few number of basic blocks to instrument and
  check

13
PostgreSQL Coverage Curve

• Wisconsin benchmark queries
• Measure the performance of database systems
• Executes select/join queries repeatedly

14
PostgreSQL Execution Time
Execution Time
Setup, Instrumentation and Deletion Time

• Using on-demand instrumentation
• Our coverage tools outperform purecov almost
  always
• On-demand instrumentation outperforms
  pre-instrumentation

15
Instrumentation Execution Frequency

• Overhead of our dynamic code coverage system is
  bursty
• Running previously unexecuted code results in the
  execution of a significant amount of
  instrumentation
• Running previously executed code does not result
  in the execution of any instrumentation

16
Overall Slowdown

• Results for our code coverage tool are for
  2-second deletion
• Slowdown using purecov ranges from 1.83 to 19.78
• Slowdown using our code coverage tools range
• From 1.002 to 2.6 for on-demand instrumentation
• From 1.002 to 4.96 for pre-instrumentation

17
Dyninst Coverage Tool
18
Conclusions

• Dominator trees
• Reduce instrumentation points by 34-49
• Plus on-demand instrumentation reduce
  instrumentation points by 42-79
• Combining dominator trees and on-demand
  instrumentation reduces
• Setup time and deletion interval overhead
• Runtime overhead by 38-90 compared to purecov
• Dynamic deletion of instrumentation
• Computes coverage information faster

19
Conclusions (cont)

• Code Coverage overhead reduced to about 10
• Code coverage can now be included as part of
  production code
• Information about the execution of extremely
  infrequent error cases will be provided
• Reduced overhead for residual testing
• Dyninst library Dyninst Coverage tools
• http//www.dyninst.org