Code Instrumentation Tools

1
Code Instrumentation Tools

• Tools that modify production code for various
  purposes during code execution
• Debuggers
• Add information to allow for break points or
  step-by-step execution.
• Coverage measurement
• Report on the number of times various code
  elements have been executed.
• Profilers
• Measure amount of time spent in various parts of
  the code.

2
Test Coverage

• Coverage can be based on
• source code
• object code
• design model
• requirements checklist
• ...

3
Coverage what to measure?

• For any coverage measure, we need
• A coverage unit an element with the properties
• We can count the total number of units in the
  software.
• We can identify which units were hit during a
  single execution run.
• This means that we can determine the percentage
  of units hit during one or more execution runs.

4
Coverage measurement

• Types of coverage
• Control-flow based based on structural elements
• Data-flow based trace data from where values
  are defined to where they are used

5
Control flow coverage

• Method coverage
• Statement coverage
• Branch coverage (also called decision coverage)
• Minimum coverage specified by the IEEE unit test
  standard
• Multiple Condition coverage
• Covers combinations of condition in decisions
• Path coverage
• 100 path coverage impossible in practice (loops)

6
Flow graph

• The flow graph on the right is determined from
  the code on the left

1
a gt 1 AND b 0
true
int proc(int a, int b, int x) if ((agt1)
(b0)) // 1 x x/a // 3 if
((a2)(xgt1)) // 4 x x1 // 5
return x // 7
false
2
x?x/a
3
4
a 2 OR xgt1
true
6
x ?x1
false
5
7
7
Statement Coverage

• Criterion All statements must be covered during
  test execution.
• This is the weakest form of coverage.
• Some branches may be missed.
• Find paths that cover all statements
• Choose input data that will result in the
  selected paths.

8
Statement Coverage

• The following path is sufficient for statement
  coverage
• 1 3 4 5 7
• Possible input
• a 2, b 0, x 4

1
a gt 1 AND b 0
true
false
2
x?x/a
3
4
a 2 OR xgt1
true
6
x ?x1
false
5
7
9
Branch Coverage

• Criterion At any branch point, each branch must
  be covered during test execution.
• The true and false branch of a 2-way if
  statement.
• Each case in a switch statement.
• Find paths that cover all branches
• Choose input data that will result in the
  selected paths.
• Branch coverage necessarily includes statement
  coverage.

10
Branch Coverage

• The following paths are sufficient for branch
  coverage
• 1 2 4 5 7
• 1 3 4 6 7
• Possible input
• a 2, b 2, x -1
• a 3, b 0, x 1

1
a gt 1 AND b 0
true
false
2
x?x/a
3
4
a 2 OR xgt1
true
6
x ?x1
false
5
7
11
Multiple Condition Coverage

• Criterion
• Every atomic (i.e. does not include AND or OR)
  condition must be true and false at some point
  during test execution.
• In a compound logical statement (i.e. includes
  AND and OR), every combination of atomic
  conditions must be covered during test execution.
• Achieving multiple condition coverage also
  satisfies statement and branch coverage

12
Multiple Condition Coverage
int proc(int a, int b, int x) if ( (agt1)
(b0) ) x x/a if ( (a2)
(xgt1) ) x x1 return x

• Need cases where
• a gt 1 is true and b 0 is true
• a gt 1 is true and b 0 is false
• a gt 1 is false and b 0 is true
• a gt 1 is false and b 0 is false
• a 2 is true and x gt 1 is true
• a 2 is true and x gt 1 is false
• a 2 is false and x gt 1 is true
• a 2 is false and x gt 1 is false

13
Multiple Condition Coverage

• Possible input
• a 2, b 0, x 2 15
• a 2, b 1, x 0 26
• a 0, b 0, x 2 37
• a 0, b 1, x 0 48

• a gt 1 is true and b 0 is true
• a gt 1 is true and b 0 is false
• a gt 1 is false and b 0 is true
• a gt 1 is false and b 0 is false
• a 2 is true and x gt 1 is true
• a 2 is true and x gt 1 is false
• a 2 is false and x gt 1 is true
• a 2 is false and x gt 1 is false

14
Multiple Condition Coverage

• Multiple condition coverage covers all branches
  and statements.
• Input values
• a 2, b 0, x 2
• a 2, b 1, x 0
• a 0, b 0, x 2
• a 0, b 1, x 0
• Paths covered
• 1 3 4 5 7
• 1 2 4 5 7
• 1 2 4 5 7
• 1 2 4 6 7

1
a gt 1 AND b 0
true
false
2
x?x/a
3
4
a 2 OR xgt1
true
6
x ?x1
false
5
7
15
All Paths Coverage

• Criterion
• All paths through the code must be covered.
• This is typically infeasible when loops are
  present.
• A version of this coverage with loops is to treat
  loops as having two paths
• The loop is executed (normally, once).
• The loop is skipped.
• Some paths may also be infeasible because there
  is no combination of data conditions that permit
  a path to be taken.

16
All Paths Coverage

• Set of all paths
• 1 2 4 6 7
• 1 3 4 6 7
• 1 2 4 5 7
• 1 3 4 5 7
• Input values
• a 0, b 1, x 0
• a 3, b 0, x 0
• a 2, b 1, x 0
• a 2, b 0, x 0

1
a gt 1 AND b 0
true
false
2
x?x/a
3
4
a 2 OR xgt1
true
6
x ?x1
false
5
7
17
Comparison

• From the previous two examples, we can see that
• Multiple condition coverage does not imply all
  paths coverage.
• All paths coverage does not imply multiple
  condition coverage.

18
Infeasible paths

• Set of paths
• 1 2 4 6 7
• 1 3 4 6 7
• 1 2 4 5 7
• 1 3 4 5 7
• To be able to take this path, we would have to
  have a lt 1 AND a gt 4 which is logically
  impossible!

1
a gt 1
true
false
2
x?x/a
3
4
a gt 4
true
6
x ?x1
false
5
7
19
Data-flow based coverage

• Coverage based on data information
• Elements
• Definition (d) A point where a variable
  received a value
• Assignment statement, left side
• Input statement (keyboard, file, network, ...)
• Input parameter passing
• Use (u) A point where a variables value is used
• Expression
• Output statement (screen, file, network, ...)
• Return values

20
Uses

• Variable uses have two sub-categories
• Predicate use (p-use)
• A use for which the current value will affect the
  control flow of the system
• Example
• Use in an expression contained within an if or
  switch statement.
• Computational use (c-use)
• A use for which the current value will not
  (immediately) affect control flow.
• Examples
• expressions on right side of assignment
• output parameter passing
• system output

21
Definition clear subpaths

• Subpath part of a flow graph that does not
  necessarily extend to both the entry and exit
  points of the flow graph.
• Definition-clear subpath for a particular
  variable value
• Trace a subpath from where a variable receives a
  value, to where that value is used
• There can be no re-definition of the value on the
  path.

22
Data flow Coverage criteria

• All definitions
• Include during execution a feasible
  definition-clear subpath from every definition to
  some use of that value.
• All uses
• For every use, include during execution at least
  one feasible definition-clear subpath from every
  definition for that use.
• All du-paths
• For every use, include during execution all
  feasible definition-clear subpaths from every
  definition for that use.

23
Flow graph
a keyboard.nextInt() b keyboard.nextInt()
1
2
3
a gt 3
a b 1
4
6
b b a a 2
5
System.out.println( a, b )
7
24
Definitions and Uses
def
a keyboard.nextInt() b keyboard.nextInt()
1
2
use
3
a gt 3
a b 1
6
4
b b a a 2
5
System.out.println( a, b )
7
25
Definition-use paths
def
a keyboard.nextInt() b keyboard.nextInt()
1
2
use
3
a gt 3
d-u path
a b 1
6
4
b b a a 2
5
System.out.println( a, b )
7
26
Definition-use subpaths
27
Coverage application

• These coverage criteria can be applied to
  anything which can be transformed to a flow
  graph
• Source code
• State machines
• Design models (UML, ...)
• Other types of coverage
• combinatorial coverage
• interaction coverage

28
Coverage tools

• Program is typically compiled with special
  options, to add extra source or object code.
• Additional data structures, such as a flow graph,
  may also be created.
• Program is run, possibly via test cases
• During execution, information is accumulated and
  written to an output file.
• Post-processing phase
• User report is generated from output file.

29
How to measure coverage?

• Possibilities
• Instrument the source code before compilation
• Instrument the virtual machine or object code.
• The Apache Byte Code Engineering Library (BCEL)
  is used by several Java coverage tools
• http//jakarta.apache.org/bcel
• Use a customized virtual machine.

30
Coverage filters

• Normally, a filtering function is provided to
  determine for which classes coverage is to be
  measured
• Include or exclude specified packages
• Filter out calls to provided class libraries or
  framework.
• Exclude certain source code statements
• Example When a Java class is not intended to
  have instances created, the default constructor
  is often set to be private. This would show up
  as uncovered code.

31
Source code instrumentation

• Create a copy of the source code, instrument the
  copy, and compile the instrumented copy to .class
  files.
• This is the approach used by Clover.
• Typical method
• Set up an array of counters.
• Insert source code at points of interest that
  increment specific counters.
• Coverage tool calls main() method
• After main() method returns, dump values of
  counters.

32
Source code before instrumentation

• public final void add( final MatrixRowltTgt
  newValues )
• if ( getNumColumns( ) 0 )
• contents newValues.copy( ).getContents(
  )
• else
• contents.addAll( newValues.copy(
  ).getContents( ) )
• A counter is inserted for each method, statement,
  and branch.
• Statement counters are before each statement, in
  case the statement throws an exception.
• Branch counters need to identify true and false
  branches taken.

33
Same code after instrumentation

• public final void add( final MatrixRowltTgt
  newValues )
• CLOVER.M348
• CLOVER.S1814
• if ( ( getNumColumns( )0
  CLOVER.CT338!0 true )
• ( CLOVER.CF338 0 false ) )
• CLOVER.S1815
• contents newValues.copy( ).getContents(
  )
• else
• CLOVER.S1816
• contents.addAll( newValues.copy(
  ).getContents( ) )

statement counters
branch counters
method counter
34
Counting branches

• Count number of times that a branch condition
  evaluates to true and false.
• if ( ( aBoolean CT338!0 true )
• ( CF338 0 false ) )
• If aBoolean is true, then the is also
  evaluated, which will increment the CT counter.
  We want the expressions overall evaluation to be
  true.
• If aBoolean is false, then execution switches to
  the right side of the clause. Increment the
  CF counter, but then ensure that the entire
  expression still evaluates to false to take the
  else branch.
• Short-circuit evaluation means that as soon as
  the expression result can be deterimined,
  evaluation stops. This avoids
• Incrementing the true counter if aBoolean is
  false.
• Incrementing the false counter if aBoolean is
  true.

35
Analysis

• Advantages
• Results are directly mapped to source code.
• If a statement throws an exception, it will be
  counted as having been executed.
• Disadvantages
• Requires keeping a second version of source code,
  and ensuring that original source is not
  over-written.
• Instrumented code must not affect original code
  execution.
• Management of .class files from compiling
  alternate source code.
• Memory usage for statement coverage alone,
  every statement requires one integer as a counter.

36
Byte code instrumentation

• Can be done in two ways
• Instrument object files in advance to produce a
  alternate version.
• Example create instrumented .class files for
  Java.
• Instrument object files as they are used.
• Example create customized .class file loader
  that inserts the instrumentation at run time.

37
Analysis

• Pre-instrumentation
• Advantages
• source code is not modified.
• can be done as a batch process before execution
• Disadvantages
• management of two .class file versions.
• mapping of source code statements to/from object
  code
• counting source code statements can be difficult
• object code may need extra information such as
  special compiler options

38
Clover

• http//www.atlassian.com/software/clover
• Commercial tool requires license
• Supports method, statement, and branch coverage
• Reports its findings in multiple formats
• From project level down to individual lines of
  source code
• Shows list of test cases that hit each line of
  course code
• Historical charting of code coverage and other
  metrics
• Plug-in versions available for Eclipse, NetBeans

39
Clover plugin for Eclipse

• Image source atlassian.com

40
Clover coverage filters

• One can choose not to instrument certain types of
  blocks in the code (e.g. assertions and exception
  catching), in order to focus on the coverage of
  interest.

41
Emma

• Open-source tool
• Supports class, method, basic block, and line
  coverage.
• Fractional line coverage supported, but not
  branch coverage.
• Standalone version works with Ant builds
• http//emma.sourceforge.net
• Eclipse plugin EclEmma also available
• http//www.eclemma.org

42
Emma

• Uses byte code instrumentation.
• Classes can be instrumented in advance, or during
  class loading.
• Tool keeps a metadata file to associate byte code
  with source code
• A regular compile will allow for only limited
  metadata.
• A compile with debug option will provide more
  information to support branch coverage.

43
Block Coverage

• Block coverage assumes that if a block of
  statements without branches is entered and
  exited, all statements in the block were
  executed.
• That is, the counter is at the end of the block,
  instead of before the source code statement.
• Result If an exception occurs in the block, the
  entire block is not recorded as having executed.
• This may be fine for application source code, but
  it does not work well with JUnit test source code
  or in code for which exceptions are commonplace.
• JUnit throws exceptions internally when tests
  fail, so the test may not have appeared to be
  executed.

44
Emma coverage report
45
Emma source code annotations
46
Fractional line coverage
Only part of conditional executed
Loop increment not executed
47
Anomalies with byte code coverage

• There are cases where the compiler may emit
  multiple copies of object code for a single
  instance of source code.
• In the above example, there is an instance
  variable initialized for each of two
  constructors. However, only one constructor is
  called. The result is the yellow annotation at
  initialization statement 17.

48
Coverlipse

• Open source Eclipse plug-in
• http//coverlipse.sourceforge.net
• Provides block coverage, all-uses data flow
  coverage
• No branch coverage
• Uses Apache BCEL to instrument bytecode
• Appears to be dormant, and not compatible with
  Java 6 or Eclipse 3.3.

49
Line coverage report
50
All-uses coverage report
51
All-uses coverage report
52
CodeCover

• Open source Eclipse plug in
• Web site http//codecover.org
• Version 1.0.0.0 available Nov. 2007
• Performs source-code instrumentation to obtain
• Statement coverage
• Branch coverage
• Loop coverage loop executed zero/once/many
  times.
• Multiple-condition coverage (called strict
  condition coverage by CodeCover)

53
CodeCover summary report
54
CodeCover detailed report
55
CodeCover features

• Boolean value analyzer shows how many Boolean
  combinations in conditions have been covered.
• Code hot spots highlighting of code that is
  executed more frequently than most.
• Test correlation matrix for each pair of test
  cases, the overlap in coverage for the two test
  cases is shown.
• Source instrumentation is done following a
  statement.
• The following statements do not appear as
  coverable items
• throw statements
• return statements
• Methods that throw exceptions are marked as not
  covered.

56
CodeCover Boolean analyzer

• For the compound condition shown, combinations of
  atomic conditions that have occurred are shown.
• The x shows a short-circuit evaluation.

57
CodeCover Correlation view

• The colours give an indication of the overlap for
  pairs of test cases.
• The selected square shows that for these two test
  cases, they have 24 and 25 coverable items, and
  that 19 are shared, for an overlap of 88.1

58
Cobertura

• Open source tool for Java code coverage
• http//cobertura.sourceforge.net
• Measures line coverage, branch coverage, and
  McCabe cyclomatic complexity metric for methods.
• Branch coverage requires compilation with debug
  option.
• Must be run as an Ant task, with four stages
• Compile source code
• Instrument classes
• Run code
• Generate report

59
Cobertura summary report
60
Cobertura detail report
61
Code Coverage Benefits

• If you havent exercised part of the code, it
  could contain a bug.
• Can be used to locate dead or unexecutable
  code.
• WARNING Be very careful when removing code, if
  the purpose is to test for some exceptional
  condition that is difficult to trigger.

62
Code Coverage Caveats

• Coverage tests that the code has been exercised,
  and not that you built what the customer wanted.
• If the logical structure of the code contains
  errors such as a missed case in a switch
  statement, the absence of code will not be
  detected.
• 100 coverage is often unrealistic, especially
  for fine-grained coverage measures
• Effort to achieve high coverage may be
  unjustified in terms of
• time spent
• producing unreliable test cases
• 80 coverage is a rule of thumb often used in
  industry

63
References

• Clover http//www.atlassian.com/software/clover
• Emma
• http//emma.sourceforge.net
• http//www.eclemma.org
• Coverlipse http//coverlipse.sourceforge.net
• CodeCover http//codecover.org
• Cobertura http//cobertura.sourceforge.net
• A. Glover, Dont be fooled by the Coverage
  Report, IBM developer works article
• http//www-128.ibm.com/developerworks/java/library
  /j-cq01316
• S. Gornett, Code Coverage Analysis
• http//www.bullseye.com/coverage.html