Evaluating the costeffectiveness of combining code inspection with static analysis tools

1
Evaluating the cost-effectiveness of combining
code inspection with static analysis tools
Margaret Wojcicki University of Queensland
2
Overview

• Verification and Validation Issues
• Empirical Evaluation
• What do we know so far?
• Comparing Tools/Techniques
• Combining Tools/Techniques
• Pilot Study
• Automated Static Analysis Tools Code Inspection
• Design and Analysis
• Replicated Studies

3
Verification and Validation Issues

• How does a practitioner decide on the best course
  of action when performing validation and
  verification?
• How many tools or techniques are being used for
  this purpose?
• How many are necessary?
• How do you know what is the best approach?

4
Evaluating Verification and Validation

• Cost-effectiveness
• Cost of learning/using the tool
• Benefits in terms of defects detected
• Is this new tool or technique more cost-effective
  than the one we used previously?
• Which testing technique is the most
  cost-effective?

5
Is this new technique or tool more
cost-effective?

• Comparing code inspection techniques against each
  other (Skoglund and Kjellgren 2004)
  object-oriented program inspection
• Comparing tool-based to paper-based software
  inspection (MacDonald and Miller 1998)
• Dealing with unexpected results (and unexpected
  information)

6
Which testing technique is most cost-effective?

• Hetzel, Myers, Basili Selby, Kamsties Lott,
  Wood et al. replicated studies of comparing code
  inspection, structural and functional testing
• Found that there was no clear best approach,
  but that techniques found different faults and
  recommended combining them

7
Evaluating combined VV techniques

• Code Inspection, Functional and Structural
  Testing, Selby and Wood et al.
• Found that combining testing techniques was more
  effective than the techniques by themselves

8
My focus Evaluating Combinations of VV Tools
and Techniques

• What combination of testing techniques and tools
  is most cost-effective?
• How can one be sure that a particular combination
  of techniques and/or tools is cost-effective?
• Analysis
• Empirical Evaluation

9
Why evaluate empirically?

• Intuition tells us it is a good idea to
  complement certain verification and validation
  tools and techniques
• This needs to be proven empirically so we can
  have confidence in this good idea evidence!
• Helps us see the interaction of techniques in
  action

10
Empirical Evaluation First Steps

• TestCon proposed a method that combines
  verification and validation techniques for Java
  concurrent components
• Pilot study focuses on its combination of
• Code Inspection
• Static Analysis Tools

11
Concurrency Issues

• Focus is on detecting these types of defects
• Deadlock
• Interference
• Static analysis tools that deal with these types
  of defects (and non-concurrency defects as well)
• Jlint (Deadlock)
• FindBugs (Interference)

12
Static Analysis Tools

• Inspect the code automatically for possible
  defects
• Defect types single threaded correctness,
  multithreaded correctness, performance issue,
  style etc.

13
Advantages of Static Analysis Tools

• Automates the process of looking through code for
  defects
• Can speed up code inspection process
• Can make the process of finding bugs more
  effective objectivity

14
Disadvantages of Running Static Analysis Tools

• Defects can be missed
• False positives can be reported by the tool

15
Example FindBugs and Inconsistent
Synchronization Defects

• Result from running FindBugs
• Why does it need to be inspected?
• What can be missed by the tool?

16
FindBugs Why do bug reports have to be
inspected? (false positives)

• Inconsistent synchronization bugs may be reported
  erroneously
• Only a thread which reads the shared variable
  will have access to this method

• public class Buffer
• private int count 0
• public synchronized void put(Object o)
• public synchronized Object get..
• public int getCount
• return count

17
FindBugs Why do bug reports have to be
inspected? (missed defects)

• public class Buffer
• private int count 0
• public synchronized void put(Object o)
• public Object get()..
• public synchronized int getCount
• return count

• FindBugs does not report any inconsistent
  synchronization defects
• The method get() should be synchronized
• Interference is possible to the count variable

18
Combining Static Analysis Tools with Code
Inspection

• Static analysis tools can complement code
  inspection by speeding up the process
• The results of the tools have to be examined in
  order to determine if the bugs reported are false
  positives and if bugs are missed
• Conclusion These approaches can complement each
  other

19
How do you evaluate VV combination empirically?

• Applying the scientific method a.k.a. controlled
  experiments
• Difficulties (costs, complexity, generalisation)
• Advantages (fewer variables affect results)
• Other options case studies, surveys can be more
  realistic

20
Empirical Evaluation Starting Out

• Goals and metrics GQM (V. Basili)
• Hypotheses are possible answers to the questions
  (null hypothesis and alternative hypothesis)

21
Coming up with an appropriate design

• How similar are the testing approaches? (control)
• How much variability is there among subjects?
  (within-subject, blocking)
• Number of projects, number of subjects, number of
  testing methods compared
• Sources Basics of Software Engineering
  Experimentation (Juristo and Moreno),
  Experimentation in Software Engineering (Wohlin
  et al.)

22
Experimental Design for Pilot Study
23
Empirical Evaluation Analysing Results

• Statistical analysis
• Hypothesis testing
• Correlations
• Detailed analysis ethnographic study
• Practical significance
• Replication How cost-effective will the approach
  be when you combine it with other
  tools/techniques?

24
Replicated Studies The Complete TestCon
Approach

• TestCon also includes functional testing through
  the use of a tool (ConAn)
• It has the human component of coming up with test
  cases and automates the process of running the
  test cases
• Is this combination cost-effective? Is the whole
  method cost-effective?

25
Replicated StudiesMore possibilities

• Race detection tools RaceFinder
• Instrumentation ConTest
• Model checking Java PathFinder

26
How and why do you need to replicate?

• Family of studies (Basili et al.)
• Framework for experimentation (Rothermel et al.
  infrastructure)
• Lab packages
• Meta-analysis (Kitchenham et al.)
• Realistic contexts generalise results, have the
  ability to apply it with confidence

27
Replicated Studies Industrial Context

• An opportunity for academic and industry
  cooperation
• C checkers FlexeLint and Illuma in large
  system at Nortel Networks
• Efficient predictors of field failures and
  effective for identification of fault-prone
  modules
• Source Nagappan et al. Preliminary Results on
  Using Static Analysis Tools for Software
  Inspection

28
Questions and Answers