A Survey of Static Software Analysis Tools

1
A Survey of Static Software Analysis Tools

• Mark Smith

2
Contents

• Overview
• Static Analysis Tools
• FindBugs
• PMD
• IntelliJ IDEA
• Benchmarks
• Results
• Conclusion

3
Overview

• Goals
• Derive comparisons and conduct an informative
  survey of three static analysis tools FindBugs,
  PMD, IntelliJ IDEA.
• To provide some guidance as to
• how the tools perform differently
• why they are different
• what techniques they use.
• Tool were selected due to
• ease of implementation
• well-known in the field of bug-finding tools.

4
Static Analysis Tools

• FindBugs
• Based on the concept of bug patterns (code idiom
  that is often an error). 
• Bug patterns arise for a variety of reasons
• Difficult language features
• Misunderstood API methods
• Misunderstood invariants when code is modified
  during maintenance
• Typos, use of the wrong boolean operator
• Uses ad-hoc techniques designed to balance
  precision, efficiency, and usability.
• One of its main techniques - syntactically match
  source code to known suspicious programming
  practice.
• Uses static analysis to inspect Java bytecode
  (compiled class files)

5
Static Analysis Tools

• PMD
• Scans Java source code and looks for potential
  problems such as
• Empty try/catch/finally/switch blocks,
• Unused local variables, parameters and private
  methods,
• Empty if/while statements,
• Overcomplicated expressions
• Duplicate code
• Performs syntactic checks on program source code,
  but does not have a dataflow component.
• Known to detect clearly erroneous code - many of
  the bugs are stylistic conventions whose
  violation might be suspicious under some
  circumstances.
• Inspects Java code using a rules-based approach.

6
Static Analysis Tools

• IntelliJ IDEA
• Commercial, cross-platform Java IDE by JetBrains
• Focuses on developer productivity and advanced
  JavaScript code editing support
• Design is centered around programmer
  productivity
• a number of features that speed up development
  time and allow programmers to concentrate on
  functionality while IDEA handles more mundane
  coding tasks.
• Associated with the Eclipse development tool

7
Benchmarks

• Photo Album Applet 1.11 (9th Oct 1999)
• Axis 1.4
• Jena 2.5.5
• Javad
• ProProxy
• Sample Test files

8
Results

• No clear winner
• A wide range of bugs were reported among the
  tools
• Rare that the same reported bugs were caught
  among a pair of tools
• Different categories and their contents are not
  always clear
• Would require more tests and time to determine
  which categories are closely related across the
  tools.
• All typically produced a large number of warnings

9
Results (cont.)

• Assumed that if one tool issued many bugs and/or
  warnings, might be likely that the other tools do
  the same as well.
• Not true (in general). It appears there is little
  correlation of warning and bug counts between the
  three tools.
• Warnings and bug counts were loosely correlated
  with lines of code.
• This was somewhat expected
• Jena and Axis 1.4 benchmarks contained by far the
  most source files and bytecode to be analyzed,
  and in turn their tests results returned the
  highest count of bugs and warnings

10
Conclusion

• Intent was to derive comparisons and conduct an
  informative survey of three static analysis
  tools FindBugs, PMD, IntelliJ.
• three of several widely-used Java bug-finding
  tools
• Six benchmarks consisting of java source files
  and bytecode were chosen
• No clear winner among the three tools.
• Rare that the same reported bugs were caught
  among any pairing of the tools.
• Assumed that if one tool issued many bugs and/or
  warnings then it might be likely that the other
  tools do the same as well.
• Little correlation of warning and bug counts
  between the three tools.