Studying the test process in open source and industrial software systems

1
Studying the test process in open source and
industrial software systems

• Andy Zaidman, Bart Van Rompaey,
• Serge Demeyer, Arie van Deursen
• TU Delft, University of Antwerp
• Andy Zaidman, IPA Herfstdagen 24th November 2008

2
How does test code co-evolve?

• WHY?

HOW?
() Developer tests
3
Well, why would you

• want to know the quality of the tests?
• When maintaining the system
• ? can you trust the existing tests/test process?
• When judging the overall quality of a software
  system
• ? is it good enough for our standards?
• When monitoring the test process
• ? Comparing intended with actual process

4
You might say

• What about test coverage?
• Nice, but somewhat hollow measure
• Easy to keep artificially high
• What about boundary testing?
• Exercising versus testing
• Test process?

5
How?
6
Repository mining

• Software developers like to use VCS
• e.g., CVS, SVN, Clearcase, MS Visual Sourcesafe,
  
• Backing up their work
• Making it easy to work in groups
• Easy to manage releases
• On the other hand
• A wealth of information is stored in there
• Its like a free museum of past development
  activities and development decisions
• Lets use this!

7
Test health

• Determine the long-term quality of your test
  suite.
• Dont only look at the testing product, but
  also at the testing process
• Tells something about the future of the testing
  (e.g., maintenance of the tests)
• Test health was introduced by Kent Beck

8
Test health VCS views

• Change History View
• When are tests adapted?
• Growth History View
• Determine impact of changes to production and
  test code.
• Test Writing Efficiency View
• How much test code for coverage?

9
3 subject software systems

• Checkstyle
• Java source-code style checker (and improver)
• 6 years development, 2260 commits, 6
  developers, 738 classes, 47 kSLOC
• ArgoUML
• UML drawing application
• 7 years development, 7477 commits, 42
  developers, 1533 classes, 130 kSLOC
• 1 industrial system
• Software Improvement Group (SIG), Amsterdam, The
  Netherlands

10
Example change history view
11
Checkstyle Change History
12
SIG Change History
CS
13
Checkstyle Growth History

14
ArgoUML growth history view
15
SIG Growth History
16
Growth History co-evolution scenarios
17
Checkstyle test writing efficiency
18
SIG test writing efficiency
19
Test writing efficiency patterns
20
Conclusion?

• In our case studies
• Big difference between OS systems and SIG!
• SIG
• Continuous testing effort
• eXtreme Programming test-driven development
• OS
• Sometimes 6 months of no testing
• No increase in testing before major release
• But sometimes after! Buy 2.1 instead of 2.0!!!
• ArgoUML project was started without testing
  process in place

21
Next steps in this research

• Try this on more (diverse) software systems
• e.g., OS projects from Apache group
• Try to correlate the testing effort to the number
  of bugs reported in Bugzilla
• More bugs when testing effort is less good, and
  vice versa?
• More information?
• See our ICST 2008 paper (its included in your
  IPA Fall Days welcome pack!)
• An STVR journal paper is currently in submission

22
?a.e.zaidman_at_tudelft.nlhttp//www.st.ewi.tude
lft.nl/zaidman