On the Effectiveness of the Test First Approach to Programming

1
On the Effectiveness of the Test First Approach
to Programming

• Hakan Erdogmus, Maurizio Morisio, and Marco
  Tochiano
• Presented by Orrin Matthews

2
Summary

• Describes a controlled experiment for evaluating
  a an important aspect of Test Driven Development
• Attempts to answer Is Test-First Programming
  more effective than conventional methods?
• If so, then how much does productivity increase?

3
Background

• Test Driven Development
• Programmers write code that exercise system
  components(repeatable unit tests)
• Popularized by Extreme Programming
• Usually supported by a unit testing tool, JUnit,
  NUnit, etc.

4
Background

• Test-First Programming
• Key aspect of Test Driven Development
• Programmers write low level functional tests
  before production code
• No formal criteria to select test cases
• Tests added gradually during implementation

5
Background

• Proponents View
• Provides instant feedback on the implementation
  of new functionality
• Encourages decomposition of program into
  manageable tasks
• Up-to-date tasks ensures a certain level of
  quality
• Provide context for low-level design decisions

6
Background

• Skeptics View
• Difficult to learn to see what tests are needed
• Hard to see how extra code increases productivity
• Testing is the exclusive job of a separate
  testing team

7
Structure

• Formal investigation of the strengths and
  weaknesses of Test-First Programming
• Background and Related Work
• Experimental Design
• Data Analysis
• Discussion and Theory Building
• Conclusion and Future Work

8
Previous Studies
9
Experimental Design Goals

• Compare Test-First with Test-Last Programming to
  evaluate
• External quality
• Programmer productivity
• In the context of
• Incremental development and
• Undergraduate object-oriented programming course

10
Experimental Design Hypotheses

• (1T) Test-First Programmers write more tests per
  unit of programming effort
• (1Q) Test-First Programmers produce higher
  quality programs
• (1P) Test-First Programmers are more productive
  over all

11
Experimental Design Hypotheses

• (2Q) Writing more tests improves quality
• (2P) Writing more tests increases productivity

12
Experimental Design Hypotheses
13
Experimental Design Experiment Design

• Two groups, Test-First and Test-Last
• Stratified by skill level and assigned randomly
  to one of the two groups

14
Experimental Design Subjects

• Third year students in 8 week intensive Java
  course
• Taught Java, basic design concepts, unit testing
  with JUnit
• Final Exam - Hands on programming task in the lab

15
Experimental Design Experiment Task

• Bowling Score Keeper problem
• Organized into small stories describing a feature
  of the end product
• Each story externally testable
• 105 black-box acceptance tests written by the
  authors
• Subjects wrote their own unit tests with JUnit

16
Experimental Design Procedure

• Subjects receive instruction on low-level
  functional testing
• Taught a specific pattern for Junit testing at
  the command line
• Subjects trained in either Test-First or
  Test-Last methodology
• Subjects implemented stories incrementally over
  several lab periods

17
Data Analysis

• (1T) TFs write more tests per unit of
  programming effort
• Effect is small but significant
• (1Q)TFs produce higher quality programs
• No significant difference in quality
• However, baseline quality is higher

18
Data Analysis
19
Data Analysis

• (2P) Writing more tests increases productivity
• The number of tests can predict productivity
  independently of the group

20
Data Analysis
21
Conclusions

• Quality
• Students who wrote few tests achieved varying
  degrees of quality depending on skill level
• If this holds, then skill becomes less of a
  factor as the number of tests increase

22
Conclusions

• Productivity
• Incremental development allows for demonstrating
  real productivity from Test-First
• Better task understanding
• Better task focus
• Faster Learning
• Lower rework effort

23
Questions

• Given the parameters of the experiment, how could
  the authors have compensated for the effects of
  skill on quality in this experiment?
• What do you think are some real world
  implications of the results?
• Why do you think that in previous experiments
  involving professionals instead of students, the
  effect of TDD on productivity ranges from no
  difference to detrimental?

24
Conclusions

• Test-First programmers did not have better
  quality on average. They did, however achieve
  more consistent quality.The first observation is
  attributable to the influence of skill on
  quality.
• Writing more tests improved the minimum quality
  achievable and decreased the variation.