Test-Driven Learning in Early Programming Courses

1
Test-Driven Learning in Early Programming Courses

• David Janzen
• California Polytechnic State University
• San Luis Obispo, CA
• Hossein Saiedian
• University of Kansas
• Lawrence, KS

SIGCSE08
2
I Have a Dream

• That all students will some day
• learn to design and test as they learn to program
• ... we have as many testers as we have
  developers. And testers spend all their time
  testing, and developers spend half their time
  testing.
• Bill Gates in Information Week, May 2002

SIGCSE08
3
How to Accomplish the Dream?

• Test-Driven Development
• TDD is a design (and testing) approach involving
  short, rapid iterations of

Refactor
Unit Test
Code
Forces programmer to consider use of a method
before implementation of the method
SIGCSE08
4
TDD in Action
class RectangleTest _at_Test public void
testArea() Rectangle r new
Rectangle(5,8) assertEquals(40,
r.getArea())
class Rectangle private length, width public
Rectangle(int l,int w) length l width
w public int getArea() return length
width
_at_Test public void testPerim() Rectangle r
new Rectangle(5,8) assertEquals(26,r.ge
tPerim())
public int getPerim() return (length 2)
(width 2)
SIGCSE08
5
Adoption Evolution
Predicting the Future

• Objects

TDD
Research Labs Dahl Nygaard
NASA Project Mercury
Industry Smalltalk, C
Industry Any SW Process
Industry XP
beyond the visionary phase and into the early
mainstream IEEE Software 2007
Grad Advanced UG
Grad Advanced UG
First Year CS
First Year CS
SIGCSE08
6
Questions We Want To Answer

1. Can we teach automated unit testing in first year
   programming courses without sacrificing other
   first year topics?
2. Is it better to teach test-first (TDD) or
   test-last programming to first year students?
3. What do students think about TDD?

SIGCSE08
7
How Do We Teach TDD?

• Test-Driven Learning involves teaching testing by
  example

XXXXXXXXXXXXX
System.out.println(Result
myObject.calculateResult())
assertEquals(25, myObject.calculateResult())
SIGCSE08
8
Experiment Design

• Proposed TDD/TDL integration into CS1 and CS2
  courses at University of Kansas in 2006 as part
  of PhD research
• We dont always get what we ask for
• Given permission to teach two labs and assign two
  projects
• Weeks 6 and 7 in a 16-week CS1 course
• Weeks 1 and 3 in a 16-week CS2 course

SIGCSE08
9
Experiment Design
CS1 Students Assigned by ID
Project 1 solution with tests provided
TF/TL Instruction Survey
Project 1
Project 2
Survey
40
40
Test-Last
Test-First
CS1
66
66
Test-Last
Test-First
6
Test-First
CS2
30
Test-Last
CS2 Students Self-selected
Project 1 solution with tests provided
SIGCSE08
10
Experiment Context

• CS1 and CS2 both used C, vi/emacs, and g

Project Topics 1st Project 2nd Project
CS1 Arrays, functions, text I/O Classes, text I/O
CS2 Classes, array-based list, text I/O Classes, pointer-based list, text I/O
SIGCSE08
11
Goal Keep Unit Testing Simple
include ltcassertgt class Exams . . . int
main() run_tests() void run_tests()
//test 1 Minimum of empty list is 0
Exams exam1 assert(exam1.getMin() 0)
//test 1 //test 2 Exams
exam1 exam1.addExam(90)
assert(exam1.getMin() 90) //test 2

SIGCSE08
12
CS1 Results Grades Effort

• Project grades were virtually the same
• Time spent on project was 10 higher for
  test-first programmers

Time (min) 1st Project 2nd Project
TF mean 285.29 343.48
TL mean 260.20 308.40
diff 10 11
p-value .60 .47
SIGCSE08
13
CS1 Results Unit Tests

• Test-first programmers wrote more tests on the
  first project, but fewer on second
• Remember that students switched approaches

asserts 1st Project 2nd Project
TF mean 5.85 1.89
TL mean 3.85 3.10
diff 52 -39
p-value .11 .11
SIGCSE08
14
CS1 Results Confidence

• No difference in confidence on first project
• On second project, test-first programmers were
  more confident of their solutions (P1 tests)

2nd Project Quality Changes Reuse
TF mean 3.98 3.90 3.69
TL mean 3.25 3.06 2.88
diff 22 27 28
p-value .05 .02 .02
SIGCSE08
15
CS1 Results Student Opinions
SIGCSE08
16
CS2 Results Grades

• Project grades were higher for test-first
• No significant difference in previous experience
  or gpa

Score 1st Project 2nd Project
TF mean 88.83 90.17
TL mean 79.47 72.83
diff 12 24
p-value .09 .04
SIGCSE08
17
CS2 Results Effort

• Time spent on project was lower for test-first
  programmers

Score 1st Project 2nd Project
TF mean 12.42 16.00
TL mean 14.70 18.06
diff -16 -11
p-value .30 .42
SIGCSE08
18
CS2 Results Unit Tests

• Test-first programmers wrote more tests
• Not significant, high standard deviation
• Students not graded on tests

asserts 1st Project 2nd Project
TF mean 53.67 47.67
TL mean 13.43 25.03
diff 300 90
p-value .08 .39
SIGCSE08
19
CS2 Results Confidence

• No significant difference in confidence on either
  project

Both Projects Quality Changes Reuse
TF mean 3.83 4.00 3.67
TL mean 3.21 3.89 3.71
diff 16 3 -1
p-value .14 .87 .74
SIGCSE08
20
CS2 Results Student Opinions
SIGCSE08
21
11 vs 63 would choose test-first
Reluctance to adopt test-first despite perceived
benefits
SIGCSE08
22
Validity Concerns

• Study too small to draw any conclusions
• Lack of randomization in CS2
• Confounding factors
• New concepts introduced with each project
• Solutions with tests provided after 1st project
• Simple assert-based unit tests, no framework
• C
• However, many of the results line up with results
  from industry and more advanced courses

SIGCSE08
23
Conclusions

• Unit testing can be integrated into CS1/2 courses
  without taking time away from other topics
• Test-first programmers tend to write more tests
• Test-first programmers may do as well (CS1) or
  better (CS2) on projects
• Test-first programmers may spend more time (CS1)
  or perhaps less time (CS2) on projects
• Early programmers are reluctant to use
  test-first, even after positive experiences with
  it

SIGCSE08
24
Future Work

• Currently conducting a study in CS1.5
• Java and JUnit
• TDD from beginning
• TDD throughout 10-week quarter
• Early Results
• TDD takes no extra instruction time
• Existing course materials can be rewritten w/TDD
• TDD could influence course refactoring
• e.g. delay coverage of I/O

SIGCSE08
25
Questions

• Resources at http//www.simexusa.com/tdl/
• If you are an early adopter of TDD in CS1/2, Id
  like to know, email me at
• djanzen (at) calpoly.edu
• Acknowledgements
• SIGCSE Special Projects Grant seed funding
• Lockheed Martin current (future work) study

SIGCSE08