Software Testing in the CS Curriculum -- A Holistic Approach

1
Software Testing in the CS Curriculum -- A
Holistic Approach
4th Australasian Computing Education
Conference December 4-6, 2000 Edward L.
Jones CIS Department Florida AM
University Tallahassee, FL USA
2
Outline

• In 25 Words or Less
• The Holistic Approach
• The SPRAE Testing Framework
• The Software Test Lab
• The Software Testing Course
• Automated Program Grading
• Synergy / Future Work

3
In 25 Words or Less

• Testing is important to industry, but missing
  from curriculum
• Only a few basic principles
• 3-Pronged Attack
• Teach
• Practice
• Mentor
• Encouraging -- a work in progress

4
What Makes It Holistic?

• Testing an integral part of curriculum
• Goal is multiple test experiences
• At least one experience in each course
• Repetition and reinforcement
• Accumulation of experiences
• Coverage of test lifecycle

5
The Holistic Approach
SPRAE Testing Framework
Software Test Lab
Elective Testing Course
Core Curriculum
Testing In Action (Automated Grading)
6
The SPRAE Framework
Specification the basis for testing
Premeditation a systematic process
Repeatability tester independence
Accountability documented process, results
Economy cost-effective practices
7
SPRAE Test Life Cycle
Specification
Test Plan
Test Cases
Test Script, Data, Driver
Test Results, Log
Defect Data Problem Reports
8
The Software TestLab
Vison

• Environment for discovery learning
• An evolving laboratory
• Tools Tutorials
• Staff (students, faculty)
• Test problem/solution test bed
• Students participate in the evolution
• Feedback on lab resources
• Create/Refine resources
• Technology insertion into classroom

9
The Big Picture
10
TestLab Infrastructure
Status

• SPRAE Framework / Lifecycle
• Software Testing Course
• Training Sequence
• Standards
• Techniques
• Problem Solution Testbeds

11
Student Mentorship Model
Mentorship

• Manage skill development
• Set clear achievement goals
• Key Practices x Competency Levels
• Certify levels of progression
• Enable student-student mentoring
• Establish recognition program

12
Training Sequence
Infrastructure

• TestLab Environment Basic Training
• Black-Box Function (unit) Testing
• Black-Box Object Testing
• White-Box Function Testing
• Clear-Box Object Testing

13
Standard Products
Infrastructure

• Specification (narrative, semiformal)
• Decision Table
• Test Plan
• Test Script
• Test Driver
• Test Driver Input File
• Test Results File
• Test Log

14
Techniques
Infrastructure

• Functional Equivalence Partitioning
• Boundary Value Analysis
• Function Encapsulation
• Control Flow Analysis
• Error-seeding for tester certification

15
Problems Testbed
Infrastructure

• Repository of testing problems
• Repository of students test artifacts
• Best in class promoted ? solutions testbed
• Deficient solutions ? almost testbed
• Almost testbed used for tester certification

16
Training Sequence (1)
Infrastructure

• TestLab Environment Basic Training
• Unix basics
• C/language refresher
• Encapsulation of function under test
• Repositories

17
Training Sequence (2)
Infrastructure

• Black-Box Function (unit) Testing
• Specification
• Stimulus-response test cases
• Function Test driver (5 styles)
• Test driver input/results files
• Test script (set-up perform)
• Test log
• Decision tables
• Functional (partition) test case design
• Boundary test case design

18
Training Sequence (3)
Infrastructure

• Black-Box Object Testing
• Specification of objects methods
• Analysis of methods stimulus-response
• Test planning
• Test cases method stimulus response
• Object Test driver
• Object Test driver input/results files
• Operational test scenarios

19
Training Sequence (4)
Infrastructure

• White-Box Function Testing
• Control flow graph basics
• Coverage criteria/measures
• Instrumentation for data collection
• Use of in-house coverage tools
• Coverage analysis
• White-box coverage during black-box test
• Supplemental white-box test cases

20
Training Sequence (5)
Infrastructure

• Clear-Box Object Testing
• Goal is to overcome information hiding
• Test windows into internal object state
• set_state ( )
• get_state ( )
• Test case
• Stimulus precondition method-stimulus
• Response method-response postcondition
• Clear-box object test driver
• Clear-box test oracles

21
Key Practices
Mentorship

• Practitioner - performs defined test.
• Builder - constructs test machinery
• Designer - designs test cases.
• Analyst - determines test needs, strategy.
• Inspector - verifies correct process, results.
• Environmentalist - establishes maintains
  the test environment.
• Specialist - performs entire test life cycle.

22
Competency Levels
Mentorship
23
Test Specialist I
Mentorship

• Practitioner I - Run function test script
  document test results.
• Builder I - Develop function test driver.
• Designer I - Develop functional and boundary
  test cases from decision table.
• Analyst I - Develop decision table from
  functional specification.
• Inspector I - Verify adherence to function test
  process and standards.

24
The Software Testing Course

• 80 practice, 20 theory
• 2/3 fine-grained testing (functions)
• 1/3 object and application testing
• Test cases, drivers, and scripts
• Decision tables the "formalism" of choice
• Functional, boundary, white-box test cases
• Evaluation via coverage error seeding

25
Course Outline

• 1 Course Overview
• 2 Software Quality Assurance
• 3 The Practice of Testing
• 4 Specification-Driven Testing
• 5 Boundary Testing
• 6 Measuring Test Effectiveness
• 7 Testing Object-Oriented Software
• 8 Application Testing
• 9 Course Review Wrap-Up

26
Course -- Lessons Learned

• Advantages
• In-depth, continuous concept treatment
• Complement to analytical programming skills
• Generic framework for test practice
• Deficiencies
• Not a mainstream course available to all
• Students compartmentalize course content

27
Automated Program Grading Life Cycle
28
Testing in Action -- Automated Program Grading

• Student shock outrage at exactness
• Specification must be better than average!
• Front-loaded Assure testability up front
• Costly automation amortized via similar
  assignment styles
• Students need the grader before submitting

29
Automation Challenges

• Does the teacher have the time?
• Is grader too strict for CS1/CS2?
• Table-driven checker scripts to reduce costliest
  step.
• The just a little more automation trap!!
• Sell colleagues on the idea!
• The guts to do it!!

30
Future??

• Evolve TestLab Mentorship Model
• Experience Ladder Certification
• Evolving Problem/Solution Artifacts
• Select Courses for Techology Transfer
• Institutionalize Automated Grading
• Testing in action -- caught, not taught
• Disseminate Results
• Web site, conference publications

31
Questions?
Questions?
Questions?
32
Thank You