Improving Effectiveness of Regression Testing of Telecommunications Systems Software

1
Improving Effectiveness of Regression Testing of
Telecommunications Systems Software

• Sami Torniainen
• 19.2.2008
• Supervisor Professor Raimo Kantola

2
Outline

• Goal
• Background
• How to improve cost-effectiveness?
• Methods
• Implementation
• Evaluation
• Further research

3
Goal

• Goal improve cost-effectiveness of regression
  testing at the target company
• Three sub-objectives
• explore present regression testing practices
• implement methods
• evaluate empirically

4
Background - Target Company

• Finnish telecommunications systems developer
• Located in Espoo
• Main product IP/MPLS routers
• Customers global mobile and internet service
  providers

5
Background V-model
6
Background Regression testing

• Regression testing is selective retesting of a
  system or component to verify that modifications
  have not caused unintended effects and that the
  system or component still complies with its
  specified requirements. (IEEE)
• Can occur on any level of the V-model
• This thesis focuses on regression testing on
  integration testing level

7
How to improve cost-effectiveness?

• Costs accumulate from time and resources consumed
  in executing tests
• Two approaches mainly utilized in literature
• test selection techniques
• test automation

8
Test selection techniques

• Reduce the number of tests
• Mainly based on code exploration
• unit testing
• Also exists risk-based method
• integration testing level

9
Test automation

• Automating manual working phases
• creating executable test programs i.e. scripts
• Popular method
• Expensive to utilize

10
Implementation

• Limited scope (10 test suites)
• Risk-based test selection technique
• evaluating test suites based on risk-analysis
• executing only five test suites of ten
• Test automation
• automating the control of Adtech AX-4000
• Tcl API
• no manual intervention needed during test
  execution

11
Influence on regression test process
12
Evaluation testing effort
13
Evaluation detected errors
14
Evaluation deploying effort
15
Evaluation - Summary

• Test automation reduced testing effort
  significantly
• but was expensive to implement (94)
• Risk-based test selection method also reduced
  testing effort
• but will likely miss errors

16
Conclusions

• Despite the great implementation costs, test
  automation is recommended
• break-even only after 34 test rounds
• Benefits
• reduced testing effort
• safer
• reduce the overall testing elapsed time
• potentially earlier time to market

17
Further research

• Develop Tcl API
• reloading of GUI-based test setups
• would reduce scripting effort
• Optimizing test coverage of risk-based test
  selection technique
• threshold
• minimizing probability to miss errors Vs.
  reducing the number of test suites

18

• Thank You!
• Questions?