Risk-based regression testing in a telecommunication system node

1
Risk-based regression testing in a
telecommunication system node

• Masters thesis presentation 14.8.2007
• Author Mika Lehto
• Supervisor Professor Heikki Hämmäinen
• Instructor M.Sc. Pekka Hämäläinen

2
Agenda

• Background and objectives
• Risk-based testing
• Regression testing
• Case Study
• Ericssons Media Gateway for Mobile Networks
  (M-MGw)
• M-MGw Node Functional Verification
• Risk model
• Risk analysis and prioritization
• Scenarios for the case study
• Results
• Conclusions
• Future research

3
Background

• The costs of regression testing comprise 20-30
  of the whole testing costs on average in software
  development
• ? regression testing is often the target
  of cost reduction activities in order to
  achieve shorter time-to-market and
  lower cost structure of the product
• The efficiency of regression testing is usually
  attempted to improve by using regression testing
  methods such as regression test selection, test
  suite reduction, and test case prioritization
• One approach that has been studied is called
  risk-based regression testing but only a little
  information about the feasibility of risk-based
  regression testing is currently available

4
Objectives

• The primary objective of this thesis was to study
  whether it would have been beneficial to use
  risk-based regression testing in an old M-MGw
  Node Functional Verfication project
• This thesis also analyzed the differences in test
  case selections between previously used and
  risk-based test case selections methods
• The aim of this thesis was also to demonstrate
  how risk-based testing approach can be utilized
  in regression testing

5
Risk-based testing

• The fundamental difference between traditional
  testing and risk-based testing is that risk-based
  testing brings formal risk assessment methods
  (risk identification, risk analysis, and risk
  prioritization) in testing process
• The aim of risk-based testing is to focus the
  test effort on the most critical areas of the
  system
• Risk-based testing approach can be used for
  instance to
• Select the most important test objects and test
  cases
• Evaluate the current level of system quality and
  risk
• Provide input for the decision to continue or
  stop testing

6
Regression testing

• Selective retesting of a system or component to
  verify that modifications have not caused
  unintended effects and that the system or
  component still complies with its specified
  requirements (IEEE Software glossary)
• Regression testing is typically used during
  iterative software development and to support
  application maintenance
• The biggest challenge in regression testing is
  the test case selection
• In risk-based regression testing, formal risk
  analysis methods are used to select the most
  risky test cases for execution

7
Ericssons Media Gateway for Mobile Networks
(M-MGw)

• M-MGw operates in the connectivity layer of the
  core network and its fundamental purpose is to
  connect the core network to other networks such
  as UTRAN and PSTN
• M-MGw also supports media conversion, bearer
  control and payload processing
• M-MGw is remotely controlled by the MSC or SGSN
  Servers which operate in the network control
  layer.

8
M-MGw Node Functional Verification

• Testing in the NFV is based on requirements (e.g.
  Specifications and call flows)
• NFV is a testing phase in which the M-MGw
  software is first time tested in a real node
  environment
• Test environment is established of one M-MGw node
  and all the external interfaces to this node are
  simulated

9
Risk model

• The case study was based on a qualitative risk
  model and a risk was measured as a risk exposure
  
• Risk Exposure Probability x Consequence
• Risk items were categorized into three classes
  according to the probability and consequence of a
  risk tolerable, undesirable, and intolerable

10
Risk analysis and prioritization

• Risk analysis was performed both for the test
  objects and test cases using systems experts
• 2 systems experts participated into the risk
  analysis of test objects and 8 systems experts
  analyzed the risks related to test cases
• Test objects and test cases were prioritized
  according to risk exposure
• In a case of test case prioritization, also other
  prioritization criteria (consequence, old test
  case priority) were needed because several test
  cases had the same risk exposure

11
Scenarios

• The analysis was based on 5 scenarios in which
  the amount of executed test cases was changed
• In scenario 1, as many test cases were selected
  for execution per test object as in the original
  case but the test cases were selected based on
  risk
• In other scenarios, the test effort was focused
  on the most risky test objects by defining how
  many percent of the test cases per test object
  were executed
• Scenario 2 Intolerable 25, Undesirable 15,
  Tolerable 10
• Scenario 3 Intolerable 20, Undesirable 10,
  Tolerable 5
• Scenario 4 Intolerable 15, Undesirable 10,
  Tolerable 0
• Scenario 5 Intolerable 10, Undesirable 5,
  Tolerable 0

12
Results used man hours

• Significant savings in used man hours would have
  been achieved in scenarios 3-5
• In the original case, the used man hours were
  distributed equally to all test objects
• In risk-based approach (i.e. scenarios 2-5),
  approximately 55 of man hours were consumed to
  20 of test objects

13
Results risk level

• Risk level defines how many percent of the total
  risk is still uncovered
• Risk level would have been 10 percentage units
  lower in the original case if the test cases had
  been selected based on risk
• In the scenarios 3-4, risk level is at the same
  level than in the original case even though much
  less man hours were consumed

14
Results detected faults

• Only 40 of the faults were found in scenario 1
  in which the test cases were selected based on
  risk but the test effort was not concentrated on
  the most risky test objects
• By concentrating also the test effort to the most
  risky test objects, 40-60 of the faults were
  found
• Deeper analysis to these poor fault-detection
  rates revealed that the risk analysis of test
  cases of TO_04 had failed and caused faults to
  slip through testing

15
Results test case selections

• Differencies in test case selections between
  previously used and risk-based approach was
  analyzed by determining test case coverage which
  defines how many percent of test cases executed
  in the original case was also executed in
  designed scenarios
• In addition, the average risk exposure per test
  case was determined to each scenario

16
Conclusions

• Risk-based regression testing would not have
  brought cost savings in used man hours because
  the costs of the faults that slipped through
  testing would have eliminated all the gained cost
  savings
• Testing effort was not concentrated on the most
  risky test object in the original case
• It can not be said that testers would have
  utilized risk-based testing approach when
  selecting regression test cases in the original
  case

17
Future Research

• A possible pilot project for risk-based testing
  in order to get more empirical experience how
  risk-based testing works in practice
• The concept and usage of risk level should be
  investigated more because risk level could be
  possible used as an indicator to tell when it is
  time to stop testing
• To develop a framework for assessing the costs of
  the faults quantitatively

18
(No Transcript)