A Tool for Risk-Based Testing

1
A Tool for Risk-Based Testing

• Linda Sherrell, Sarah Bowen, and Hima Puppala

2
Importance of Risk Management

• Increased interest in industry
• Key process area of CMMI
• Typical methods
• Majority emphasize final product
• Most of risk assessment occurs during
  requirements
• Some analysts feel this is too narrow a view

3
Risk Management Process

• Risk identification uncovering of risks
• Risk analysis quantification of risks
• Risk planning contingency and mitigation plans
• Risk monitoring continuous assessment and
  adjustment of risk mitigation plans

4
Risk Identification Techniques

• Taxonomies
• Decision-driver analysis
• Assumption analysis
• Decomposition
• Generic risk lists
• Risk catalogs
• Quality criteria categories

5
Risk Analysis

• Two major techniques
• Cause-effect analysis
• Risk Exposure
• Risk Exposure Probability Impact
• Prioritization of risks based on risk exposure or
  individual values
• Probability and impact values may be either
  quantitative or qualitative

6
Heuristic Risk Based Testing

• Inside-out approach
• Begin with situation details
• Then perform risk identification
• Outside-in approach
• Use predefined risk list
• React to risks in current circumstance

7
QUART-ET (Quick Assessment of Risks Tool for
Engineering Testing)

• Project Goals
• To provide an easy-to-use tool to support
  heuristic risk based testing
• To allow the testing team to compare current risk
  assessment values to previous data
• Tool Characteristics
• Supports the first two steps of risk management
• Provides users with generic lists for both
  product risks and process risks

8
Generic Risk Lists

• High-level risk categories
• Detailed risk factors
• Example Complexity
• Adding a new feature/module that is highly
  complex
• Modifying a new feature/module that is highly
  complex
• Modifying a feature/module with a large number of
  lines of code

9
Product Risk Categories

• Failure history
• Internal dependency
• External dependency
• Complexity
• Distributed components
• General

• Data
• Unfamiliar
• Resources
• Global
• Environment/Configura-tion/System setup

10
Process Risk Categories

• Requirements discipline
• Modifying requirements late in process
• Lack of testing personnel on requirements team
• Design discipline
• Modifying design late in development
• Testing discipline
• Lack of full regression testing
• Lack of security testing
• All disciplines
• Lack of backward traceability
• Noncompliance with company standards

11
Conclusions

• QUART-ET can be used to foster a culture that
  supports risk management
• The process supported by QUART-ET complements
  prioritization of testing items
• Tool can be extended to handle additional risk
  management process steps

12
References

• Bach, J. (1999, November). Heuristic Risk-Based
  Testing. Software Testing and Quality Engineering
  Magazine, pp. 1-9. Retrieved from
  http//www.satisfice.com.
• Boehm B.W. (1991). Software Risk Management
  principles and practices, IEEE Software, vol. 8,
  pp. 32-41.
• Boehm, B. and R. Turner (2004). Balancing
  Agility and Discipline A Guide for the
  Perplexed. Boston Addison-Wesley.
• Carr, M. J (1997). Counterpoint Risk Management
  May Not Be for Everyone. IEEE Software, vol. 14, 
  no. 3,  pp. 21,23.
• DeMarco, T. and T. Lister (2003). Waltzing with
  Bears Managing Risk on Software Projects. New
  York Dorset House.

13
References (cont.)

• Gotterbarn, D. (2001). Reducing Software
  Failures Addressing the Ethical Risks of the
  Software Development Cycle. In T.W. Bynum et
  al., eds. Proceedings of Ethicomp 2001 The
  Fifth International Conference on the Social and
  Ethical Impacts of Information and Communication
  Technology. Vol 2. Gdansk, Poland Wydawnictwo
  Mikom, pp. 9-21.
• Hall, E. (1998). Managing Risk Methods for
  Software Systems Development. Reading, MA
  Addison-Wesley.
• Lister, T. (1997). Point Risk Management Is
  Project Management for Adults, IEEE Software,
  vol. 14,  no. 3,  pp. 20-22.
• Ould, M. A. (1999). Managing Software Quality and
  Business Risk. Chichester John Wiley Sons.