RiskBased SoftwareTesting Workshop

1
Risk-Based SoftwareTesting Workshop

• with Larry McCartney

2
My Background

• Mathematician (Yuma Proving Ground)
• SMTS (Sandia Labs)
• Sr. Automated Test Engineer (Applied Biosystems)
• Quality Assurance (RESPEC/POD)
• Teach software engineering courses (UNM)

3
Questions This Workshop Will Address

• What is risk-based testing (RBT)?
• How does RBT differ from traditional testing?
• When should RBT be used?

4
References

• Risk-Based E-Business Testing, Paul Gerrare and
  Neil Thompson, Artech House, 2002.
• Risk Based Testing, Strategies for Prioritizing
  Tests against Deadlines, Hans Schaefer, 2005.

5
Introduction

• What is your definition of risk-based testing?
• Name a situation in which you have used it.
• The definition of RBT in this workshop will use
  the method described and the associated tool used
  in the book by Gerrare and Thompson.

6
Project Risks vs. Product Risks

• Project Risks
• Potential budget, schedule, staffing and
  requirements problems and their impacts on the
  project
• These risks are not the focus of RBT
• Product Risks
• Potential for problems in the product
• The focus of RBT

7
Product Risks

• Business Risks
• What does it mean to the business if this feature
  doesn't work correctly?
• How important is the feature to the customer?
• Supplied by a business analyst, customer
  representative or domain expert
• Technical Risks
• Does a component need more testing because it has
  recently changed or may adversely affect other
  components?
• Is the code unusually complex and therefore more
  error prone?
• Supplied by the test and development teams

8
How Does Traditional Testing Differ from RBT?
9
Workshop Activity for RBT

• Student testing project - overview
• Students meet with the customer to elicit
  requirements

10
Product Customer

• I want to keep existing customers and allow the
  largest number of customers. Custom configuration
  of the system environment shall be minimal.
  Installation shall be easy. The system shall be
  compatible with the 3 most popular browsers.

Silvio Dante
11
Product Sponsor

• Ill be very unhappy unless my customers are
  happy.

Tony Soprano
12
Student Project - 2

• Web-based e-Business application
• Evolutionary Prototype Model with iterative
  releases every 2 weeks

13
Evolutionary Prototype Model
Design
Analysis
Build/Revise
Implementation
Release/ User Test
Maintenance
14
Initial Prototype

• A working prototype of the application is
  available for testing

15
Tonys Online Pizza Store (TOPS) Prototype
16
Risk Analysis

• The customer participates in pre-release reviews

17
Pre-release Product Reviews

• Pre-Release Product Reviews are meetings held
  prior to the iterative releases of the product
• After an initial workable application has been
  tested
• Include a risk analysis brainstorming session
• Attendees the stakeholders of the project
• Business analyst, domain experts
• Customer end users, management
• Supplier marketing manager, project manager,
  developers, test manager

18
Risk Analysis Brainstorming Session

• During the brainstorming session
• Test manager presents business and technical
  risks based on testing an early released
  iteration
• Stakeholders prioritize the risks
• Stakeholders assume responsibility for the risks
• Questions to consider for stakeholders
• Does the product meet our expectations?
• Is the application ready for users?
• What are we risking if we release now?

19
Risk Analysis Brainstorming Session-2

• Output of this meeting
• Risk Analysis Spreadsheet containing a
  prioritized list of business technical risks to
  be tested
• Responsible tester assigned to each risk
• Time allotted to testing each risk
• Subsequent brainstorming sessions are scheduled
  as risks and their priorities change

20
Activity

• Distribute materials

21
Risk Analysis Spreadsheet

• Test Process Worksheet for Tonys Online Pizza
  Store (TOPS) Risk Analysis Spreadsheet.xls
• Tabs are Risk Categories
• Availability
• Client Configuration
• Component Functionality
• Documentation
• Installation
• Integration
• Localization
• Performance, Volume, Stress
• Security
• Browser Configuration
• Usability

22
Spreadsheet Headers

• Failure Mode describe the specific feature that
  fails
• Benefits Threatened business objectives
  threatened
• Risk Type T technical, B business
• Probability of Failure 1 low, 10 high
• Consequence of Failure 1 low, 10 high
• Exposure Probability of Failure Consequence
  of Failure
• Test Effectiveness tester confidence in
  addressing the risk 1 low, 10 high
• Test Priority Number Exposure Test
  Effectiveness
• Test Objectives describe objective to address
  the risk
• Test Techniques scripted, exploratory, automated
• Effort person-months (1 or greater)
• Timescale time in hours (1 or greater)
• System Test people responsible

23
Risk Categories

• Client Configuration, Installation and Usability

24
Picking Risks to Test

• Steps
• Identify the risks in each category (rows of the
  worksheet)
• Pick the risks to test based on Test Priority
  Number and Time Scale

25
Exercise

• The stakeholders consisting of the customer
  (Silvio) and the supplier (the student
  development team) have decided that you (the test
  team) will have 12 hours to test the second
  iteration of TOPS
• Pair up with a partner
• Using the Risk Analysis Spreadsheet, identify the
  risks that you will test by filling in the TOPS
  RBT Solution form

26
A Solution
27
Questions

• When should RBT be used?
• When should RBT not be used?