Test Process Assessments Using the TMM Model

1

• Test Process Assessments Using the TMM Model

Patrik Wikström
2
Agenda

• Introduction
• TMM
• Background
• Levels
• Structure
• Example
• Level 2
• Assessment
• Critical Success Factors
• Verdict
• TMMi

3
What is a process?

• Process documentation
• Process knowledge
• Process results

4
The test process

• Planning
• Specification
• Execution
• Reporting
• Unit test
• Integration test
• System test
• Acceptance test

5
Why improve the test process?

• To decrease the time of the testing
• To decrease the cost of testing
• To increase the quality of the product
• To increase the knowledge of the quality of the
  product

6
Process improvement

• Initiating
• Diagnosing
• Establishing
• Acting
• Leveraging
• Start over
• A reference model make the diagnosing phase easier

• IDEAL (SEI)

7
Process improvement models
8
TMM Testing Maturity Model

• Illinois Institute of Technology
• Ilene Burnstein
• C.R. Carlson
• Taratip Suwannasart
• Ariya Homyen
• Robert Grom
• Crosstalk (The Journal of Defense Software
  Engineering)
• Developing a Testing Maturity Model (2
  parts Aug. 1996 Sept. 1996)
• A Model to Assess Testing Process
  Maturity (Nov. 1998)
• Using SW-TMM to Improve the Testing
  Process (Nov. 2002)
• Practical Software Testing, Ilene Burnstein,
  2003

9
Levels
10
Level 2 Phase Definition

• Develop testing and debugging goals
• Initiate a test planning process
• Institutionalize basic testing techniques and
  methods

11
Level 3 Integration

• Establish a software test organization
• Establish a technical training program
• Integrate testing into the software life cycle
• Control and monitor the testing process

12
Level 4 Management and Measurement

• Establish an organizationwide review program
• Establish a test measurement program
• Software quality evaluation

13
Level 5 Optimization, Defect Prevention and
Quality Control

• Defect prevention
• Quality control
• Test process optimization

14
Structure

• 5
• 13
• 43
• 384

200 159 25
15
Example (1/3)

• Level 2, Phase Definition
• Maturity goal 2.3, Institutionalize basic
  testing techniques and methods
• Maturity subgoal 2.3.3
• Software testing is planned and implemented at
  the unit, integration, system, and acceptance
  levels according to policy.

16
Example (2/3)

• Activities/tasks/responsibilities for managers
• Ensure that testing policies and standards
  require multilevel testing
• Ensure that developers (testers) have the needed
  education and training to perform testing at
  multiple levels
• Ensure that multilevels of testing are covered in
  the test plans
• Promote cultural changes needed to support
  multilevel testing
• Adjust project schedules so that multilevel
  testing can be performed adequately

17
Example (3/3)

• Activities/tasks/responsibilities for testers
• Attend classes and training sessions, read
  materials, acquire tools, work with knowledgeable
  colleagues, and gain hands-on experience testing
  at the unit, integration, system, and acceptance
  levels.
• Ensure that a balance of test approaches is used
  for test case design at all levels of testing
• Execute test cases at all levels of testing
• Work with project (test) managers to ensure there
  is time and resources to test at all levels
• Mentor colleagues who wish to acquire the
  necessary background and experience to perform
  multilevel testing
• Activities/tasks/responsibilities for
  users/clients
• -

18
Level 2 Develop testing and debugging goals

• There should be procedures, goals, and policies
  for testing and debugging
• Testing and debugging policies/goals are
  reflected in project/test plans
• A basic defect classification scheme is
  established, and a basic defect repository is put
  into place
• Simple testing and debugging measurements are
  identified and collected

19
Level 2 Initiate a test planning process

• There should be procedures, goals, and policies
  for test planning
• Test plan templates for all test levels
• Technical training is available to cover use of
  test plan templates and development of test plans
• A procedure is put in place to include
  user-generated requirements as inputs to the test
  plan
• Basic planning tools and test measurements are
  evaluated, and applied

20
Level 2 Basic testing techniques and methods

• A set of basic testing techniques and methods are
  studied, evaluated, and recommended
• Technical training and basic tools are available
  to support use of testing techniques and methods
• Software testing is planned and implemented at
  the unit, integration, system, and acceptance
  test levels according to policy
• Basic testing strategies, techniques and methods
  are used organizationwide to design test cases.

21
The test process TMM level 2

• Test planning process
• Test techniques
• Tools
• Basic measurements
• Testing is planned and executed at all test
  levels according to test policy

• Planning
• Specification
• Execution
• Reporting
• Unit test
• Integration test
• System test
• Acceptance test

22
Assessment

• Input
• Interviews
• Questionnaire
• Documentation

• Output
• TMM level
• Process strengths and weaknesses
• Process goals
• Action plan

23
Assessment procedure

• Analysis of the application area, the system
  development process and environment as well as
  the test environment and organization
• Interviews with managers, testers, developers and
  clients/users
• Analysis of available and relevant documentation
• Assessment based on TMM
• One or more workshops at which the assessment is
  presented and discussed
• One or more workshops at which short and
  long-term goals are discussed using the
  assessment as a basis
• Final report
• 120-160 hours

24
Critical Success Factors

• Senior management commitment
• Staff involvement
• Test process improvement awareness
• Staff time and resources
• Training and mentoring
• Creating process action teams
• Improvement of all test levels

25
Advantages

• Similar to CMM
• Deals with many important aspects of strong test
  process
• Improved software quality
• Improved software testing control

26
Disadvantages

• No focus on risk management
• Not very concrete
• Does not use the same terminology as CMM
• Much focus on documentation
• Does not provide a continuous view as CMMI

27
TMMi

• TMMi Foundation - European non-profit
  organization
• The TMMi Foundation will provide a standard TMMi
  Model that can be used in isolation or in support
  of other process improvement models and is either
  staged or continuous
• http//www.tmmifoundation.org/

28
Next seminar

• TMM Testing Maturity Model
• TMM Testability Maturity Model
• TPI Test Process Improvement
• TIM Test Improvement Model
• MTPF Minimal Test Practice Framework
• TOM Test Organization Maturity
• TAP Testing Assessment Program
• V2M2 - Verification Validation Maturity Model
• Which one to choose?

29
Thank you for listening!

• Any questions?
• patrik.wikstrom_at_testway.se