Software Defect Prevention Using Orthogonal Defect Classification

1
Software Defect PreventionUsing Orthogonal
Defect Classification

• Twin-SPIN
• January 6, 2005
• Presented by
• Megan Graham, ASQ CSQE
• megan_at_metagraham.net

2
Overview

• Defect prevention is a process used to improve
  software quality
• Orthogonal Defect Classification is a tool that
  characterizes defect data used in defect analysis

3
Agenda

• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary

4
Agenda

• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary

5
Defining Defect Prevention

• What does it mean to you?
• Defect prevention is a process whose purpose is
  to
• identify the common causes of defects, and
• change the relevant process(es) to prevent that
  type of defect from recurring. (SEI)
• Take what we already know and apply it to what we
  think we know to produce quality software.

6
A Bugs Life
7
A Birds Eye View of DP
8
Applying Defect Prevention

• A defect in the software is also a defect in the
  process (injection and/or detection)
• For DP to work, we need to turn software defects
  into actionable process defects

9
And then there was ODC

• Orthogonal Defect Classification
• Developed at IBM in the 1990s by Ram Chillarege
• Methodology to characterize software defects and
  translate into process defects

10
Agenda

• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary

11
Simple ODC Classification Scheme
Type
Defect
Trigger
12
Defect Types

• FUNCTION Affects significant capability,
  end-user interfaces, product interfaces,
  interface with hardware architecture or global
  data structure(s).
• LOGIC Affects the functionality of a code module
  and can be fixed by re-implementing an algorithm
  or local data structure without a need for
  requesting a high level design change.
• INTERFACE Affects the interaction of components
  via macros, call statements and/or parameters.
• CHECKING Affects program logic that would
  properly validate data and values before they are
  stored or used in computation.
• ASSIGNMENT Requires change in a few lines of
  code, such as initialization of control blocks or
  data structures.
• TIMING/SERIALIZATION Affects the proper
  management of shared and real-time resources.
• BUILD/PACKAGE/MERGE Affects product version or
  configuration requires a formal changes to
  reconfigure or rebuild the product.

13
Defect Triggers
14
Review/Inspection Triggers

• DESIGN CONFORMANCE Comparing the implemented
  design against a reference design document,
  pattern, or guideline.
• LOGIC/FLOW Checking for correctness or flaws
  using knowledge of the practice.
• BACKWARD COMPATIBILITY Examining compatibility
  with prior version of the product.
• LATERAL COMPATIBILITY Examining for
  compatibility with other products and platforms
  that need to work with this release.
• CONCURRENCY Serialization, shared resources,
  multi-threaded tasks, timing, etc.
• INTERNAL DOCUMENT Inconsistencies in prologs,
  and sections in the same work product.
• LANGUAGE DEPENDENCY Programming standards,
  specific implementation considerations,
  environment restrictions, execution modes, etc.
• SIDE EFFECTS Usage behavior beyond design, but
  relevant in context. Do A B happens.
• RARE SITUATION Unusual issues related to
  idiosyncrasy of environment, hardware, or
  software.

15
Function Test Triggers

• SIMPLE PATH White box Straightforward usage of
  code path and branches.
• COMPLEX PATH White box Exercising
  conditionals, and circuitous coverage.
• COVERAGE Black box Straightforward usage of
  function or single parameterized execution.
• VARIATION Black box Straightforward like
  coverage, but with a variety of parameters.
• SEQUENCING Black box Multiple functions, with
  a specific sequence (order is important).
• INTERACTION Black box When two or more bodies
  of code are involved (order is not important).

16
System Test Triggers

• WORKLOAD STRESS Pushing the limits of
  performance, resources, users, queues, traffic,
  etc.
• RECOVERY Invoke exception handling, recovery,
  termination, error percolation, etc.
• STARTUP/RESTART Major events of turning on, off,
  or changing the degree of service availability.
• HARDWARE CONFIGURATION Issues surfaced as a
  consequence of changes in hardware setup.
• SOFTWARE CONFIGURATION Issues surfaced as a
  consequence of changes in software setup.
• BLOCKED TEST/NORMAL MODE Test could not run
  during System Test, or customer found nonspecific
  trigger. Look for additional trigger.

17
ODC v5.11 Classification Scheme
Trigger
Impact
Activity
Defect
Age
Target
Type
Source
Qualifier
Source ODC v5.11, IBM Center for Software
Engineering, www.research.ibm.com/softeng
18
Agenda

• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary

19
When to Apply DP

• DP can be applied to one or more phases of the
  software lifecycle
• Depends on maturity, goals, etc.

20
Defect Analysis
Orthogonal Defect Classification What are the
attributes of the defects?
Defect CausalAnalysis When are defectsbeing
injected?
Defect TriggerAnalysis How are defectsbeing
detected?
ODC provides a structure for the defect data
21
Defect Causal Analysis

• Purpose Characterize process issues that lead to
  injection of defects.
• Step 1 ID a set of defects.
• Step 2 Identify Defect Types with team of
  experts.
• Step 3 Plot the distribution of Defect Types.
• Step 4 Map Defect Types back to development
  activity.
• Step 5 Develop action plan to address process
  deficiencies.
• Step 6 Monitor process to ensure changes were
  effective.

22
Defect Type Mapped to Phase
23
Sample Distribution
24
Defect Trigger Analysis

• Purpose Characterize process issues that allowed
  defects to escape to later phases.
• Step 1 ID a set of defects.
• Step 2 Identify Defect Trigger with team of
  experts.
• Step 3 Plot distributions Defect Trigger by
  Family, Review Triggers, Function Test Triggers,
  System Test Triggers.
• Step 4 Map Defect Trigger family back to
  detection activity(ies).
• Step 5 Develop action plan to increase missed
  Defect Triggers.
• Step 6 Monitor process to ensure changes were
  effective.

25
Defect Trigger Family Mapped to Phase
26
Sample Distributions
27
Additional Use for DTA

• Purpose Determine quality status.
• Step 1 Plot distribution of Defect Triggers
  after each detection activity.
• Step 2 Compare to historical defect profile to
  determine if profile is as expected.
• Step 3 Develop action plan, if necessary, to get
  back on track.
• If your project thinks its one phase, but the
  distribution is of an earlier phase, then youre
  really in the earlier phase!

28
Agenda

• About Defect Prevention
• ODC Concepts
• Applying ODC to the DP Process
• Summary

29
The Big Picture DP ODC

• Mission Improve software quality by using
  readily available data to decrease defects
  injected and increase defects detected.

Defect Prevention
30
Important Points about ODC

• A defect in the software is a defect in the
  process.
• Implementing ODC is very cost-effective
• Enhances data already collected (software
  defects)
• Adding fields that are completed real-time make
  data collection virtually free!
• Tooled to quickly identify process defects
  (mapping)
• ODC can be implemented in stages
• Start with field defects, then move to in-process
  analysis
• Utilize defect profiling in-process to predict
  quality and project status
• Fields can be tailored to your own organization

31
References

• Handbook of Software Reliability Engineering
  (Michael R. Lyu, Editor IEEE Computer Society
  Press/McGraw-Hill)
• Ram Chillarege (www.chillarege.com)
• IBM Research Center for Software Engineering
  (www.research.ibm.com/softeng)
• Soheil Khajenoori, SIAGroup (soheil_at_siagroup.us)