Evaluation of Information Systems Defect Analysis and Removal

1
Evaluation of Information SystemsDefect Analysis
and Removal

• INFO 630
• Glenn Booker

2
Motivation

• Much of software measurement is devoted to
  improving the quality of the product
• To do so, it helps to understand how, when, and
  why mistakes are made during the software life
  cycle, resulting in defects
• It is assumed that defects are queued for being
  fixed (removed) after they are detected

3
Defect Analysis Overview

• Defect analysis looks at
• When a defect was created (injected)
• When a defect was found (detection)
• What caused a defect (type of defect, and/or
  orthogonal defect classification)
• How a defect was found (triggers)

4
Defect Creation (Injection)

• Defects may be created, detected, and removed
  during every phase of the software life cycle,
  including during maintenance
• Most are created during requirements analysis,
  design, and coding
• Testing and maintenance create relatively few
  defects

5
Waterfall Life Cycle
6
Defect Detection

• Defects may be found
• During quality assurance activities between
  phases of the life cycle (such as during major
  reviews and inspections) and often
• Within each phase (during preliminary reviews,
  peer reviews and testing activities)
• Here we mostly consider the former
  (between-phases activities)

7
Defect Detection

• Defects are found (discovered) by
• Reviews and formal inspections
• For requirements analysis, high level design, low
  level design, and coding
• Testing
• Unit testing
• Integration testing
• System testing
• Acceptance testing

8
Origin of Defects Type

• What kind of defects are made in each life cycle
  phase?
• Requirements - incorrect specification missing
  requirements
• High level design - design does not cover all
  requirements, or is inflexible
• Low level design - mismatch between HLD and LLD
  design does not cover all requirements

9
Origin of Defects Type

• Coding - code errors
• Integration interface, compatibility problems
• Unit testing - bad fixes
• Component testing - bad fixes
• System testing - bad fixes
• Acceptance testing - bad fixes
• Maintenance (enhancement) - bad requirements,
  design, coding, testing

10
Defect Causal Analysis

• Regardless of when defects are created or
  discovered, they may be analyzed to determine
  their cause
• Causes may range from the mundane to the esoteric
• When identified, causes should be recorded to
  support defect prevention activities

11
Types of Defect Causes

• Some causes may include
• Unclear requirements
• Incorrect architecture
• Incomplete design
• Unfamiliarity with programming language
• Typos
• Inadequate understanding of interfaces
• Inadequate understanding of standards (e.g.
  TCP/IP, ODBC, SQL, etc.)

12
Types of Defect Causes

• and
• Poor documentation for legacy system
• Design based on outdated requirements
• Code written to outdated design
• Poor design for future expansion (e.g. hardwired
  constants, buried assumptions, etc.)
• Conflicting requirements
• Bad fix of a previous problem
• and many more

13
Orthogonal Defect Classification

• ODC is a formal type of defect causal analysis
• Classify defects by the type of defect, and the
  life cycle phase when it was created
• This particular scheme is still somewhat
  experimental, but the basic concept is widely
  accepted

14
ODC - Defect Type

• A possible set of definitions
• Function erroraffects capability, user, product
  or hardware interfaces, global data structures
• Assignment error errors with initialization of
  control blocks or data structure
• Interface error errors in interacting with
  other components, modules, or or device drivers
• Checking errors in validating data and values

15
ODC - Defect Type

• Timing/serialization errors in management of
  shared and real-time resources
• Build/package/merge errors due to mistakes in
  library systems, change management, or version
  control
• Documentation errors in publications and
  maintenance notes
• Algorithm errors regarding efficiency or
  correctness that affect the task

16
Orthogonal Defect Classification

• Type Phase Caused
• Functional Design
• Interface Low Level Design (LLD)
• Checking LLD or Coding
• Assignment Coding
• Timing LLD
• Build/package Library tools
• Documentation Publications
• Algorithm LLD

17
Defect Trigger

• Is a condition that allows a defect to surface
  (become visible) examples include
• Design conformance
• Logic or data flow incomplete
• Workload or stress (performance)
• Boundary conditions (extremes)
• Bug fix
• Recovery timing
• User code

18
Defect Removal Modeling

• Defect removal is critical to reducing
  (development or maintenance) cycle time and cost,
  and improving quality
• Michael Fagan and Capers Jones are noteworthy
  authors
• See, for example, Fagan Inspections

19
Defect Removal Model (DRM)

• Software development phase-based DRM covers
• Defect injection into a phase
• Defect removal during a phase
• Defect carryover across phases
• Effectiveness of defect removal within a phase

20
Defect Removal Model

• DRM is a quality management tool
• Provides insights into the defect removal process
  and where this process might be improved
• Used after a project is completed provides post
  mortem information about that project
• If defect removal process is similar for a new
  project, then the DRM can be used to improve the
  defect removal process of that new project

21
Defect Removal Model

• Defects are created (injected) by incorrect
  requirements gathering, analysis, design, coding,
  or by bad fixes
• Any given defect can be removed in the phase in
  which it was injected or in a later phase
• Remove defect means fix bug, redo specs and/or
  design as needed

22
Defect Removal Effectiveness

• Phase based defect detection activities
• Requirements analysis inspection
• Design inspection
• Code inspection, Build verification testing
• Unit, string, integration, regression, system
  testing
• Fix verification inspection testing

23
Defect Removal Model
Applies to each life cycle phase
24
Defect Removal Effectiveness

• For any given phase of defect removal
• Defects present at removal operation
• Defects found during removal operation defects
  found later
• May be determined ex post facto (after the fact),
  or using a statistical prediction model

25
Defect Removal Effectiveness

• Defect removal effectiveness ( of defects
  found by inspection) /( of defects originally
  present) 100
• Early detection percentage ( of major
  inspection errors) /( of major and minor
  errors) 100

26
Defect Matrix Assumptions

• Defects are removed in the same life cycle phase
  when they are found
• No defects are knowingly left unfixed
• No bad fixes or at least they are blended into
  the number of defects created in that life cycle
  phase

27
Sample Defect MatrixWhen Originated vs. When
Found
28
Defect Removal Effectiveness
Next (Previous Created) FoundDRE Found /
(Previous Created) 100
29
High Level Design Effectiveness

• High (Top) Level Design (I0) Inspection
  Effectiveness
• Defects found and removed at I0 730
• Defects existing on step entry (escapes from
  requirements phase 122
• Defects injected in current phase 859
• E(I0) 730/(122859) x 100 74

30
Low Level Design Effectiveness

• Low Level Design (I1) Inspection Effectiveness
• Defects found and removed at I1 729
• Defects existing on step entry (escapes from
  requirements phase and I0) 122859-730 251
• Defects injected in current phase 939
• E(I1) 729/(251939) x 100 61

31
Code Inspection Effectiveness

• Code Inspection (I2) Effectiveness
• Defects found and removed at I2 1095
• Defects present on step entry (escapes from
  requirements phase, I0, and I1)
  122859939-730-729 461
• Defects injected in current phase 1537
• E(I2) 1095/(4611537) x 100 55

32
Unit Testing Effectiveness

• Unit Testing (UT) Effectiveness
• Defects found and removed at UT 332
• Defects existing on step entry (escapes from
  previous phases) 1228599391537-730-729-1095
  903
• Defects injected in current phase (bad fixes) 2
• E(UT) 332/(9032) x 100 37

33
Summary Effectiveness Measures

• Overall Design Coding Inspection Effectiveness
• IE (7307291095)/(1228599391537) x 100
  74
• Overall Effectiveness of all Testing activities
• TE (332387111)/(903241)x100 91
• Overall Defect Removal Effectiveness of the
  development process
• DRE (07307291095332387111)/(122859
  9391537241) x 100 3384/3464100 97.7

34
Rayleigh Model

• The Rayleigh Model describes the number of
  defects which will be discovered, by development
  phase
• Its a special case of the Weibull family of
  distributions, which well cover later in the
  course

35
Rayleigh Model Assumptions

• Defect rate observed during development process
  is positively correlated with defect rate in
  field
• Given the same defect injection rate, if more
  defects are discovered and removed earlier, fewer
  will remain in later stages, leading to fewer
  defects in the field

36
Rayleigh Model Inspections

• I0 High Level Design Inspection
• I1 Low Level Design Inspection
• I2 Code Inspection
• UT Unit Testing
• CT Component (Integration) Testing
• ST System Testing
• GA after General Availability (release or
  Fielding of the system)

37
Rayleigh Model
Each bar represents the number of defects found
during that life cycle phases inspection activity
38
Desired Rayleigh Curve Trends
Effect of Early Defect Removal and Reducing
Error Injection
39
Cost of Defect Removal by Phase

• Defect removal and rework is less costly the
  closer that the defects are found relative to the
  phase in which they are injected
• Rework in the I0, I1, and I2 inspection levels
  can be 10 to 100 times less expensive than if it
  is done during formal testing
• Reviews can reduce number of defects reaching
  testing phases by factor of 10
• These reductions cut testing costs by 50-80,
  even including the review costs

40
Cost Effectiveness of DRM

• Relative cost of fixing a problem found in
  design/coding, testing, or after release
  are12082 (Remus, 1983)11392 (Kan, 1989)
• Cost of defect removal can be analyzed by
  inspection type, testing phase, defect severity,
  defect origin, etc.