An empirical study of defects in industrial projects

1
An empirical study of defects in industrial
projects

• PhD-seminar23. November 2004
• Jon Arvid Børretzen

2
Relation to BUCS

• This investigation aims at doing analysis on
  projects with respect to metrics and attributes
  that are relevant to BUCS.
• In business-critical systems, the critical part
  is regularly related to the dependability of the
  system.
• I will be trying to make comparisons between
  component-based systems and more traditional
  architecture systems.

3
An empirical analysis of defects in industrial
projects

• This empirical study is planned to look at how
  defects have been introduced into the system, and
  how they have been found and dealt with.
• By analysing defect-/change reports for several
  (semi-)completed development projects, we want to
  investigate if there are common causes for
  defects being introduced in systems and/or not
  being discovered earlier.

4
Defects, errors and failures

• Defects (or faults) are potential flaws in a
  hardware/software system, that later may be
  activated to produce an error.
• An error is the execution of a "passive fault",
  leading to erroneous behaviour or system state.
• A failure is a fault execution (an error) that
  results in observable and erroneous external
  behaviour.

5
Data material

• Fault reports
• Design documents
• Relevant code

6
What do we hope to learn?

• Why did we make these mistakes in the first
  place?
• Why did we not find them earlier (inspection,
  testing)?

7
Defect categorisation

• Type of defect ()
• Severity (negligible ? severe)
• System component
• Origin of work (reuse, cots, in-house)
• Life-cycle phase for defect (where the defect was
  introduced)
• (concept ? maintenance)
• more likely (analysis ? operation)
• Phase of discovery
• Reason for not discovering earlier

8
Analysis steps

• 1. Separate actual faults/defects from cosmetic
  changes
• 1b. Sort out duplicate defect reports
• 2. Categorise defects according to criteria
• 3. Analyse results
• 4. Generalise results
• 5. Describe what we can learn

9
Generalising results (problem)

• Just a few projects to study
• Differences in
• Domain
• Environment
• Programming environment/style
• Defect/failure report data
• How to make findings interesting in general, not
  just to the one project

10
Categorisation issues

• What do we consider to be a fault?
• Design faults?
• Wrong implementation choices?
• Changes in specification?
• Changes in design?

11
To sum up

• Empirical study of industrial projects
• Analysis of historical data in form of fault
  reports
• Categorisation of defects
• Relate this to dependability/reliability

12
Questions

• What should be considered as defects/faults?
• What about design defects?
• How can knowledge about system defects improve
  the software process?
• How to show improvement effects for projects only
  with study result without executing change
  actions?
• Or will it only be ideas for improvement?