EEEGEF 492'17 Inspections

1
EEE/GEF 492.17Inspections
Royal Military College of Canada Electrical and
Computer Engineering

• Dr. Diane Kelly
• kelly-d_at_rmc.ca
• 613-541-6031

Dr. Terry Shepard shepard_at_rmc.ca 613-541-6031
2
This set of slides is based on a tutorial
presented by Dr. Terry Shepard and Dr. Diane
Kelly at the International Conference on
Software Engineering in Toronto in May 2001 under
the heading How to do Inspections When There is
No Time. A summary appears on pp. 718-719 of the
Conference Proceedings.
3
Outline

• Current Status of Inspection in Industry
• Fagan Inspections
• The 3 Ms of Inspection
• Management
• Mechanics
• Metrics
• Eight Maxims of Inspection

4
Current Status of Inspection in Industry (1)

• Informal survey in 1998 80 of developers
  practice inspection irregularly or not at all
  (Johnson 1)
• A more recent survey (2005) suggests 53 practice
  inspection irregularly or not at all (El Emam
  48)
• not to be regarded as definitive, or even an
  improving situation!
• Tech bubble peak and bust both happened in that 7
  year period
• In companies that do inspections, code
  inspections predominate (Laitenberger et al 2)
• 50 or more of development time spent testing
  debugging (Harrold 47 )

5
Current Status of Inspection in Industry (2) 25

• Recommendation to Microsoft to shift focus from
  testing to inspection (Cusumano et al 1995 23)
• Microsoft practices are changing, but details are
  not known.
• At best, the state of practice is we inspect our
  key components
• At worst, its we dont have the time to do
  inspections at all
• The value of inspection is probably the topic on
  which computing research agrees more consistently
  than any other

6
Rationale for Inspections

• The original argument payback on cost of fixes
  (e.g. debugging) leading to improved productivity
• according to work done by Glen Russell at BNR
  (now Nortel Technologies) in 1991, 65 to 90 of
  operational (execution-time) defects are detected
  by inspection at 1/4 to 2/3 the cost of testing
  13
• Other arguments for inspection
• better control of process
• higher quality
• reduced defect rates
• reduced cost of finding defects
• payback on fixing soft issues (Votta 3)

7
What makes inspections hard?

• Mentally demanding
• Specific skills needed may be in short supply
• Role of moderator requires special people skills
• meeting dynamics
• possible tension between inspectors and product
  author
• Time pressures
• squeezed by tight schedules
• inspection interval problem
• Attitudes
• perceived as uninteresting work
• not glamorous, support role only
• Lack of ownership
• who owns the inspection?
• Unclear goals, multiple goals

8
Industry Standard Inspection

• First described by Michael Fagan 1976 30
• subsequent paper in 1986 31
• Fagan defines inspection
• defines a process
• defines roles for the participants
• suggests feedback to improve the development
  process
• Fagan approach to inspections has become an
  industry standard
• Several variations
• e.g. Gilb and Graham 4 - next slide

9
Inspection process from Gilb and Graham -
variation on Fagan
10
Fagan-style inspections

• Entry and exit criteria
• Preparation
• Moderator
• Meeting 3-6 people
• Controlled rate typically 100 to 200 loc/hr
• rates that are too fast reduce effectiveness,
  while going too slow increases cost with little
  benefit
• Controlled meeting length max 2 hours
• Controlled number of meetings per day max 2
• Track and report defects found, time spent
• rate of finding defects per hour, rate per kloc
• Classify defects

11
New Approaches

• Research and industry practice suggest variations
  on
• goals
• meetings
• techniques
• size of teams
• entry criteria
• exit criteria
• when to inspect
• activities to include/exclude
• ...
• There are better approaches than the commonly
  used Fagan method. 25

12
The 3 Ms of Inspection
Inspection
Mechanics
Metrics
Management
13
Three Ms of Inspection

• Management
• Mechanics
• Metrics

14
Management

• Goals
• Culture
• Resources
• Process

15
Management goals (1)

• Management sets goals
• goals can be related to operations, stock price,
  profits, marketing, product lines, safety,
• In order for an inspection program to be
  successful --
• management and development teams must agree on
  the goals
• goals must be clear, relevant and meaningful

16
Management goals (2)

• Goals have to be operationalized and interpreted
• there are usually difficult choices among
  alternative approaches to operationalization
• management guides the definition of expected
  deliverables
• inspection criteria should be based on realistic
  goals
• scope of inspection must be under control
• Circumstances, and therefore goals, may change
• flexibility and consistency must be balanced

17
Setting and evaluating goals

• One good framework is the Goal/Question/Metric
  paradigm (GQM) developed by Vic Basili
• Goals can be set at many levels
• Questions are formulated to determine when
  particular goals are met
• Metrics are chosen to supply answers (or parts of
  answers) to one or more questions
• see The Goal/Question/Metric Method A Practical
  Guide For Quality Improvement of Software
  Development, by Rini van Solingen and Egon
  Berghout, McGraw Hill, 1999

18
GQM example

• Goal Analyze the final product to characterize
  it with respect to the various defect classes
  from the point of view of the software
  development organization
• format of goal follows GQM template
• Sample Question What is the distribution of the
  introduction of errors across all the development
  phases?
• Metrics Number of requirements errors, number of
  design errors,

19
Management culture (1)

• Culture is made up of qualities like attitudes
  and perceptions, degree of openness, visible and
  hidden rewards and priorities,
• these determine the likelihood of success of
  inspections
• e.g.
• easier if the culture is open
• harder if management focus is on visible
  operational goals so that software is relatively
  invisible
• is being a good inspector considered a core
  competency in the organization?
• is there a standing VV group?
• is the organization willing to invest in
  long-term diversion of resources to inspections?

20
Management culture (2)

• Making the case depends on the culture
• need a champion for all but personal inspections
• The culture of keeping out of the way
• managers cant participate as managers,
  especially if they use inspections for evaluation
  of people
• on the other hand, the president of the company
  can be an inspector
• there are many other ways to evaluate people

21
Management resources (1)

• Kinds of resources
• personnel
• skill set issues
• availability
• inspectors must be perceived as impartial,
  objective, respected
• inspectors must understand the criteria for the
  inspection
• space
• mental overloaded people are poor inspectors
• physical quiet, comfortable, private, lack of
  interruptions
• inspection interval can increase if meeting rooms
  are scarce
• time
• time for inspections must be focused and
  dedicated
• schedule pressures may dominate, especially late
  in a project

22
Management resources (2)

• Quantifying soft factors 21
• human-oriented factors influence productivity
  more than process does (list from COCOMO model)
• analyst experience
• communications factors
• personnel continuity
• programmer capability
• capability of requirements analyst
• language and tool expertise
• programmer experience
• Soft factors collectively exert an influence on
  productivity ranging up to 25.8 times
• one source suggests that moving from CMM level 1
  to level 5 only improves productivity by 1.43
  times

23
Management process (1)

• Process definition is a choice
• a chaotic process is one possible choice
• Inspections help bring order out of chaos
• inspections are a form of discipline
• e.g. time estimation depends on time tracking
• weekly timesheets arent precise enough
• learning to track time is necessary
• granularity must be better than hourly
• making inspections an integral part of the
  development process gives structure to that
  process
• inspections lead to predictability
• collecting data from inspections should lead to
  further process improvement

24
Management process (2)

• Data-Based Process Improvement
• try inspection at different stages
• collect data to see which is most effective
• compare different process approaches
• collect data on testing and debugging effort
  levels with and without inspections
• the care needed to track times for inspections
  spills over into estimation and tracking of time
  for other activities
• Process improvement approaches that are not based
  on actual data are more likely to fail

25
Mechanics

• Inspect for specific goals
• Types of inspectors
• Desk checking
• What to inspect
• When to inspect
• Process variations
• Techniques
• Tools

26
Mechanics Inspect for Specific Goals - Examples

• Set an inspection goal to identify usability
  issues
• Ask questions such as
• What are these identifiers used for?
• What is the default choice in these switch
  statements?
• Have a numerical techniques expert check one
  module that does a matrix inversion
• Have a junior programmer check the entire product
  for meaningful identifiers
• Have a meeting focused solely on consistency
  issues of the user interface

27
Mechanics Inspect for Specific Goals Particular
Approaches 6,7

• Set objectives of inspection ahead
• Inspector skills are matched to specific aspects
  of the inspection
• Each inspector focuses on a specific purpose
• inspectors actively work with the document
• questionnaires dont have passive questions
  requiring simple yes/no answers
• Inspectors may either
• examine specific modules -or-
• examine entire product for one property
• Small meetings focused on specific aspects
• meetings may not raise new findings

28
Mechanics Types of Inspectors 6

• Suggestions for finding good inspectors
• specialists
• application area
• hardware
• operating system
• potential users of system
• those familiar with design methodology used
• those who enjoy finding logical inconsistencies
  and are skilled at doing so

29
Mechanics Desk Checking

• Part of your personal tool kit
• some studies show that desk reviews can be
  extremely effective (e.g. 32)
• track time it takes
• track defects found
• take ideas from Pair Programming 15 and Testing
  Buddies 23
• team up to do inspections with your buddy

30
Mechanics what to inspect

• All software artifacts can be inspected
• In some cases, code inspections can be replaced
  by design, architecture or requirements
  inspections
• In some cases, code inspections can replace unit
  testing
• Architecture and design inspections have high
  payback (XP challenges this current debate)
• Start small even one page can contain 50 or more
  defects (e.g. early requirements inspection)
• Inspection and testing are complementary
  activities

31
Mechanics when to inspect (1)

• Inspection is possible at any time
• Based on your goals
• Fit inspections into your existing process
• even an immature organization can implement an
  effective inspection program (e.g. 24)
• Dont allow the inspection to become a bottleneck
• do what you can in parallel with other activities
• Look at what resources are available

32
Mechanics when to inspect (2)

• Determine the criteria
• entry criteria
• code after clean compile, before testing?
• documents after spell checking, after syntax
  checking?
• anything can I comprehend what youve given me?
• re-inspection criteria
• more than 5 errors per page?
• exit criteria
• all findings addressed?
• inspection tasks completed?
• metrics at acceptable levels?

33
Mechanics Process variations
Process goal is to combine structure and rigour
with freedom to act effectively and so encourage
participants to add value

• Should process include sampling?
• Does every inspection need a Fagan-style meeting?
• alternatives
• Is there a case for more than one meeting?

34
Should process include sampling?

• Sampling selected portions of a document is an
  excellent practical alternative to complete
  checking. 4
• estimate issue density per page in unchecked
  pages
• can determine value of checking other pages
• can suggest systematic corrections to document
• can determine value of further inspection
• based on assumption that the same process that
  produced the entire document produced the sample
• mistakes on one page will often recur on many
  other pages
• will not identify specific issues on unchecked
  pages
• satisfies the purpose of quality measurement,
  teaching the product author, gaining insights
  about defects that could help in process
  improvement

35
Does Every Inspection Need a Fagan-style Meeting?
36
Does Every Inspection Need a Fagan-style Meeting?
(1)

• Report on an experiment Votta 5
• investigates the need for large inspection
  meetings
• two-person meetings are enough?
• Report on an experiment Porter Votta 26
• collection meetings produce no net improvement
  in terms of meeting losses and meeting gains
• Report on an experiment Johnson Tjanjono 17
• shows that the number of defects found does not
  increase as a result of meetings, but that
  meetings help to detect false positives

37
Does Every Inspection Need a Fagan-style Meeting?
(2)

• Report on an experiment Laitenberger et al 28
• non-traditional structure of meeting
  (DaimlerChrysler Germany)
• each inspector prepares a model of the
  requirements document under inspection
• each inspector presents their prepared model
• in the presentation, each inspector explains the
  defects found in the context of their model
• ensures
• each inspector is prepared
• technical justification for each defect
• discussions with author are related to technical
  content
• helps to avoid personal conflicts
• meetings all consist of two inspectors plus the
  author
• experiment concludes that meeting results are
  better than individual results

38
Is there a case for more thanone meeting?

• An Experiment to Assess the Cost-Benefits of Code
  Inspections in Large Scale Software Development
  (Porter, Siy, Toman, Votta 8)
• varied the the number of reviewers, the number of
  teams inspecting the code unit, and the
  requirement of fixing defects between first and
  second teams inspections
• two teams per inspection with fix was dropped
  part way through experiment as infeasible
• results on two teams in parallel are
  inconclusive, where the total number of people is
  the same (e.g. 2 teams of 1 vs. 1 team of 2)
• one of the conclusions
• structural changes to the inspection process do
  not always have the intended effect significant
  improvements to the inspection process will
  depend on the development of defect detection
  techniques

39
Mechanics techniques

• Unstructured inspection techniques
• ad hoc and paraphrasing
• check lists
• Structured individual techniques
• scenario based reading
• defect based reading
• test based perspective
• program comprehension case study
• modeling the inspection artifact
• task-directed inspections

40
Unstructured Techniques (1)

• Ad-hoc is common
• often based on paraphrasing
• no focus on particular issues
• no guidance on how to consider the product
• no explicit use of inspectors expertise
• not repeatable
• no record of inspectors thought processes and
  actions
• does the beginning of the product get more
  attention than the end?
• are conceptually difficult parts ignored?

41
Unstructured Techniques (2)

• Checklists are popular but have several
  shortcomings2, 9
• checklists are based on past experience
• some categories will not have been found yet
• depends on inspectors using individual expertise
  to find issues in missing categories
• inspectors may not look for anything beyond what
  is in checklist
• checklists may be lengthy and hard to use
• checklist categories may not be effective or
  productive
• categories need to be prioritized
• some categories may not provide enough guidance
• checklist for one project may not work on another
• some checklists should be personalized
• checklist effectiveness needs to be evaluated
• constant improvement needed

42
Unstructured Techniques (3)

• Checklists are effective when 12, 33
• well established history of the product exists
• product is predictable
• inspection goals are accompanied by a well
  documented method of operationalization
• e.g. style manual, standards
• providing a source of accumulated wisdom to a
  junior member of the team
• an inspector is working independently and
• the inspector is not experienced and
• the checklist provides a relatively complete tour
  of the product under inspection

43
Overview of Structured Inspection Techniques
(University of Maryland 10)

• Systematic
• inspector knows how to inspect the document
• Specific roles
• inspector is responsible for a specific role with
  a specific focus
• Distinct
• each role is distinct in the sense of minimizing
  overlap
• coverage of different defect classes is achieved
  by having multiple roles

44
Scenario Based Reading

• A technique that is used individually in order
  to analyze a product or a set of products. Some
  concrete instructions are given to the reader on
  how to read or what to look for in a document.
  (University of Maryland, Notes on Perspective
  based Scenarios 10)
• defect-based and perspective-based scenarios
  (Porter, Votta, Basili 11)
• focus on defect classes and use questionnaire to
  guide inspectors
• focus on a role (perspective) and use
  questionnaire to guide inspectors (may also
  produce a product associated with that role)
• can prepare a scenario for any inspection

45
Scenario example Defect-based reading 11

• Requirements inspection - Ambiguities or Missing
  Functionality Scenario
• Identify the required precision, response time,
  etc. for each functional requirement.
• Are all required precisions indicated?
• For each requirement, identify all monitored
  events.
• Does a sequence of events exist for which
  multiple output values can be computed?
• Does a sequence of events exist for which no
  output value will be computed?
• For each system mode, identify all monitored
  events.
• Does a sequence of events exist for which
  transitions into two or more system modes is
  allowed?

46
Scenario examplePerspective-based reading (1)
27

• Requirements inspection - Test-based Perspective
• For each requirement, make up a test or set of
  tests that will allow you to ensure that the
  implementation satisfies the requirement. Use
  your standard test approach and test criteria to
  make up the test suite. While making up your test
  suite for each requirement, ask yourself the
  following questions
• next slide ...

47
Scenario examplePerspective-based reading (2)
27

• Requirements inspection - Test-based Perspective
• Do you have all the information necessary to
  identify the item being tested and to identify
  your test criteria? Can you make up reasonable
  test cases for each item based upon the criteria?
• Is there another requirement for which you would
  generate a similar test case but would get a
  contradictory result?
• Can you be sure the test you generate will yield
  the correct value in the correct units?
• Are there other interpretations of this
  requirement that the implementor might make
  based upon the way the requirement is defined?
  Will this affect the test you made up?
• Does the requirement make sense from what you
  know about the application and from what is
  specified in the general description?

48
Program Comprehension Case Study 20,25

• Most developers have weak program comprehension
  skills
• A group of inspectors in a particular company was
  trained in program comprehension
• focus shifted from process to product
• built mental models and understood the
  translations from one model to the next
• inspectors developed their own checklists from
  these models
• there was a 90 reduction in post-release errors
  when inspectors used program comprehension
  approach compared to two other groups of
  inspectors in the same company

49
Modeling the Inspection Artifact

• Example DaimlerChrysler requirements inspections
• Create a model of the requirements specifications
  
• e.g. using UML, usually sequence diagrams
• Two inspectors read specifications
• focus on qualities such as correctness,
  consistency, testability, maintainability
• prepare slides about the created model for the
  subsequent inspection meeting
• document any defect found on a prescribed form
• inspectors dont need inspection specific
  training
• technique is effective at operationalization of
  given qualities?

50
Task Directed Inspection (TDI)

• What is TDI?
• inspectors produce a usable product for future
  evolution of the software system
• inspectors expertise is matched to the task
• used with a light-weight inspection process
• focus on the work of the individual rather than
  that of the team
• based on observation that creating careful
  documentation often shows up defects in product
  being documented
• Case Study at OHN
• inspection exercise piggybacked on an existing
  software development activity

51
Mechanics tools

• Comments about tools here are focused on code
• use tools to make inspection more productive
• reduce tedious trivia burden
• for code clean up, restructuring
• enforce code styles
• e.g. enforce comment style consistency
• some style errors slip past
• automatic generation of documentation
• e.g. call trees, flowcharts, UML diagrams, ...
• find some syntactic errors
• tools cannot find errors that result in correct
  syntax
• tools can get confused in complex code
• report false positives
• miss real issues
• still need someone to read the code
• asynchronous inspections virtual meetings
• can reduce inspection intervals, but no synergy
• current research on Computer Supported
  Collaborative Work (CSCW) may improve inspections
• still very much in research phase

52
Inspection Metrics defect data

• Robert Grady of HP 1996 34
• software defect data is the most important
  available management information source for
  software process improvement decisions
• ignoring defect data can lead to serious
  consequences for an organizations business

53
Inspection Metrics What can be collected?

• Time used
• hardware support for time tracking would help?
• e.g. palmtop
• Findings
• may generate change requests
• only warranted when change is large
• minor changes can be fixed in the inspection
  process
• Others counts, location, sources, impact, time
  to fix, role (user, developer, inspector,...),
  phase
• Data collection effort must be proportional to
  the value of the data

54
Inspection Metrics When There Is No Time

• Measurement gathering must be an integral part of
  the software development process
• must involve as little human effort as possible
• must be supported with tools if at all possible
• minimize the number of measurements gathered
• those gathered must be useful and useable
• there must be a payback on the metrics chosen, or
  the time needed to collect them is better spent
  elsewhere
• be willing to change the metrics being collected

55
Metrics Defect Classification

• Why classify and track defects?
• Sample defect categorization schemes
• Definition is a form of classification What is a
  defect?
• IEEE Standard 1044 Classification for Software
  Anomalies
• Defect Classification for Defect Causal Analysis
• What to do about classification when there is no
  time?

56
Why classify and track defects? (1)

• Project management reasons
• allocation of resources
• allocation of priorities
• monitoring project progress
• Project execution reasons
• guide the selection of test cases
• guide what to inspect for
• Responsiveness to customers
• nothing falls off the list
• dont make the same mistake again
• which types of defects cause greatest downtime

57
Why classify and track defects? (2)

• Process improvement reasons
• identify what parts of process are injecting
  defects
• analyze faulty work processes and improve them
• identify activities which
• capture specific defect classes
• capture more defects
• identify effective personal practices
• analysis of steps in process
• do the steps meet their goals?
• evaluate impact of changes to development process
• defect classification as part of a measurement
  program

58
Why classify and track defects? (3)

• Product improvement reasons
• defect profiles for specific components/modules
• statistical assessment of software
• Research
• evaluation of methodologies
• evaluation of VV techniques
• identifying different types of defects
• differentiating types of defects found by
  different inspection techniques

59
Sample defect classification schemes

• IBM ODC
• http//www.research.ibm.com/softeng/ODC/DETODC.HTM
  
• Hewlett Packard
• Boris Beizer
• Defect Classification for Computational Codes

60
IBM ODC Categories

• Defect Removal Activities
• Triggers
• Impact
• Target
• Defect Type
• Qualifier
• Age
• Source

61
IBM ODC defect types for code and design

• 1. Assignment/Initialization
• 2. Checking
• 3. Algorithm/Method
• 4. Function/Class/Object
• 5. Timing/Serial
• 6. Interface/O-O Messages
• 7. Relationship

62
HPs Categorization Scheme 1987
63
Beizers taxonomy (1)

• Developed as a statistical framework for
  determining testing strategy
• Four levels of detail with eight categories at
  top level
• 1. Requirements and Features
• defects related to the requirements specification
• 2. Functionality and Features
• completeness, correctness, input domain, error
  messages
• 3. Component Structure
• control flow, case selection, initialization,
  processing
• 4. Data
• definition, structure, or use of data

64
Beizers taxonomy (2)

• 5. Implementation
• typographical errors, standards violation,
  documentation
• 6. Integration
• integration and interfaces between components
• 7. System and Software Architecture
• architectural errors that affect entire system
• 8. Test Definition Execution
• definition, design, execution of tests or data
  used in tests
• Example
• 4232 Data(4) Access and Handling(2)
  Dimension(3) Initialization(2)

65
Defect Classification for Computational Codes
DC-CC (1)

• Description
• structured to have multiple levels of detail
• five categories at top level
• documentation
• calculation/logic
• error checking
• support code
• external resources
• as currently defined, three more levels each
  providing more detail
• five defect qualifiers associated with each
  category
• missing, wrong, superfluous, inconsistent, obscure

66
Defect Classification for Computational Codes
DC-CC (2)

• Additional classification as a tool for research
• DC-CC is overlaid with levels of understanding
• Comparative level C
• consistency of code with documentation
• Identifier level I
• consistency of variable naming with use
• Structural level S
• data and logical structures that support code
  architecture
• Logical level L
• calculation and logic correct

67
Definitions

• What is a Defect?
• too many terms
• examples
• Simple and complex definitions
• Our definition

68
What is a Defect?Too many terms
error
issue
botch
problem
defect
incident
delusion
fault
bug
elision
finding
complaint
goof
slip
mistake
flaw
failure
trouble
boner
error
anomaly
gripe
glitch
blunder
howler
oversight
69
What is a Defect? Examples (1)

• Difficulty in categorizing 35
• one-character error in source statement
• passes syntax checking
• data is corrupted in another area of the system
• function then performs incorrectly
• is this a typing error, coding error, data error
  or functional error?
• is the error due to inadequate data validation,
  inadequate code review?
• Defect or opportunity?
• Help system could be improved
• Scale issues
• e.g. typing error in internal comment
• group all typing errors as a single defect?
• e.g. system fails to complete computations in
  required time limit
• several components may be contributing to problem

70
What is a Defect? Examples (2)

• Definition/scope/details of defect may change
  after initial report filed
• e.g. variable used multiple times but only
  guarded (e.g. range checked) in one case
• missing guards?
• unnecessary guard?
• missing documentation for a special case?
• e.g. setting of a flag indicating a fatal error
  is commented out
• mistake?
• undocumented change of error handling style?
• kluge to stop the program from terminating?

71
What is a Defect?

• An argument may be made that its not a defect
  since it is ...
• not a fault
• defects may not lead to failures
• e.g. incorrect comment
• but this may lead indirectly to a failure
• code is changed to be consistent with comment
• not an error
• defects may be inadvertent
• e.g. new version of a component or a platform
• sometimes not clear
• e.g. dead code that may have a purpose
• really an enhancement
• e.g. limitations on current functionality

72
What is a defect? Definitions(1)

• Simple definitions
• Gilb 4 (1)
• a defect is identified as something that is
  incorrect
• Gilb 4 (2) for defect prevention process
• anything that impacts or interferes with
  maintaining or improving quality and production
• Fredericks and Basili 29 trying to find a
  consistent terminology
• any fault, failure, or error occurring in a
  software system

73
What is a defect? Definitions (2)

• Definitions from standards 36
• The non-fulfillment of intended usage
  requirements. ISO8402, QMPPQuality Management
  for Projects and Programs, Lew Ireland Project
  Management Institute
• Any non-conformance of a characteristic with
  specified requirements. MIL-STD 105, QMPP
• Any condition or characteristic in any supplies
  or services furnished by the contractor under the
  contract that is not in compliance with the
  requirements of the contract. GATGlossary of
  Acquisition Terms, US federal govt
• A substandard condition. CSM Centre for Systems
  Management

74
What is a defect? Definitions (3)

• Definition from Robert Grady at HP 37
• A defect is a deviation from the product
  specification or an error in the specification if
  the error could have been detected and would have
  been corrected. If the error could not possibly
  have been detected, or it could have been
  detected and would not have been corrected, then
  it is an enhancement, not a defect. Defects do
  not include typographical or grammatical errors
  in the engineering documentation.

75
What is a defect?Our Definition
A defect is an observable property of code or
other work product that could diminish the
desired level of any quality factor defined for
the software system.
Quality factors (Ilities) Timeliness,
Functionality, Cost, Correctness, Reliability,
Maintainability Usability, Flexibility,
Adaptability, Testability, Readability,
Portability, Device independence, Self
containedness, Reusability, Interoperability,
Efficiency, Device efficiency, Integrity,
Security, Accuracy, Robustness, Completeness,
Consistency, Accountability, Accessibility, Human
Engineered, Self descriptive, Fault tolerant,
Structuredness, Conciseness, Legibility,
Augmentability, Modifiability, Understandability,
Clarity, Resilience, Validity, Generality,
Minimality, Modularity, Expandability,...
76
IEEE Standard 1044 - Classification for Software
Anomalies 38

• Describes classification itself as a process with
  four steps
• recognition
• when anomaly found
• investigation
• propose solutions
• action
• all activities to prevent occurrence of similar
  anomalies
• disposition
• At each step
• record
• classify
• identify impact

77
Defect Classification for Defect Causal Analysis
(DCA) 39

• Make use of most widely available types of
  quality information - the software problem
  report
• Classify or group problems
• classify problem by programmer when implementing
  the fix
• define or choose a scheme before starting DCA
• most useful dimensions for classification
• when was the defect injected into the software?
• when was the defect detected?
• what type of defect was introduced?

• Identify systematic errors
• Determine principle cause
• Develop corrective actions

78
What to do about classification when there is no
time?

• Keep classification system very simple
• e.g. Major, Minor
• Track only defects in progress
• dont keep historical data unless it has value
• small defects that can be quickly fixed are not
  tracked, other than by the existence of new
  versions
• Classify only when there is a very clear purpose
• stop classifying when purpose is achieved
• Use classification to save time
• make judicious use of defect causal analysis on
  subsets of defects

79
Eight Maxims of Inspection

• Observations from the three experiments conducted
  at Ontario Hydro
• 1. Specify the goals of the inspection
• define all terminology
• e.g. What does maintainability or correctness
  mean?
• e.g. What is a defect?
• specify style guides if any
• determine scope of inspection
• do inspectors look at associated pieces of the
  product?
• 2. Identify required skills
• specific technical skills linked to goals
• communications skills
• 3. Clean up the product to be inspected
• support judicious use of the inspectors time
• use tools to clean up the product under
  inspection
• supply complete and consistent documentation
• if possible, make product author available as a
  resource

80
Eight Maxims of Inspection

• 4. Use structured inspection techniques
• provides guidance for breadth and depth of
  understanding of product
• encourages better coverage of the product
• 5. Build a process based on inspection
• product author knows the product will be
  inspected
• infrastructure in place to leverage benefits of
  inspection
• identify the best time to carry out the
  inspection
• 6. Give inspectors responsibility and authority
• a sense of ownership encourages dedicated work
• allow inspectors access to all resources needed

81
Eight Maxims of Inspection

• 7. Ensure inspectors have the space to inspect
• mental space to be able to focus
• physical space free from distractions
• schedule space to get the job done
• 8. Use metrics cautiously when assessing the
  effectiveness of inspection activities
• defect count is an ambiguous and ill-defined
  metric
• too many variables affect time-to-inspect
• what about subconscious thought time??

Diane Kelly and Terry Shepard, "Eight Maxims for
Software Inspectors, Journal of Software
Testing, Verification and Reliability, Volume
14, Issue 4 , pp. 243 256, Dec. 2004
82
How to do Inspections When there is No Time
If you dont have time to do it right ...
you must have time to do it over.
83
References 1

• Phillip M. Johnson, Reengineering Inspection,
  Communications of the ACM, Feb. 1998, Vol 41, No
  2, pp 49-52
• Oliver Laitenberger, Khaled El Emam, Thomas
  Harbich, An Internally Replicated
  Quasi-Experimental Comparison of Checklist and
  Perspective-Base Reading of Code Documents, IEEE
  Transactions in Software Engineering, May 2001,
  Vol. 27, No. 5, pp.387-421
• Lawrence G. Votta, Does the Modern Code
  Inspection Have Value? Presentation at the NRC
  Seminar on Measuring Success Empirical Studies
  of Software Engineering March 1999
  http//www.cser.ca/seminar/ESSE/slides/ESSE_Votta.
  pdf

84
References 2

• Tom Gilb and Dorothy Graham, Software Inspection,
  Addison Wesley, 1993. see also http//www.result-p
  lanning.com/
• Lawrence G. Votta Does Every Inspection Need a
  Meeting?, SIGSOFT93 - Proceedings of 1st ACM
  SIGSOFT Symposium on Software Development
  Engineering, ACM Press, New York, 1993, pp
  107-114
• David L. Parnas, David M. Weiss Active Design
  Reviews Principles and Practice, Proceedings
  8th International Conference on Software
  Engineering, Aug. 1985

85
References 3

• J.C.Knight, E.A. Myers An Improved Inspection
  Technique, Communications of the ACM, Nov. 1993,
  Vol 36, No 11, pp. 51-61
• Adam A. Porter, Harvey P. Siy, Carol A. Toman,
  Lawrence G. Votta An experiment to Assess the
  Cost-Benefits of Code Inspections in Large Scale
  Software Development, IEEE Transactions on
  Software Engineering, Vol 23, No 6, June 1997, pp
  329-346
• Y. Chernak A Statistical Approach to the
  Inspection Checklist Formal Synthesis and
  Improvement IEEE Transactions on Software
  Engineering, 22(12)866-874, Dec. 1996

86
References 4

• University of Maryland, Notes on Perspective
  based Scenarios online http//www.cs.umd.edu/p
  rojects/SoftEng/ESEG/manual/pbr_package/node8.html
  Nov. 1999
• Adam A. Porter, Lawrence G. Votta, Victor R.
  Basili Comparing Detection Methods for Software
  Requirements Inspections A Replicated
  Experiment, IEEE Transactions on Software
  Engineering, Vol 21, No 6, June 1995, pp 563-575
• Diane Kelly, Terry Shepard Task-Directed
  Software Inspection Technique An Experiment and
  Case Study, Proceedings IBM CASCON, Nov. 2000

87
References 5

• Glen W. Russell, "Experience with Inspection in
  Ultralarge-Scale Developments", IEEE Software,
  Jan. 1991, pp.25-31
• R. Chillarege, et al., "Orthogonal Defect
  Classification - A Concept for In-Process
  Management", IEEE Transactions on Software
  Engineering, v. 18 n.11, Nov 92, pp. 943-956
• Kent Beck, Extreme Programming Explained Culture
  Change, Addison Wesley, 1999
• Robert B. Grady, Successful Software Process
  Improvement, Prentice Hall, 1997

88
References 6

• Philip M. Johnson and Danu Tjahjono, Does Every
  Inspection Really Need A Meeting?, Journal of
  Empirical Software Engineering, 4, 1, pp 9-35,
  Jan. 1998
• David A. Wheeler, Bill Brykczynski, and Reginald
  N. Meeson Jr., Software Inspection An Industry
  Best Practice, IEEE CS Press, 1996
• Gregory Abowd, Len Bass, Paul Clement, Rick
  Kazman, Linda Northrop, Amy Zaremski
  Recommended Best Industrial Practice for
  Software Architecture Evaluation, Technical
  Report, CMU/SEI-96-TR-025, January 1997

89
References 7

• Stan Rifkin, Lionel Deimel Program
  Comprehension Techniques Improve Software
  Inspections A Case Study, Proceedings IWPC 00,
  IEEE 2000
• Steve McConnell, Quantifying Soft Factors, IEEE
  Software, Nov/Dec 2000, pp 9-11
• Terry Shepard, Margaret Lamb, and Diane Kelly,
  More Testing Should be Taught, Communications
  of the ACM, June 2001, 44 (6), pp. 103-108
• Michael Cusumano, Richard Selby, Microsoft
  Secrets, Simon Shuster Inc., 1995

90
References 8

• Edward Kit, Software Testing in the Real World -
  improving the process, Addison-Wesley, 1995
• Robert L. Glass, Inspections - Some Surprising
  Findings, Communications of the ACM, April 1999,
  Vol. 42, no. 4, pp.17-19
• Adam Porter, Lawrence Votta, Comparing Detection
  Methods for Software Requirements Inspections A
  Replication Using Professional Subjects,
  Empirical Software Engineering Journal, 1997

91
References 9

• Victor Basili, Scott Green Oliver Laitenberger,
  Filippo Lanubile, Forrest Shull, Sivert
  Sorumgard, Marvin Zelkowitz, The Empirical
  Investigation of Perspective-Based Reading,
  Empirical Software Engineering An International
  Journal, 1(2), 1996, pp.133-164
• Oliver Laitenberger, Thomas Bell, An Industrial
  Case Study to examine a non-traditional
  Inspection Implementation for Requirements
  Specifications, Proceedings 8th IEEE Symposium
  on Software Metrics, June 2002, pp.97-106

92
References 10

• Michael Fredericks, Victor Basili, Using Defect
  Tracking and Analysis to Improve Software
  Quality A DACS State-of-the-Art Report, Rome
  NY, 1998
• M.E. Fagan, Design and Code Inspections to
  reduce Errors in Program development, IBM
  Systems Journal, Vol. 15, No. 3, 1976,
  pp.182-211
• M.E. Fagan, Advances in Software Inspections,
  IEEE Transactions on Software Engineering,
  Vol.12, No. 7, July 1986, pp.744-751
• Watts S. Humphrey, A Discipline for Software
  Engineering, Addison Wesley, 1995

93
References 11

• Stefan Biffl, Michael Halling, Investigating the
  Influence of Inspector Capability Factors with
  Four Inspection Techniques on Inspection
  Performance, Eight IEEE Symposium on Software
  Metrics, 2002, pp. 107-117
• Robert B. Grady (1996). Software Failure
  Analysis for High-Return Process Improvement
  Decisions. Hewlett-Packard Journal, 47(4)
  (August)
• 35 Boris Beizer, Software Testing Techniques,
  2nd ed., Van Nostrand Reinhold, NY, 1990

94
References 12

• http//www.pmforum.org/library/glossary/
  PMG_D00.htm
• Robert Grady, Deborah Caswell Software Metrics
  Establishing a Company-Wide Program,
  Prentice-Hall, 1986
• IEEE Std 1044-1993 IEEE Standard Classification
  for Software Anomalies
• David Card, Learning from our Mistakes with
  Defect Causal Analysis, IEEE Software Jan-Feb
  1998, pp.56-63

95
References 13

• http//www.research.ibm.com/softeng/ODC/DETODC.HTM
  and FAQ.HTMconcepts
• Barry Boehm, Victor Basili Software Defect
  Reduction Top 10 List IEEE Computer, January
  2001, pp.135-137
• Ram Chillarege, Inderpal Bhandari, Jarir Chaar,
  Michael Halliday, Diane Moebus, Bonnie Ray,
  Man-Yuen Wong Orthogonal Defect Classification
  - A Concept for In-Process Measurements IEEE
  Transactions on Software Engineering, Vol.18,
  No.11, Nov. 1992, pp.943-956

96
References 14

• Jarir Chaar, Michael Halliday, Inderpal Bhandari,
  Ram Chillarege In-Process Evaluation for
  Software Inspection and Test, IEEE Transactions
  on Software Engineering, Vol.19, No.11, Nov.
  1993, pp.1055-1070
• Karl Weigers Process Impact. Review Checklists,
  http//www.processimpact.com/process_assets/review
  _checklists.doc
• Diane Kelly, Terry Shepard, A Case Study in the
  Use of Defect Classification in Inspections,
  Proceedings IBM CASCON, Toronto, November 2001

97
References 15

• Victor Basili, David Weiss, A Methodology for
  Collecting Valid Software Engineering Data, IEEE
  Transactions on Software Engineering, Vol. Se-10,
  No.6, Nov. 1984
• Mary Jean Harold, Testing A Roadmap,
  Proceedings ICSE 2000
• Khaled El Emam, Software Practices and Project
  Success, Cutter Consortium, Agile Project
  Management Advisory Service Executive Update,
  Vol. 6, No. 17, 2005