CSTE Domain 9

1
CSTEDomain 9

• Defect Tracking
• and Correction

Presented by Mike Webb, for the SISQA CSTE Study
Group
2
Study Guidelines

• A major test objective is to identify defects.
  Once identified, defects need to be
• recorded
• monitored
• reported
• corrected
• The CSTE candidate should be skilled in the
  entire defect process.

3
Part 1 Overview of Defect Management Process

• Primary goal is to prevent defects
• Based on risk assessment and impact
• Integrated into development process
• Uses automated defect capture and analysis, where
  possible
• Used to improve the development process

4
Part 2 - The Defect Management Process

• Defect prevention
• Deliverable baselines
• Defect discovery
• Defect naming
• Defect resolution
• Process improvement
• Management reporting

5
(No Transcript)
6
Defect Prevention

• Best approach to defects is to eliminate them
  altogether!
• Since technology doesnt exist yet to eliminate
  all defects, strategies are needed to find
  defects quickly and minimize their impact.
• High priority is to identify and implement the
  best defect prevention techniques

7
(No Transcript)
8
Identify Critical Risks

• Identify the risks that pose the largest threat
  to the system.
• Risks vary based on type of project, technology,
  users skills, etc.
• Dont try to identify every conceivable risk,
  just those critical to success of the project.

9
Examples of Critical Risks

• Missing a key requirement
• Critical application software or vendor supplied
  software doesnt function properly
• Software doesnt support major business
  functions, requiring reengineering of the
  software
• Unacceptably slow performance
• Hardware malfunction
• Hardware and/or software integration problems
• Users unable or unwilling to embrace new system

10
Estimate Expected Impact

• For each critical risk, an assessment can be made
  of the impact, in dollars, if the risk does not
  become a problem, and the probability that the
  risk will become a problem.
• The product of these two numbers is the expected
  impact.
• Precision is not important. What is important is
  the order of magnitude of risk.

11
Annual Loss Expectation Formula (ALE)

• ALE is one of the more effective methods of
  estimating the expected impact of a risk.
• ALE dollar loss per event times number of
  events per year.
• Estimated loss can be calculated by using average
  loss for a sample of loss events.

12
Factors Influencing Expected Impact

• Probability that the risk will become a problem
• How long it takes to recognize the problem
• How long it takes to fix the problem
• Knowing the impact provides insight into how to
  reduce the risk

13
Minimize Expected Impact

• Eliminate the risk, or at least reduce risk
• Reduce scope of system
• Avoid untested technology
• Reduce possibility of risk becoming a problem
• Inspections
• Testing
• Reduce impact if there is a problem
• Contingency plans
• Disaster recovery plans

14
Two Ways to Minimize Risk

• Reduce expected loss per event
• Reduce frequency of an event
• If both are reduced to zero, the risk is
  eliminated
• If loss per event is reduced, impact is reduced
  when problem does occur.
• If frequency is reduced, probability of risk
  becoming a problem is reduced.

15
Defect Prevention Techniques

• Quality Assurance
• Ensure that processes used are adequate to
  produce the desired result
• Train and Educate the Work Force
• Better training leads to higher quality
• Train and Educate Customers
• Use creative strategies, such as more elaborate
  Help, cue cards, multimedia tutorials, etc.
• Standardize the Process
• Methodology and standards produce a repeatable
  process that prevent defects from reoccurring
• Defensive Design and Defensive Code
• Design such that two or more independent parts of
  system must fail before a failure can occur.
• Use Assertions (code which brings unexpected
  conditions to the attention of the programmer or
  user)

16
Deliverable Baselining

• A deliverable is baselined when it reaches a
  predefined milestone in development.
• A milestone involves transferring the product
  from one stage of development to the next.
• Cost of making changes and fixing defects
  increases at each milestone.
• Deliverable should be baselined when changes or
  defects can have an impact on deliverables on
  which other people are working.

17
Baseline Activities

• Identify key deliverables
• Determine the point in the development process
  where the deliverable will be baselined
• Define standards for each deliverable
• Define the requirements and criteria that must be
  met before deliverable can be baselined

18
Defects and Baselining

• A defect exists only when one or more baselined
  components does not satisfy its requirements.
• Errors caught before baseline are NOT considered
  defects.

19
Defect Discovery

• A defect is discovered only when the defect has
  been formally brought to attention of developers,
  AND developers acknowledge that the defect is
  valid.
• Finding a problem is NOT discovering a defect!

20
(No Transcript)
21
Step 1 Find Defect

• Two usual methods for finding a defect
• Preplanned activities intended to uncover
  defects, ie inspection or testing.
• By accident, such as user reports

22
Defect Finding Techniques

• Static techniques
• Deliverable is examined (manually or by a tool)
  for defects
• Examples reviews, walkthroughs, inspections
• Dynamic techniques
• A deliverable is used to discover defects
• Example testing
• Operational techniques
• An operational system contains a defect found by
  users, customers, or control personnel.
• Found as a result of a failure

23
Comments About Defect Finding Techniques

• An effective defect management program requires
  each of the three categories
• In each category, the more formal the integration
  of the techniques, the more effective they are
• Static techniques generally find defects earlier,
  so they are more efficient
• Inspection is effective at removing defects.
• Inspection is also a good way to train new staff
  in best practices and the functioning of the
  system being inspected.

24
Step 2 Report Defect

• Developers must quickly become aware of the
  defect
• If found during the defect finding phase,
  complete and submit a defect report
• If found by other means, plan for other reporting
  methods, such as computer forums, email, call
  center, etc

25
Step 3 Acknowledge Defect

• Developer must decide if defect is valid, usually
  by trying to reproduce the problem
• Submitter can help the developer by including
  details in the defect report about how to
  reproduce the defect

26
Strategies to PinpointCause of a Defect

• Instrument the code to trap environment state
  when error occurs
• Write code to check the validity of the system
• Analyze reported defects. If not reproducible,
  look for patterns in similar defect reports.

27
Defect Naming

• Level 1 Name of the defect
• Gather representative sample of defects, from
  help desk, QA, project teams, etc.
• Identify major developmental phases and
  activities. Start with broad list, then reduce to
  no more than 20 items.
• Sort identified defects by the phase/activity in
  which they were found.
• Within each phase/activity, categorize defects
  into groups with similar characteristics. Follow
  Paretos rule About 80 will fall into very few
  groups, and remain 20 will be widely dispersed
  among other groups. Call these all other.

28
Defect Naming (continued)

• Level 2 Developmental phase of Activity
• The phase should coincide with the organization
  development methodology e.g. business
  requirements, technical design, development,
  acceptance, installation or an activity such as
  data preparation.
• Level 3 Defect Category
• Suggested defect categories for each phase
• Missing
• Inaccurate
• Incomplete
• Inconsistent

29
Defect Naming Example

• If an incorrect requirement was found, it could
  be named as
• Level 1 Incorrect requirement
• Level 2 Requirements phase
• Level 3 Inaccurate
• Note that Levels 2 and 3 are qualifiers to the
  Level 1 name.

30
(No Transcript)
31
Prioritize Fix

• Consider these questions
• Is this a new defect, or previously submitted?
• What priority should be given to fixing defect?
• Are there actions to minimize impact of defect
  prior to the fix? Is there a workaround?

32
Three-level Prioritization Method

• Critical Defect would have serious impact on
  organizations business operation
• Major Defect would cause output of software to
  be incorrect or stop execution
• Minor Defect doesnt directly affect user, such
  as documentation error, or cosmetic GUI error

33
Schedule Fix

• Schedule the fix based on priority of the defect
• All defects are not created equal from the
  perspective of how quickly they need to be fixed.

34
Fix Defect

• Correct the problem
• Verify the fix
• Review test data, checklists, documentation, etc.
  (and enhance them if needed), in order to find
  similar defects earlier in the future.

35
Report Resolution

• After defect is fixed and verified, the
  developers, users, management, and others need to
  be notified that the defect has been fixed.
• Include details such as the nature of the fix,
  when and how the fix will be released.

36
Process Improvement

• Take the opportunity to learn how to improve the
  process and prevent potentially major failures.
• Even if defect was not critical the fact that
  there was a defect is a big deal.
• It is only the developers good luck that
  prevents a defect from causing a major failure.
• Review the process which originated the defect.
• Review the validation process, which should have
  caught the defect earlier in the process.
• If defect got this far before being caught, what
  similar defects may not have been discovered yet?

37
Two Viewpoints of a Defect

• The producers view
• A defect is a deviation from specifications,
  whether missing, wrong, or extra.
• The customers view
• A defect is anything that causes customer
  dissatisfaction, whether in the requirements or
  not. This view is known as fit for use.

38
Purposes of Defect Reporting

• To correct the defect
• To report status of the product
• To gather statistics used to develop defect
  expectations in future products
• To improve the development process

39
Examples of Required Data for Defect Reports

• Defect ID number
• Descriptive defect name and type
• Source of defect, ie test case or other source
• Defect severity
• Defect priority
• Defect status, ie open, fixed, closed, etc
• Date and Time tracking
• Detailed description, including how to reproduce
• Component or program where defect was found
• Screen prints, log files, etc
• Stage of origination
• Person assigned to research and/or correct the
  defect

40
Sample Defect Tracking Process

• Execute test, compare actual results to expected
  results. If discrepancy found, log it with status
  of open.
• Test Manager or Tester reviews defect report with
  developer to determine if really a defect.
• Assign defect to developer for correction.
  Developer fixes problem, and changes status to
  fixed or retest.
• Defect goes back to test team for retest.
  Regression testing performed as needed.
• If retest results match expectations, update
  status to closed. If still not fixed, change
  status back to open and send back to developer.
• Repeat 3-5 until problem is resolved.

41
The End

• CSTE
• Knowledge Domain 9
• Defect Tracking
• and Correction