Software Engineering Process I Quality Management

1
Software Engineering Process IQuality Management

• INFO 636
• Glenn Booker

2
Software Quality Management

• The PSP focuses on managing defects to produce a
  quality product
• This, in turn, helps improve productivity
• Productivity still falls as the size of a system
  increases, due to its increasing complexity
• To improve further, must improve your process
  quality

3
What is Quality?

• Crosby considered software quality to be
  conformance to requirements
• Meeting user needs is often key to determine
  quality
• We can distinguish between the quality of the
  product created, and the quality of the process
  used to create it

4
Product Quality

• A software product must provide the functions
  defined in its requirements, and must be a
  functioning application
• Beyond that, how sensitive the user is to other
  aspects of usability, reliability, compatibility,
  etc. varies wildly
• Poor quality products tend to have a lot of
  focus on testing to find defects

5
Process Quality

• Process quality focuses on meeting the
  developers needs
• A quality process supports creating good products
• Process consistency and repeatability are
  generally critical
• Efficiency and usability are less obvious
  objectives

6
Cost of Quality

• Quality is a big cost issue for software
• Each defect has costs for detecting it, finding
  the source, determining a solution, fixing the
  problem (reqts, design, and/or code), reviewing
  the fix, testing the fix to see if it works,
  testing the fix to check for regression problems,
  and updating documentation to reflect the fix

7
Cost of Quality

• The relative cost of finding and correcting a
  defect increases through the life cycle
• During requirements or design phases, a defect
  has a relative cost of about 1
• Coding and testing phases, cost 10-40
• After release to the customer, cost 100-1000

8
Cost of Quality

• Many of you also noted that the time to fix
  defects found in testing or later is much higher
  than those found earlier (like pages 276-7)
• PSP experience has shown that reviews are twice
  as efficient at finding defects as testing

9
Testing Schedules

• Many organizations focus on getting into system
  testing quickly, as a sign of progress
• However, more time in requirements and design
  would produce higher quality code which needs
  less testing time

10
Cost of Defect Removal

• Typical developers inject (create) about 100
  defects per thousand LOC
• About half are found by compiler
• The rest need to be found by inspection or
  testing
• Using the PSP, about 70 of defects entering
  inspection are found

11
Cost of Defect Removal

• Inspection and review average about ½ hour per
  defect, whereas testing averages 8-10 hours per
  defect so the cost benefit of inspection
  becomes clear

12
An Example

• Consider a 50,000 LOC project
• About 5000 defects are initially created, but
  2500 are found by the compiler (and are
  presumably fixed immediately)
• Without reviews or inspections, those 2500
  defects would take at least 25008 20,000 hours
  to find and fix using testing alone

13
An Example

• Using inspections, we would expect to find 1750
  of those defects, and fix them in ½ hour each
  for 875 hours
• The remaining 750 defects would be found by
  testing, and fixed in 75086000 hours
• Total defect removal time with inspection and
  test is 6,875 hours, versus 20,000 hours for
  testing alone

14
Cost of (Low) Quality

• Poor quality can be measured to determine its
  impact on the project
• Failure cost the cost of finding and repairing
  failures
• Appraisal cost the cost of conducting reviews
  (called), and testing which finds no defects
• Prevention cost the cost of preventing the
  failures from occurring

15
Cost of Quality (COQ)

• For the PSP we simplify these to
• Failure cost time for compile and test
• Appraisal cost time for reviews and inspections
• Prevention costs are low, since we have few
  projects to examine for defect trends
• Based on this, define COQ measures

16
COQ Measures

• Failure COQ 100(compile time test
  time)/(total devel time)
• Appraisal COQ 100(design review time code
  review time)/(total devel time)
• Total COQ Failure COQ Appraisal COQ
• Appraisal cost as of total quality
  100(appraisal COQ)/(total COQ)
• A/FR ratio Appraisal COQ/(Failure COQ)

17
COQ Measures

• The purpose of these measures is to help look for
  trends in quality over several projects
• Hope to see higher appraisal cost as of total
  quality over time
• This should lead to lower defect rates for later
  projects

18
Quality Strategy

• Important to keep product and process measures
  separate both are important
• Avoid productivity measures by themselves they
  can help with planning, and be compared to other
  measures, but dont use them as a sole objective

19
Process Yield

• Total process yield is helpful the percent of
  defects removed before test
• You can also evaluate yield for every life cycle
  phase, but only after hidden defects have been
  later discovered
• The life cycle is a filtering process
• Each phase has some kind of step to help find
  and remove defects to keep from feeding them
  into the next phase

20
No One Answer

• The PSP is a personal process for many reasons
  one is that there is no one best way to find and
  remove defects
• Need to look for the best ways for you to improve
  your work
• Then reevaluate your strategy periodically

21
Benchmarking

• It can be helpful to compare your product and
  process measures to other peoples, or to
  industry standards
• For example, plot the A/FR ratio over time it
  should increase (p. 288)
• Or look at A/FR versus the yield
• Or Test defects per kLOC versus A/FR

22
Benchmarking

• Productivity can be compared to other measures,
  such as yield, to look for trends
• Yield acts like the filter part of the life
  cycle process
• Each steps yield reduces the number of defects
  to be found later, so a small improvement in
  yield can be significant

23
Injection and Removal

• The key to understanding your software
  development process is to balance the injection
  of new defects with activities to detect and
  remove them
• To achieve greater control, need to examine the
  factors which cause defects to occur

24
Defect Causes

• Education didnt know how to do it
• Communication werent informed about something
• Oversight forgot to do something
• Transcription knew what to do, but made a
  mistake
• Process the process misled your actions

25
Yield Management

• While the overall process yield doesnt care if
  you have 80 review yield, and 40 test yield, or
  40 review and 80 test, the costs and A/FR ratio
  of those situations are quite different
• This is why so much emphasis is placed on the
  quality of early reviews (see recommendations on
  pp. 298-300)

26
Defect Prevention

• Finding and fixing defects is purely defensive
• To be more proactive, need to prevent defects
  from being created
• Analyze the types of defects which occur
  frequently in your experience
• Determine the causes of common defect types (see
  slide 24 again)

27
Defect Prevention

• Then look for ways to prevent those defects from
  being made again, e.g.
• Improve review checklists
• Get additional instruction
• Conduct more detailed design
• Get independent reviews
• And change your processes accordingly

28
Defect Prevention

• The critical follow-up is to re-examine your
  defects later, and see if your changes helped or
  not
• Incidentally, defect prevention and continuous
  process improvement are the highest levels of
  process maturity
• CMM and CMMI level 5 activities