Evaluation of Information Systems Quality Management and Earned Value

1
Evaluation of Information SystemsQuality
Management and Earned Value

• INFO 630
• Glenn Booker

2
Quality Management System

• Every organization should have a Quality
  Management System (QMS)
• The QMS describes the quality organization
  (members, duties, and to whom they report) and
  activities which they need to perform
• Quality Assurance, Quality Management, and
  Quality Control are often used interchangeably
  for the QMS name

3
Typical QMS Activities

• Activities by a quality management organization
  typically include
• Process Evaluation to determine if process is
  clear, and consistent with other related
  processes
• Process Assessment - help ensure processes meet
  quality standards (CMMI, ISO 9000)

4
Typical QMS Activities

• Process Verification to ensure that processes
  are actually being used
• Product Evaluation to inspect documents or
  other products (SW, HW) to verify their quality
• Think of the tag Inspected by 19 in new clothes
  thats a product evaluation

5
QMS Activities

• Quality control for manufacturing, assembly,
  shipping, receiving, etc.
• Support definition, collection, and analysis of
  metrics for each organization
• The quality organization should be able to stop
  any activity if the product quality is inadequate
  (what defines inadequate?)

6
Quality Road Map for Software Maintenance

• An organizations Quality Management System may
  be guided by some sort of Quality Road Map (high
  level plan)
• The Quality Road Map is the basis for each
  releases Quality Plan
• The Road Map is the highest level corporate
  strategy for improving the product

(Warning IBM terminology here)
7
Quality Road Map

• The Road Map should systematically introduce more
  comprehensive quality activities and measures
• Let each team develop own quality goals
• Consider process, product, people, and tools how
  will each contribute to quality?
• FAA uses 3-year overall plan, with one-year goals
  to help meet it

8
Supplier Buy-in

• Your quality efforts must be matched by your
  suppliers
• This must be stated in the Quality Plan or
  equivalent
• Hence the quality measurements need to fulfill
  the overall goals of the organization, and define
  objectives for each release or product, and for
  major suppliers

9
Tracking, Measurement, Analysis

• These are key to making quality management work
• Software development must become quantitatively
  predictable to be regarded as a science
• Large volume of data not needed instead,
  focused, accurate, useful data is better

10
Customer Problems

• Problems include defects, user errors, usability
  problems, unclear instructions, duplicates of
  known problems, etc.
• Usually express as Problems per User Month (PUM)
• PUM (Problems reported) / (total
  license-months)

11
Customer Satisfaction

• Often measured with 5-point Likert scale (with
  optional Net Satisfaction Index (NSI))
• Very Satisfied (NSI 100)
• Satisfied (NSI 75)
• Neutral (NSI 50)
• Dissatisfied (NSI 25)
• Very Dissatisfied (NSI 0)

12
Customer Satisfaction

• Note that average NSI is a weak metric, since it
  hides the type of distribution
• The average NSI for half Very Satisfied and half
  Neutral customers looks the same as for all
  Satisfied
• Like measuring the mean of a distribution,
  without measuring the standard deviation (width
  of the bell curve)

13
In-Process Quality Metrics

• Defect density during automated (machine) system
  testing
• Good for comparing different releases of one
  product, or very similar products
• Defect arrival rate
• of defects found vs. time (e.g. week, month)
• Cumulative defects found vs. time
• Look for stability to decide done with testing

14
In-Process Quality Metrics

• Clarify actual defects, vs. reported problems,
  vs. backlog of defects
• Actual defects measures how many known defects
  have been found in the code
• Reported problems includes code defects, user
  errors, etc.
• Backlog shows how fast defects fixed, as well as
  found

15
In-Process Quality Metrics

• Phase-based defect removal pattern tracks
  defects during the entire life cycle
• Ways to identify defects include requirements and
  design reviews, code inspections, formal
  verification, and all phases of testing
• Want most defects found early in project (key
  theme)

16
Software Maintenance Metrics

• Focus on fixing existing problems, not changing
  features or requirements
• Fix backlog number of outstanding known
  problems
• Backlog Management Index ( closed problems) /
  ( new problems) 100
• BMI gt 100 means backlog of problems is being
  reduced

17
Software Maintenance Metrics

• New, closed, and the status of problems can also
  be tracked
• Fix response time, measured by the mean time
  problems are open
• Percent delinquent fixes of problems which
  exceeded desired response time
• Note that this implies an active goal, since the
  expected fix time has been defined

18
Fix Quality

• Fixes are, in general, not all perfect - they may
  create new problems!
• Fix quality of fixes which are defective
  (i.e. result in new problems)
• Fix propagation rate average number of new
  problems resulting from a fix

19
What is Earned Value?

• Earned Value is a method for planning and
  tracking the cost, schedule, and technical
  accomplishments of a project
• Earned Value focuses on comparing the ACTUAL
  cost, schedule, and technical accomplishments to
  their BUDGETED (or planned) values

20
Earned Value
Earned Value balances measurement of cost,
schedule, and technical performance.
21
Why do we care?

• Some organizations are contractually required to
  use earned value to report their progress
• Payments are then based on the accomplishments
  claimed under earned value hence earned value
  reporting claims can be legally binding!

22
Foundation

• Earned value requires having a written plan which
  describes each task, before that task is
  performed.
• The plan must define, for each task
• Schedule (start, stop, and calendar duration)
• Cost (or earned value) it is worth, and
• Earned value measurement method (see later)

23
Budget Basics

• The total amount of money available on a contract
  is the Contract Budget Base (CBB)
• Wise managers leave a Management Reserve (MR) of
  about 10-15, and budget the rest to the
  projects Performance Measurement Baseline (PMB)
• Hence CBB MR PMB

24
Budget Basics

• Most mortals (project staff) are only aware of
  the PMB, and do not know the extent of the MR
• The PMB may be broken into the undistributed
  budget (a more visible reserve than MR), plus
  distributed budget (which is the sum of all
  tasks cost accounts)
• PMB undistributed budget distributed budget

25
Budget Basics
26
Earned Value Terminology
27
The Plan

• The project plan produces a curve for the BCWS
  (budgeted cost of work scheduled) through its
  planned completion. This curve is typically
  S-shaped, and is plotted as cost versus time
  (e.g. K versus months).
• The total expected cost of the task, located at
  the end of the BCWS curve, is the BAC (budget at
  completion)

28
Budgeted Cost of Work Scheduled
BAC
29
Tracking Progress

• As the task is being performed, measure the
  amount of money actually being expended (ACWP,
  actual cost of work performed)
• Keep track of the tasks which have actually been
  worked on, and look up how much earned value you
  get credit for that work (BCWP, the budgeted cost
  of work actually performed)

30
Tracking Progress

• Plot the ACWP, BCWP, and BCWS over time - note
  that ACWP and BCWP can only be shown to date,
  whereas the BCWS covers the entire tasks
  schedule
• BCWS is the original project plan
• ACWP is how much real money has been spent
• BCWP is how much real work has been accomplished
  with that money

31
Earned Value Example
32
Analysis

• At any time during the project
• Cost Variance (CV, in ) ACWP - BCWP
• Schedule Variance (SV, ) BCWS - BCWP
• Schedule Slip (time) (current date) minus
  (date when current value of BCWP was found on
  the BCWS curve)
• Positive numbers are bad for all of these
  negative is good

33
Projection

• The ACWP curve is used to estimate when the task
  will actually be completed
• The projected ACWP curve is called Estimate To
  Completion (ETC)
• The end of the projected ETC curve is the
  Estimate At Completion (EAC)
• Thats the expected end of the project (in and
  schedule)

34
Projected Project Completion
35
Project Completion

• Compare BAC (the originally planned completion of
  the project) and EAC (the current estimated
  completion of the project)
• Projected Program Delay (schedule overrun) Time
  difference between BAC and EAC
• Variance At Completion (VAC, cost overrun) Cost
  difference between BAC and EAC

36
Earned Value Summary

37
Earned Value Measures
Key measures well use are SPI and CPI
38
More Earned Value Measures
39
What are good values?
Per the U.S. Marine Corps Acquisition Procedures
Handbook, June 1997
40
Projection

• Another method for projecting the end of the
  actual project is
• Plot CPI and SPI over time
• Based on the trends seen, estimate the CPI and
  SPI at the end of the project
• EAC(cost) BAC(cost) / CPI
• EAC(schedule) BAC(schedule) / SPI

41
Example - Railroad Project

• Given the following project build 4 miles of
  railroad track in 4 weeks for 4 million.
• Actual performance to date is After 3 weeks, 2
  million has been spent and one mile of track has
  been built.
• Pure cost analysis would only show that the
  project has spent 1 million less than expected
  (3M - 2M). Is this good?

42
(No Transcript)
43
EAC
BAC
44
Example - Status Analysis

• Todays earned value status
• BCWS 3M (expected expenditure work)
• ACWP 2M (actual expenditure)
• BCWP 1M (actual work accomplished)

45
Example - Status Analysis

• Cost Variance (CV) ACWP - BCWP 1M
• Schedule Variance BCWS - BCWP 2M
• Schedule Slip (3 weeks) - (1 week) 2 weeks

46
Example - Status Analysis

• Earned value analysis
• Planned 1M/week expenditure for 4 weeks
• Hence planned 3M after 3 weeks (BCWS)
• Earned value of only 1M (BCWP is 66 behind the
  plan)
• Actual expenditure of 2M to date (ACWP has 100
  overrun, given how much was accomplished)

47
Example - Projected Outcome

• Find EAC using SPI and CPI
• Only are given one data point for status, so use
• SPI BCWP/BCWS 1M / 3M 0.333
• CPI BCWP/ACWP 1M / 2M 0.500

48
Example - Projected Outcome

• BAC was 4 weeks and 4 million
• EAC is
• EAC(cost) BAC(cost)/CPI 4 million / 0.500
  8 million
• EAC(schedule) BAC(schedule)/SPI 4 weeks /
  0.333 12 weeks
• Hence projected cost overrun is 4M, and schedule
  overrun is 8 weeks

49
Measurement Methods

• The methods used to take credit for earned value
  are of key significance
• It is critical that earned value claims are
  accurate and consistent
• Credit, or Earned Value, for performing each task
  may be obtained in many different ways, depending
  on the type of task.

50
Measurement Methods

• Milestone Events
• Weighted Milestone
• Fixed Formula
• Percentage Complete Estimates
• Percentage Complete and Milestone Gates

• Equivalent Completed Units
• Earned Standards
• Apportioned Relationship to Discrete Work
• Level of Effort

51
Milestone Events

• Milestone events earn 100 credit when
• The event has occurred, AND
• All follow-up work is done (e.g. preparing and
  distributing minutes after a formal review) AND
• All action items are closed
• Until all of the foregoing activities are
  completed, no earned value may be taken

52
Weighted Milestone

• Useful for short-span tasks
• Weighted budget amounts are applied to milestones
  distributed across the duration of the task
• As each milestone is reached, the budget amount
  is earned

53
Fixed Formula

• Useful for detailed short-span tasks
• An amount ranging from 0 to 100 percent of the
  task's budget is earned when the task begins
• The remaining percentage is earned when the task
  is complete, e.g. 0/100, 25/75, or 50/50
• These measures are typically used to track earned
  value on nonrecurring tasks

54
Percentage Complete Estimates

• Easiest method to administer
• A "subjective" estimate of the percentage of
  work completed is used for the earned value
• Requires well-defined work packages and
  guidelines to determine an accurate
  percent-complete value
• This is the biggest source of inconsistencies in
  earned value reporting

55
Percentage Complete

• For example, define the following allowable
  earned value for each phase of new feature
  development
• Analysis and design are ready for peer review up
  to 30 earned value
• Successful completion of peer review 35
• Preparation for customer review 45

56
Percentage Complete Estimates

• Completion of customer review and approval to
  proceed 55
• Coding and unit testing completed 75
• Integration testing completed 85
• System testing completed 95
• New feature released 100

57
Percentage Complete and Milestone Gates

• Popular method used by government organizations -
  uses a combination of weighted milestones and
  percent complete
• "Subjective" percent-complete estimates are
  allowed up to a specific ceiling associated with
  each milestone
• Advancement past the milestone is not allowed
  until tangible criteria have been met, hence the
  term "gate"

58
Equivalent Completed Units

• Useful for extended duration and repetitive tasks
• The overall project is divided into distinct
  units of accomplishment
• Earned value is computed by summing the units
  completed

59
Earned Standards

• Useful for production-type work
• Perhaps the most sophisticated method to compute
  earned value
• Standards of performance based on historical cost
  data, time and motion studies, etc., are used to
  compute the earned value on a given task...

60
Earned Standards

• Often, several standards are used simultaneously,
  and a management consensus determines which
  standard is ultimately used
• These measures are typically used to track
  earned value on either nonrecurring or recurring
  tasks

61
Apportioned Relationship to Discrete Work

• Useful for tasks in which their performance has a
  direct relationship to other tasks
• The earned value for apportioned tasks is a
  summary of the earned value measurements made on
  the work to which it is related
• Schedule variances in the apportioned work are
  usually identical to schedule variances in its
  related work

62
Apportioned Relationship to Discrete Work

• However, cost variances in the apportioned work
  are substantially different from the related work
  because of the dynamics of actual costs
• This measurement can employ any of the previous
  six methods

63
Level of Effort

• Level of effort tasks are those earn value based
  entirely on the actual amount of effort expended,
  hence whatever is set as the planned value is
  always the earned value, regardless of what work
  was done
• This method is generally not recommended to
  track earned value no accountability!
• Used mainly for pure research or overhead tasks
  (e.g. clerical support)

64
References

• Earned Value Management
• Sean Alexander, Earned Value Management Systems
  Basic Concepts
• Wayne Abba, The New Management Revolution
  Integrating Contractor Performance with Agency
  Budgets
• See Balanced Scorecard for another management
  measurement system