Estimating the Cost of a Defense Contract with Performance Indices Do the Old Methods Still Work

1
Estimating the Cost of a Defense Contract with
Performance IndicesDo the Old Methods Still Work?

• David S. Christensen, Ph.D.
• Southern Utah University
• [phone]
• ChristensenD_at_suu.edu
• Air Force Institute of Technology
• 20 May 2002

2
Overview

• Background
• Earned Value Management Systems
• Variance Analysis
• Evaluating the EAC
• Are the rules still valid?

3
Earned Value Management Systems (EVMS)

• Links
• http//www.acq.osd.mil/pm
• http//www.suu.edu/faculty/christensend/ev-bib.htm
  l

4
Cost Management Reports(DOD 5000.2-R)

• Contract Cost Data Report
• Contract Fund Status Report
• Cost Schedule Status Report
• Cost Performance Report

5
Cost/Schedule Status Report

• Monthly
• Cost thresholds(FY00)
• Prod lt 315 MIL
• RD lt 73 MIL
• FFP contracts normally exempt

6
Cost Performance Report

• Monthly
• Cost thresholds
• Prod gt 315M
• RD gt 73M
• FFP normally exempt
• EVM criteria required

7
Objectives of EV Criteria

• To provide insight into contract progress,
  implementing EVMS provides the program manager
  with data which
• Relate time-phased budgets to specific contract
  tasks/work
• Indicate work progress
• Properly relate cost, schedule, and technical
  accomplishment
• Are valid, timely, and auditable
• Supply managers with information at a practical
  level of summarization and
• Are derived from the same internal EMVS used by
  the contractor to manage the contract

8
Cost Performance Report

• Does it provide data or information?
• Is it reliable?
• Is it relevant?

9
Decision Usefulness
Relevance
Reliability
Predictive
Feedback
Timely
Verifiable
Valid
Neutral
Value
Value
ACCESS
SAME
OBJECTIVE
MONTHLY
VARIANCE ANALYSIS
EAC
DATABASE
DATA
Comparability/Consistency
DIDs, CWBS
Benefits gt Costs
THRESHOLDS, SUMMARY REPORTS
10
THE BASIC DATA ELEMENTS

• Actual Cost of Work Performed (ACWP)
• Budgeted Cost of Work Scheduled (BCWS)
• aka Planned Value
• Budgeted Cost of Work Performed (BCWP)
• aka Earned Value

11
THE BASIC DATA ELEMENTS

• Estimate At Completion
• (EAC)
• Budget At Completion
• (BAC)

12
THREE VARIANCES

• Variance at Completion BAC - EAC
• Cost Variance
• BCWP - ACWP
• Schedule Variance
• BCWP - BCWS

EAC
VAC
BAC
ACWP
CV
BCWS
SV
BCWP
TIME
NOW
13
Performance Measurement Baseline (PMB)

EAC
BAC
PMB
ACWP
BCWS
BCWP
TIME
NOW
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
JAN FEB MAR APR MAY JUN
DESIGN
MATERIAL
ORDER
ASSEMBLY
TEST
4. SCHEDULE WORK PACKAGES
18
(No Transcript)
19
JAN FEB MAR APR MAY JUN
DESIGN
9
3
2
MATERIAL
2
ORDER
4
7
ASSEMBLY
9
15
5
1
TEST
3
3
6. SPREAD BUDGET OVER TIME
20
JAN FEB MAR APR MAY JUN
DESIGN
2
9
3
MATERIAL
ORDER
4
7
2
ASSEMBLY
9
15
5
1
TEST
3
3
2
17
BCWS
13
19
8
4
7. CALCULATE MONTHLY BCWS
21
PERFORMANCE MEASUREMENT BASELINE (PMB)
60
50
40
30
20
10
0
JAN
FEB
MAR
APR
MAY
JUN
8. CREATE PMB
22
Numerical Example

• The Plan
• Build 5 sections of railroad track in 5 weeks for
  5000
• BCWS per week ?
• BAC ?
• Performance
• After two weeks, one section is completed for
  2600
• Cumulative ACWP ?
• Cumulative BCWS ?
• Cumulative BCWP ?
• EAC ?

23
Numerical Example

EAC ?
BAC 5000
ACWP 2600
BCWS 2000
BCWP 1000
5 Weeks
2 Weeks
24
Variance Analysis
An example from an aircraft project
25
(No Transcript)
26
Level 4 detail supporting Level 3, Air Frame
27
Level 5 detail supporting Level 4, Crew Station
28
(No Transcript)
29
Evaluating the Estimate At Completion Using
Earned Value Performance Indices
30
Estimate At Completion(EAC)

• The EAC appears on the Cost/Schedule Status
  Report the Cost Performance Report
• Accuracy controlled by EVMS Criteria
• Factors to consider
• Actual Costs to date Overhead and Inflation
  rates
• Performance to date Future performance on work
• Cost and schedule variances Changes to
  requirements
• Reliability and relevance of data Organizational
  culture

31
EAC Formula
EAC Costs to date Estimated Cost of
Remaining Work
Predicted final cost

Plan
Cost to date
Performance to date
now
Time
32
EAC Formula
EAC ACWPcum (BAC - BCWPcum)/Performance
Factor
EAC

BAC
ACWP
BCWP
now
Time
33
Performance Factor

• Usually a performance index
• CPI
• SPI
• Combination of CPI and SPI
• w1(CPI) w2(SPI), where w1 w2 1
• CPI x SPI

34
Cost Performance Index
Example Data BCWSc 10 million BCWPc 8
million ACWPc 12 million BAC 20 million LRE
25 million
CPI BCWP/ACWP 8 / 12 0.67 Interpretation C
umulative, Current, Average
35
Schedule Performance Index
Example Data BCWSc 10 million BCWPc 8
million ACWPc 12 million BAC 20 million LRE
25 million
SPI BCWP / BCWS 8 / 10 0.8 Interpretation
Cumulative, Current, Average Value is 1 at end
of contract
36
BCWS BCWP BAC when all work is completed
EAC

BAC
BCWS
ACWP
BCWP
now
Time
37
Combinations of CPI and SPIw1(CPI) w2(SPI)
Example Data BCWSc 10 million BCWPc 8
million ACWPc 12 million BAC 20 million LRE
25 million

• .8(CPI) .2(SPI) is an AF favorite
• Always between CPI and SPI
• .8 (.67) .2 (.8) 0.69

38
Combinations of CPI and SPIw1(CPI) w2(SPI)
Sometimes based on Percent Complete
Weight
.8
SPI
CPI
.2
0 100
39
Combinations of CPI and SPICPI x SPI
Example Data BCWSc 10 million BCWPc 8
million ACWPc 12 million BAC 20 million LRE
25 million

• An OSD favorite
• When CPI lt1 and SPI lt 1, SCI ltlt1
• 0.67 x 0.80 0.53

40
Twelve index-based EAC formulas
Which one is best?
41
A-12 CPR Data (April 1990, MIL)
Which EAC is best?
42
(No Transcript)
43
Evaluating the EAC
1. Compare the CV to the VAC CV VAC Implication
-459 -354 LRE too small DOD Experience Once a
contract is more than 15-20 complete, the final
overrun will be worse than the present overrun
(Christensen/Wilson 1992)
44
Evaluating the EAC
2. Compare the CPI with the TCPILRE CPI TCPI Im
plication 0.7646 1.043 LRE is too
small TCPILRE (BAC - BCWPcum) / (LRE -
ACWPcum) (4046 - 1491) / (4400 -
1950) DOD Experience Once a contract is over
20 compete, the cum CPI does not change by more
than 10 percent, and in most cases it only
worsens (Christensen/Heise 1993)
45
Evaluating the EAC
3. Compare to a range of independent
EACs LRE IEAC floor IEAC ceiling Implication 4400
5292 6612 LRE is too low DOD
Experience CPI-based EAC is floor to final
cost SCI-based EAC is often the most accurate
estimate (Christensen 1996)
46
Evaluating the EAC
DOD Experience No single EAC formula is always
best. (Christensen, Antolini, McKinney 1992)
47
Evaluating the EAC
DOD Experience No single EAC formula is always
best. (Christensen, Antolini, McKinney 1992)
48
Organizational Culture and the EAC

• Accuracy controlled by EVMS Criteria
• Factors to consider
• Actual Costs to date Overhead and Inflation
  rates
• Performance to date Future performance on work
• Cost and schedule variances Changes to
  requirements
• Reliability and relevance of data Organizational
  culture

49
Organizational Culture and the most likely EAC

• Program managers do not support EACs most likely
  to be experienced on their programs.
• Instead they support EACs most likely to be
  tolerated by OUSD and Congress.

50
Why were the contractor and government EACs the
smallest?
51
How did the government and contractor estimates
compare to this range?
(Christensen, 1996)
52
(Christensen, 1996)
53
Percent Complete
(Christensen, 1994)
54
EAC Evaluation MethodsDo They Work On Post-A12
Contracts?
David S. Christensen, Carl Templin, and David
Rees Southern Utah University [phone]
ChristensenD_at_suu.edu 2002 National Conference of
the Society of Cost Estimating and
Analysis Scottsdale, Arizona (June 11-14, 2002)
55
References

• Christensen, David S. and Carl Templin. EAC
  Evaluation Methods Do They Still Work?
  Acquisition Review Quarterly (forthcoming)
• Christensen, David S., and David A. Rees. Is The
  CPI-based EAC a Lower Bound to the Final Cost of
  Post-A12 Contracts? Journal of Cost Analysis and
  Management (forthcoming)
• Earned value literature bibliography
• http//www.suu.edu/faculty/christensend/ev-bib.htm
  l

56
Background

• A-12 cancellation a catalyst for change
• The A-12 EAC was too low
• Realistic EACs were suppressed
• DOD-wide epidemic (Beach 1992)
• DoD policy changes to achieve cost realism
• Have not worked previously (Drezner et al 1993)
• Will they work this time?

57
Background

• Current reform initiatives are improving cost
  performance (Coopers and Lybrand 1997)
• Methods to evaluate EACs which assume
  deteriorating cost performance may no longer be
  valid

58
EAC Evaluation Rules

• The final cost variance (in dollars or as a
  percentage) will be worse than the cost variance
  at the 20 percent completion point
• The cumulative cost performance index (CPI) will
  not change by more than 0.10 from its value at
  the 20 percent completion point, and in most
  cases only worsens
• The EAC computed using the cumulative CPI is a
  reasonable lower bound to final cost
• Based on observations made by OUSD validated by
  researchers at AFIT on contracts completed in
  1970s and 1980s cited by Beach
• A12 Example
• CV20 -459M Estimated
  Final Cost Variance -354M
• CPI20 0.765 To
  Complete Performance Index 1.043
• CPI-based EAC 5,292M Contractor EAC
  4,400M

59
Hypotheses

• The final cost variance will be worse than the
  cost variance at the 20 completion point
• CVfinal CV20 lt 0
• CVfinal CV20 lt 0
• CV Earned Value Actual Cost
• CV (CV / Earned Value) x 100

60
Hypotheses

• The cumulative CPI will not change by more than
  0.10 from its value at the 20 completion point,
  and in most cases only worsens
• CPIfinal CPI 20 lt 0.10
• CPIfinal CPI 20 lt 0
• CPI Cumulative Earned Value / Cumulative Actual
  Cost

61
Hypotheses

• The EAC computed using the cumulative CPI is a
  reasonable lower bound to final cost
• CPI-based EAC lt Final Cost
• To control for differences in contract size,
• DAC lt 0
• EAC (BAC Earned Value)/ Cum CPI Actual
  Cost
• Deviation at completion (DAC) (EAC Final
  Cost) / Final Cost

62
More Definitions

• CV20 Cumulative CV at 20 percent completion
  point
• CVfinal Cumulative CV at end of contract
• CPI20 CPI at end of contract
• CPIfinal CPI at end of contract
• Percent Complete Cumulative Earned Value / BAC

63
Hypothesis testing

• Averages of the differences were computed and
  hypotheses were tested at alpha of .05 using
  paired-t test
• Sensitivity of results were tested across
• Contract phase (development versus production)
• Contract type (fixed price versus cost
  reimbursable
• Managing service (Army, Air Force, Navy)

64
The data

• DAES database
• Based mostly on CPR data
• Its reliability is controlled via EVMS Criteria
• Every completed contract was included with actual
  cost, earned value, and BAC at 20 complete and
  at completion, resulting in
• 147 pre-A12 contracts (finished before A12
  cancellation)
• 52 post A12 contracts (started after A12
  cancellation)
• 42 transition period contracts (started before
  and finished after A12 cancellation)
• 31 Dec 91, about one year after the A-12
  cancellation, was chosen as the cut-off date

65
Results

• Cost variances did not worsen significantly on
  Post-A12 contracts from the 20 completion point
• The cumulative CPI is a reasonable floor to final
  cost in the early and middle stages of contract
  life on Post-A12 contacts
• Cost performance during the Transition period
  contracts was worse than cost performance during
  Pre-A12 and Post-A12 periods
• Results generally insensitive to contract size,
  contract phase, and military service

66
The Final Cost Variance Will Be Worse Than The
Cost Variance At The 20 Percent Completion Point
67
The Final Cumulative CPI Will Not Change By More
Than 0.10 From Its Value At The 20 Completion
Point, And In Most Cases It Only Worsens
68
The Cumulative CPI-based EAC Is A Floor To Final
Cost
69
Conclusion

• Cost performance on post-A12 contracts is
  improving
• The A-12 cancellation changed the acquisition
  culture
• The EAC evaluations should continue to be used
• Acquisition reform initiatives can adversely
  affect cost performance of on-going contracts