Reducing Error in Estimating Production Costs of Multiple Unit Procurements

1
Reducing Error in Estimating Production Costs of
Multiple Unit Procurements

• Scott M Vickers
• MCR Federal, Inc.
• [phone]
• svickers_at_bmdo.mcri.com

Army Conference on Applied Statistics Santa Fe,
New Mexico 22-26 October 2001
2
Presentation Topics

• Learning Curve models.
• Developing Learning Curve parameters.
• Shifting Reference Point from T1.
• How to select appropriate Reference Point.
• Demonstration of improvement achieved.

3
Standard Practice for Estimating Costs of
Multiple Unit Procurements

• Apply a cost improvement or Learning Curve (LC)
  rate to account for improvements in
• Management
• Engineering processes
• Production efficiency
• Experience has shown that unit costs decrease
  (although at a declining rate) during the
  production process - regardless of how long the
  production runs.
• There are two predominate schools of thought on
  how to apply LCs to estimate production cost.
• Cumulative Average Unit Cost Theory
• Unit Cost Theory

4
Cumulative Average Unit Cost Theory

• Cumulative Average Unit Cost (CAUC) Theory posits
  that CAUC of successive production units decrease
  at a constant rate each time the production
  quantity is doubled.
• That constant rate is referred to as the CAUC
  Learning Curve Slope and is often expressed as a
  percent (e.g., 90).
• Standard form of the CAUC Theory equation is
• Y axb, where
• Y is Cumulative Average Unit Cost of x units.
• a is theoretical first unit cost (T1 or CAUC1).
• b is learning curve exponent,
• x is production quantity.

5
Total Production Costs using CAUC

• Total Production Cost (TPC) for x units
• Lot Total Cost (LTC)
• where j is last unit of lot in question and i is
  last unit of prior lot.
• Lot Average Cost (LAC) can be determined by
  dividing LTC by Lot Quantity (q).

6
Unit Cost Theory

• Unit Cost Theory posits that unit costs of
  production units decrease at a constant rate each
  time production quantity is doubled.
• That constant rate is referred to as the Unit
  Learning Curve Slope and is often expressed as a
  percent.
• The standard form of the Unit Cost Theory
  equation is very similar to the CAUC model.
• Y axb, where
• Y is Unit Cost of the xth unit.
• a is theoretical first unit cost.
• b is learning curve exponent,
• x is number of units produced.

7
Total Production Costs using Unit Theory

• Unit Theory is discrete so Lot Total Cost may be
  determined by summing unit costs for each unit
• It is often convenient to estimate lot
• costs using a continuous approximation
• of the discrete distribution.
• Where i is first unit and j is last unit of the
  lot

8
Developing a CAUC Model from Actual Data

• T1s and Learning Curve slopes can be derived from
  historical production costs. Here is a sample
  data set. (Cum Cumulative)

CAUC is the dependent variable. Cum Quantity is
the independent variable.
9
Curve Fitting Model

• Iteratively Re-weighted Least Squares (IRLS)
  Analysis
• Minimizes the sum of squared percentage error
• May be performed using Excel Solver.

Our first iteration uses values for a and b
derived from a log/log regression model to
determine predi(j-1) and then finds values of a
and b that minimize the squared percent error
function. We then iterate this process, each
time retaining our previous predictions in the
denominator, until differences between our new
predictions and previous predictions approach 0.

• IRLS has several desirable properties vis-à-vis
  log/log regression
• The minimization function is in unit space (vice
  log space).
• Weights each data point equally.
• Percent bias approaches 0.

10
The Covariance Matrix

• The covariance matrix can be developed as shown
• Adaptation of Dr. Matthew Goldbergs presentation
  at DODCAS 1999.

The covariance matrix enables us to develop
variability parameters for the IRLS coefficients.
11
Model Results Using IRLS
11
Using our sample data from Chart 8.
Nice tight interval on Learning Curve Slope, but
T1 value has wider variability.
Cost Formula CAUCn 3165.5 n(-0.213) error
Variability parameters tell us how well this
equation predicts CAUC for historical system, but
additional sources of variability are introduced
when predicting cost of a new system.
12
Estimating Costs of a New System

• Learning Curve Slope (b). Typical methods for
  estimating Learning Curve Slope include
• Analogy to another program
• Average of several similar programs
• Analyst Assumptions or Expert Judgement
• First Unit Cost (a). Typical methods for
  estimating First Unit Cost include
• Cost Estimating Relationship derived from
  historical data on earlier programs..
• Analogy to another program.
• Derived from prior data from same program.
• Annual Production Quantities (x). Usually
  determined by mission requirements and
  availability of procurement funding.
• So, how good are these methods? Lets look first
  at some Learning Curves derived from historical
  data.

13
Learning Curve Slopes for Missile Programs

• Study of missile programs shows that CAUC
  Learning Curve Slopes developed from historical
  data range widely.
• Slopes derived for 13 historical programs range
  from 95.8 to 75.5, with median slope of 84.0
• Ranges only slightly narrowed when stratified by
  contractor, developing service, missile type, or
  first year of manufacturing.
• When stratified using multiple categories, sample
  sizes are too small for analysis.
• Therefore cost analysts tend to look for closest
  analogy using multiple criteria - but we dont
  know how close the analogy fits the new program.
• Lets look at how much a cost estimate can be
  affected by choice of learning curve.

14
Impact of Learning Curve Choice

• Lets assume we know that first unit cost of a
  new missile is exactly 1.0M, and the production
  requirement is 5000 missiles.
• Lets accept the median historical LC slope (84)
  for our pre-production cost estimate, so that
• Total production estimate for 5000 missiles is
  then
• But if, when production begins, the contractor is
  only able to manage a 90 learning curve slope,
  and
• the actual production cost is now
• So we underestimated total production cost by
  783M!
• This error is too large, even for DoD cost
  estimates.
• How can we improve the estimate?

15
Mitigating Effects of Learning Curve Choice
Estimating production costs based on T1 magnifies
any error we make in selecting an appropriate
learning curve. The charts below illustrate how
much better the estimate can be if we move the
reference point away from T1.
90 Slope
90 Slope
84 Slope
84 Slope
500
As we move our reference point to the right
towards 5000 units, the cost estimate is impacted
less and less by a wrong choice in learning
curves.
16
What if the Reference Point is the Delivery
Quantity?

• The total production cost estimate will not be
  impacted by choice of learning curve.
• But we risk large errors in the cost estimate of
  early production lots, and this will cause
  budgeting problems.

T1
T1
84 Slope
90 Slope
17
Proposed Model Standard Form

• N is the number of production units.
• m is the Cost Reference Point.
• Tm is the CAUC of m units.
• For fitting a learning curve model
• Dependent variable is CAUC(N)
• Independent variable is (N/m)
• Analysis of data produces estimates for Tm and b.

18
Selecting a Good CAUC Reference Point (CRP)

• Desirable Characteristics
• Mitigates effect of choosing wrong learning curve
  for both
• Production Total Cost
• Annual Production Costs
• Somewhere between T1 and Total Delivery Quantity
• Robust for use with multiple programs - doesnt
  exceed total production requirement of most
  programs.
• Least possible error in estimating the CAUC at
  the Reference Point.

19
Least Possible Estimating Error is Important

• Cost Estimating Relationships (CERs) are
  influenced by Tm estimating error.
• CERs use physical properties or characteristics
  of systems to predict cost.
• They start with development of Tm estimates for
  several similar systems from historical data
  using learning curve models (usually at T1).
• They use these Tm estimates as dependent
  variables in regression models - usually assuming
  that the Tm values are known with certainty.
• Error in estimating Tm for the historical systems
  degrades the accuracy of the CER.
• If we use an analogy, the Tm is derived from a
  learning curve model for the analgous system.
• So, if we minimize the error in developing the Tm
  in our learning curve models, we improve the
  accuracy of our cost estimating reference point.

20
Minimizing Error in Estimating Tm

• Selecting an appropriate value for m can be
  done by examining SE of Tm estimates at various
  values of m.
• The table and chart below (based on the data in
  chart 8) show that SE is minimized near T500,
  and every value has lower SE than T1.
• In support of BMDO, we use T250 for missile
  programs based on relatively small SE and the
  anticipated procurement quantities of BMD missile
  systems.

21
Variability in Production Estimates Using T1

• Lets assume missile learning curves are
  triangularly distributed between 95.8 and 75.5,
  with a most likely value of 84.
• Lets assume our T1 is lognormally distributed
  with a Point Estimate of 1 and a 30 SE.
• Now lets randomly select a learning curve and T1
  from their respective distributions 5000 times in
  a simulation to model the distribution of
  production cost outcomes.
• Here are the simulation results
• Wouldnt you like to have an estimating
  methodology that narrows the range of probable
  outcomes better than this?

Statistic Value Trials 5,000 Mean 815.00 Media
n 652.44 STD 577.47 Skewness 1.79 Kurtosis 7.
33
Percentile M 10 276.04 30
459.26 50 652.44 70 928.32 90 1,585.77
The 10-90 Range is 1,309M.
22
Variability in Production Estimates Using T250

• Lets continue to assume missile learning curves
  are triangularly distributed between 95.8 and
  75.5, with a most likely value of 84.
• Lets assume our T250 is lognormally distributed
  with a Point Estimate of .25 and a 25 SE
  (Assumes that better knowledge of dependent
  variable gives us a 5 reduction in SE in the
  CERs as shown on chart 19).
• Now lets simulate the production run 5000 times.
• Here are the simulation results
• The T250 Reference Point substantially reduces
  the risk associated with our cost estimates.

Statistic Value Trials 5,000 Mean 535.11 Media
n 506.59 STD 176.49 Skewness 0.92 Kurtosis 4.1
4
Percentile M 10 336.69 30
425.01 50 506.59 70 602.59 90 773.36
The 10-90 Range is 337M, down almost 1,000M
from a T1 Cost Estimate.
23
Summary

• T1 is a poor reference point from which to start
  an estimate.
• It magnifies the impact of errors in selecting a
  learning curve slope.
• The SE in estimating a T1 is much larger than the
  SE for estimating Tm, where 1 lt m lt very large
  number.
• Using Tm as a reference point improves the
  accuracy the estimate.
• Error in selecting a learning curve is mitigated.
• Estimates of Tm are more precise and less
  influenced by the value of the learning curve
  slope.
• Provide a better basis for CER development.

Dont Use T1s in Cost Estimates