Training on Cost Estimation

1
Training on Cost Estimation Analysis

• Karen Richey
• Jennifer Echard
• Madhav Panwar

2
Outline

• Introduction to Cost Estimating
• Life Cycle Costs
• Data Collection
• Data Analysis
• Cost Estimating Methodologies
• Software Cost Modeling
• Cross-checks and Validation
• Risk and Sensitivity Analysis
• Documentation Requirements
• Cost Estimating Challenges
• Cost Estimating Auditor Checklists

3
Introduction to Cost Estimating

• The National Estimating Society has defined Cost
  Estimating1as
• The art of approximating the probable cost of
  something based on information available at the
  time.
• Cost estimating cannot
• Be applied with cookbook precision, but must be
  tailored to a particular system,
• Substitute for sound judgment, management, or
  control,
• Produce results that are better than input data,
  or
• Make the final decisions.
• Despite these limitations, cost estimating is a
  powerful tool because it
• Leads to a better understanding of the problem,
• Improves management insight into resource
  allocation problems, and
• Provides an objective baseline to measure
  progress.
• 1 The NES Dictionary, National Estimating
  Society, July 1982

4
Introduction to Cost Estimating

• The reliability of cost estimates varies over
  time.
• The closer you get to the actual completion of a
  project, the estimate becomes more accurate.
• Four types of cost estimates represent various
  levels of reliability .
• Conceptual Estimate Rough order of magnitude or
  back of the envelope.
• Often inaccurate because there are too many
  unknowns.
• Preliminary Estimate Used to develop initial
  budget, more precise.
• Detailed Estimate Serves as a basis for daily
  project control.
• Definitive Estimate Accuracy should be within
  10 of final cost.
• Important to repeat estimating process (i.e.,
  re-estimate) on a regular basis as more
  information becomes available
• This will keep estimate current as well as
  increase the accuracy

5
Introduction to Cost Estimating (contd)

• All cost estimates are constructed by the
  following tasks
• Identifying the purpose and scope of the new
  system.
• New software development, software reuse, COTS
  integration, etc.
• Choosing an estimate type.
• Conceptual, preliminary, detailed, or definitive
  type estimate
• Identifying system performance and/or technical
  goals.
• Laying out a program schedule.
• Selecting a cost element structure (CES).
• Collecting, evaluating, and verifying data.
• Choosing, applying, cross-checking estimating
  methods to develop the cost estimate.
• Performing risk and sensitivity analysis
• Time-phasing the cost estimate by fiscal year for
  cash flow purposes.
• Example 4 years to develop and 10 years
  operations and support beginning in FY2003
• Providing full documentation.

6
Life Cycle Costs

• Most cost estimates in the federal government
  represent total Life Cycle Costs (LCC).
• LCC estimates include all costs to develop,
  produce, operate, support, and dispose of a new
  system.
• Important to look beyond the immediate cost of
  developing and producing a system and consider
  all costs of a systems life cycle.
• What may appear to be an expensive alternative
  among competing systems may be the least
  expensive to operate and support
• Life Cycle Cost Estimates can be used to
• Compare various alternatives before committing
  funds to a project,
• Support Estimate-to-Budget transition after
  time-phasing to account for when funds will be
  spent.

7
Life Cycle Costs (contd)

• A LCC estimate is summarized using a detailed
  cost element structure (CES) that
• Identifies the activities required to complete
  project development and the effort, loading, and
  duration of each task,
• Provides a framework against which the cost
  estimate is organized.
• Enhances cost data collection and estimate
  reporting.
• Facilitates comparing estimates when a standard
  CES is used.

8
Life Cycle CostsCost Element Structure (CES)

• A CES provides a standard vocabulary for the
  identification and classification of cost
  elements to be used in cost estimating.
• Helps to identify costs that may be initially
  overlooked.
• Should be tailored for each project.
• The CES should be reviewed to ensure that there
  is no double counting of costs that could be
  allocated to more than one element.
• For example, logistics support costs could be
  included in the investment or operations and
  support phase.
• The CES is hierarchical in nature to accommodate
  early development (when relatively little data is
  available) through deployment, when more detailed
  data is available.

9
Life Cycle CostsExample of CES Elements (DOD)
1.0 Investment Phase 2.0 System Operations
Support Phase 1.1 Program Management 2.1
System Management 1.2 Concept Exploration 2.2
Annual Operations (supplies/spares) 1.3 System
Development 2.3 Hardware Maintenance 1.3.1
System Design Specification 2.4 Software
Maintenance 1.3.2 Development Prototype and
Test Site Investment 2.5 Outsource Provider
Support 1.3.3 Software Development 2.6 Data
Maintenance 1.3.4Training 2.7 Site
Operations (personnel, training,etc.) 1.3.6
Facilities 1.4 System Procurement 1.4.1
Deployment Hardware 1.4.2 Deployment
Software 1.4.3 Initial Documentation 1.4.4Logi
stics Support Equipment 1.4.5 Initial
Spares 1.4.6 Warranties 1.5 Outsource Provider
Investment 1.6 System Initiation, Implementation
Fielding 1.7 Upgrade (Pre-Planned Product
Improvement (P3I)) 1.8 Disposal Costs
10
Data Collection Data Required

• All CES elements will need data to support the
  estimate
• Historical cost and non-cost data need to be
  collected to support various estimating
  techniques
• Technical non-cost data describes the physical,
  performance, and engineering characteristics of a
  system
• Examples weight, of design drawings, SLOC,
  function points, of integrated circuit
  boards, square footage, etc.
• Important to pick data that is a predictor of
  future cost
• Important to have technical and schedule data
  because they act as cost drivers

11
Data CollectionData Required(contd)

• Identify both direct and indirect costs
• Direct costs are called touch labor and include
  direct manufacturing, engineering, quality
  assurance, material, etc. costs which have a
  direct bearing on the production of a good.
• Also included are direct non-wage costs such as
  training, supplies, and travel.
• Indirect costs are considered overhead and
  include such things as general administrative
  support, rent, utilities, insurance, network
  charges, and fringe benefits. These expenses are
  typically charged to a company as a whole.
• Example sick/annual leave, retirement pay,
  health insurance, etc.

12
Data CollectionData Required (contd)

• Some direct costs may be burdened with indirect
  costs and some may not
• Need to know to avoid double-counting or, worse
  yet, underestimating
• Important to ask when collecting data whether
  costs are burdened with indirect costs

13
Data CollectionData Required (contd)

• Data can be collected in a variety of ways
• Contractor site visits
• Data requests for all relevant CES elements
• Documented cost estimates, if available for
  earlier versions of the current system
• Can save valuable research time for statistical
  analysis
• Published cost studies
• Data collection is a critical and time consuming
  step in the cost estimating process!

14
Data Collection Typical data sources

• Two types of data sources
• Primary Secondary
• Primary data is found at the original source
  (e.g., contractor reports, actual program data,
  etc.)
• Preferred source of data
• Secondary data is derived from primary data of
  another similar system such as documented cost
  estimates, cost studies/research, proposal data,
  etc.
• Second choice to primary data due to data gaps
• May be best alternative when time constraints or
  data availability limit primary data collection

15
Data CollectionTypical data sources

• Current Program Milestone schedule (for phasing
  program costs over time)
• Current Program System Description
• Important to have a technical understanding of
  how the system will work
• Approved program funding (Compare to proposal
  estimate)
• Contractor actual cost reports (from internal
  Management Information System)
• Current program estimate documentation (if
  available)
• Contractor proposals (Compare to program funding
  profile)
• Commercial Off-the-Shelf (COTS) catalogs
• Manufacturer websites (e.g., Dell, Microsoft,
  etc.)

16
Data CollectionTypical Data Sources (contd)

• Forward Pricing Rate Agreements (FPRA)
• Similar program historical actual costs and
  estimate documentation
• Engineering drawings/specifications,
• Interviews with technical and program management
  personnel
• Surveys
• Professional journals and publications
• Industry guides and standards
• Technical Manuals

17
Data AnalysisData Validity/Integrity

• Important to ensure that the cost data collected
  is applicable to the estimate.
• Identifying limitations in the historical data is
  imperative for capturing uncertainty
• When using historical cost data for a similar
  system, appropriate adjustments need to be made
  to account for differences in the new system
  being estimated.
• Data may need to be mapped from the
  contractors accounting system to the Cost
  Element Structure (CES)

18
Data AnalysisData Validity/Integrity (contd)

• Proposal data should be validated to ensure that
  contractor motivations to buy-in or low-bid
  their estimates are not occurring.
• Compare previous contractor proposal bids and
  actual costs for similar programs.
• Look for trends in underbidding.
• Participate in a fact-finding trip to discuss
  contractor proposal estimates and gather
  supporting data/evidence.

19
Data AnalysisNormalization

• Involves making adjustments to the data so that
  it can account for differences in
• Inflation rates,
• Direct/indirect costs,
• Recurring and non-recurring costs,
• Production rate changes or breaks in production,
• Anomalies such as strikes, major test failures,
  or natural disasters causing data to fluctuate,
  and
• Learning curve (cost improvement) effects due to
  efficiencies gained from continually repeating a
  process
• Analysis of the data may indicate the need for
  more suitable data to add credibility to the
  estimate.

20
Data AnalysisNormalization (contd)

• Accounting for Economic Changes (e.g., inflation)
• Lack of cost data uniformity due to upward
  movement in prices and services over time.
• Index numbers are used to deflate or inflate
  prices to facilitate comparison analysis.
• Wrong to compare costs from 1980s to today
  without accounting for inflation over the past 20
  years.
• Cost estimators use inflation indices to convert
  costs to a constant year dollar basis to
  eliminate distortion that would otherwise be
  caused by price-level changes.
• Constant dollar estimates represent the cost of
  the resources required to meet each years
  workload using resource prices from one reference
  year (e.g., constant 2003 dollars).
• Constant dollars reflect the reference year
  prices for all time periods allowing analysts to
  determine the true cost of changes for an item

21
Data AnalysisNormalization (contd)

• For the United States, the Office of Management
  and Budget (OMB) is responsible for developing
  inflation guidance by appropriation for
  government estimates.
• Most common indices used by United States Cost
  Estimators are published by the U.S. Department
  of Labor, Bureau of Labor Statistics.
• Producers Price Index (http//www.bls.gov/ppi/)
  for goods
• Employment and Earnings Index (http//www.bls.gov/
  ces/home.htmdata) for services
• Important to pick the appropriate market basket
  of goods index that most closely matches the
  costs to be estimated.

22
Data AnalysisNormalization (contd)

• For budgetary purposes, however, data expressed
  in constant dollars should be inflated to
  represent current year (or Then-year) dollars.
• Current year estimates calculate the cost of the
  resources using the estimated prices for the year
  in which the resources will be purchased.
• Current year estimates reflect inflation
• Necessary to express estimates in current year
  dollars when requesting funding to avoid budget
  shortfalls.

23
Data AnalysisNormalization (contd)

• Example Assume 5 escalation rate compounded
  annually
• Base-year cost
  Then-year
  cost
• Current year 273,100 1.0000
  multiplier 273,100
• Current year 1 1,911,700 1.0500
  multiplier 2,007,285
• Current year 2 4,096,500 1.1025
  multiplier 4,516,391
• Current year 3 8,193,000 1.1576
  multiplier 9,484,217
• Current year 4 6,144,750 1.2155
  multiplier 7,468,944
• Current year 5 1,092,400 1.2763
  multiplier 1,394,230
• Total Estimated Labor cost (budget quality)
  25,144,167

24
Cost Estimating Methodologies

• Once data has been collected and normalized to
  constant dollars, there are five methodologies
  available for estimating costs
• Expert Opinion,
• Analogy,
• Parametric,
• Engineering, and
• Actual.

25
Estimating Methodology Considerations

• Choice of methodology is dependent upon
• Type of system
• Software, hardware, etc
• Phase of program
• Development, Production, Support
• Available data
• Historical data points from earlier system
  versions or similar system
• Technical parameters of system

26
Methodology Expert Opinion

• Often called Delphi method, proposed by Dr. Barry
  Boehm in his book, Software Engineering
  Economics.
• Useful in assessing differences between past
  projects and new ones for which no historical
  precedent exists.

27
Methodology Expert Opinion (contd)

• Pros
• Little or no historical data needed.
• Suitable for new or unique projects.
• Cons
• Very subjective.
• Experts may introduce bias.
• Larger number of experts will help to reduce bias
• Qualification of experts may be questioned.

28
MethodologiesExpert Opinion - Steps

• Gather a group of experts together,
• Describe overall program in enough detail so
  experts can provide an estimate,
• Each member of the expert group then does an
  independent of the resources needed,
• Estimates are gathered anonymously and compared,
• If there exists significant divergence among the
  estimates, the estimates will be returned to the
  expert group,
• The expert group then discusses the estimates and
  the divergence and works to resolve differences,
  and
• The expert group once again submits anonymous,
  independent estimates which continues until a
  stable estimate results.

29
Methodology Expert Opinion - Example

• Three software engineers are renown in the ERP
  software
• development.
• You hold interviews which each explaining the
  requirements,
• sizing level, and development process for
  your new system.
• Each member of the group submits their opinion
  of the final
• cost and the estimate converges to 50M.

30
MethodologiesAnalogy

• Estimates costs by comparing proposed programs
  with similar, previously completed programs for
  which historical data is available.
• Actual costs of similar existing system are
  adjusted for complexity, technical, or physical
  differences to derive new cost estimates
• Analogies are used early in a program cycle when
  there is insufficient actual cost data to use as
  a detailed approach
• Compares similarities and differences
• Focus is on main cost drivers.

31
MethodologiesAnalogy (contd)

• Often use single historical data point.
• May have several analogy estimates of sub
  elements to make up the total cost.
• Comparison process is key to success.
• May have to add or delete functionality from
  historical costs to match new program
• Consult technical experts for advice on
  complexity factors, technical, performance or
  physical differences.
• (not to be confused with expert opinion method)
• Impact of differences on cost is subjective.

32
MethodologiesAnalogy (contd)

• Good choice for
• A new system that is derived from an existing
  subsystem.
• Make sure actual cost data is available
• A system where technology/programmatic
  assumptions have advanced beyond any existing
  cost estimating relationships (CER).
• Secondary methodology/cross check
• Provides link between technical assumptions and
  cost.

33
MethodologiesAnalogy (contd)

• Pros
• Inexpensive
• Easily changed
• Based on actual experience (of the analogous
  system)
• Cons
• Very Subjective
• Large amount of uncertainty
• Truly similar projects must exist and can be hard
  to find
• Must have detailed technical knowledge of program
  and analogous system

34
MethodologiesAnalogy - Steps

• Determine estimate needs/ground rules,
• Define new system,
• Breakout new system into subcomponents for
  analogy estimating,
• Assess data availability of historical similar
  systems,
• Collect historical system component technical and
  cost data,
• Process/normalize data into constant year dollars
  (e.g., constant FY2003),
• Develop factors based on prior system,
• Example Program Management is 10 of total
  development cost
• Develop new system component costs.
• Obtain complexity (and other translation) factors
• Example Historical database is for 4 million
  records and new database will need to house 24
  million records gt complexity factor of 6 (46
  24) times the historical database cost
• Apply factors to historical costs to obtain new
  system costs

35
MethodologiesAnalogy - Example

• Your company is developing a new IT payroll
  system serving 5,000 people and containing
  100,000 lines of C code. Another company
  developed a similar 100,000 lines of code system
  for 20M for only 2,000 people. Your software
  engineers tell you that the new system is 25
  more complicated than the other system.
• Other system development cost was 20M.
• Estimated cost for new system 1.25 20M
  25M
• Plus 5,000/2,000, or 2.5 25M 62.5M total
  cost

36
MethodologiesParametric

• Utilizes statistical techniques called Cost
  Estimating Relationships (CER).
• Relates a dependent variable (cost) to one or
  more independent variables
• Based on specific factors that have a high
  correlation to total cost
• Number of software lines of code (SLOC) or
  function points,
• Square feet for office floor space,
• Number of floors in a high rise building for
  cabling estimates,
• Database size, etc.
• Can be used prior to development.
• Typically employed at a higher CES level as
  details are not known.
• Most cases will require in-house development of
  CER.

37
MethodologiesParametric (contd)

• Pros
• Can be excellent predictors when implemented
  correctly
• Once created, CERs are fast and simple to use
• Easily changed
• Useful early on in a program
• Objective
• Cons
• Often lack of data on software intensive systems
  for statistically significant CER
• Does not provide access to subtle changes
• Top level lower level may be not visible
• Need to be properly validated and relevant to
  system

38
Methodologies Parametric Example

• You have a previously developed CER to estimate a
  new IT system based on SLOC.
• Cost SLOC 25 /SLOC
• The CER is based on systems ranging from
  1,000,000 to 3,000,000 SLOC.
• You have estimated 2,600,000 SLOC for new system
• Cost 2,600,000 25 65M

39
MethodologiesParametric (contd)

• Cost estimators can develop their own CERs or
  they can use existing commercial cost models.
• Various Software cost estimating models will be
  discussed next
• Learning curves specialized type of CER.
• CERs can be cost to cost or cost to non-cost.
• Cost to Cost e.g., Manufacturing costs are 1.5
  times Quality Assurance costs
• Cost to Non-Cost /pound, or engineering
  hours/ of engineering drawings yields
  hours/drawing metric that can be applied to new
  program
• Factors and ratios are also examples of
  parametric estimating.

40
Methodologies Parametric CERs

• CERs are defined as a technique used to estimate
  a particular cost by using an established
  relationship with an independent variable.
• Can be as simple as a one variable ratio to a
  multi-variable equation.
• CERs use quantitative techniques to develop a
  mathematical relationship between an independent
  variable and a specific cost.

41
Methodologies Parametric CERs (contd)

• Reliable, normalized data is most important for
  CER development.
• Must determine range of data for which the CER is
  valid.
• Useful at any stage in a program.
• Typically CERs are the main cost estimating
  methodology in early stages of a program.
• In later stages of a program, CERs serve as a
  cross check to other methods
• Must be logically sound as well as statistically
  sound.
• High correlation (r2 0.75 or higher) for
  goodness of fit test
• Different statistical techniques may be used to
  judge the quality of the CER.
• Least squares best fit (regression analysis, or
  the ability to predict one variable on the basis
  of the knowledge of another variable)
• Multiple regression (a change in the dependent
  variable can be explained by more than one
  independent variable)
• Curvilinear regression (relationship between
  dependent and independent variable is not liner,
  but based on a curve)
• Learning curve (describe how costs decrease as
  the quantity of an item increases)

42
Methodologies Parametric CERs (contd)

• Statistics may be used to evaluate how well the
  CER will produce a reliable estimate.
• Coefficient of determination, R2
• Percent of the variation in the Y-data explained
  by the X-data, (ie. How close the points are to
  the line)
• Standard Error, SE
• Average estimating error when using the equation
  as the estimating rule
• Coefficient of Variation, CV
• SE divided by mean of the Y-data, relative
  measure of estimating error
• t-stat
• Tests whether the individual X-variable(s) is/are
  valid
• F-stat
• Tests whether the entire equation, as a whole, is
  valid
• No single statistic may validate or invalidate a
  CER, quality of the input data is just as
  important.
• Best to use a statistical software package like
  SAS or SPSS to quickly evaluate alternative CERs.

43
MethodologiesParametric CERs - Steps

• Define the dependent variable (e.g., cost
  dollars, hours, etc.) and what the CER will
  estimate,
• Select independent variables to be tested for
  developing estimates of the dependent variable,
• Variables should be quantitatively measurable and
  available
• If there is a choice between developing a CER
  based on performance or physical characteristics,
  performance characteristics are generally the
  better choice because they are known early on
• Collect data concerning the relationship between
  the dependent and independent variables,
• Most difficult and time consuming step, but
  essential that all data is checked to ensure that
  all observations are relevant, comparable, and
  free of unusual costs
• Explore the relationship between the dependent
  and independent variables,
• Use statistical analysis to judge strength of
  relationship and validity of equation
• Select the relationship that best predicts the
  dependent variable, and
• A high correlation often indicates that the
  independent variable will be a good predictive
  tool
• Document your findings.
• Identify independent variables tested, data
  gathered along with sources, time period
  (normalization for inflation effects), and any
  adjustments made to the data

44
Methodologies Parametric Learning Curves

• Basic premise
• Repetitive tasks should result in productivity
  for subsequent, similar tasks
• This improvement is usually quantified at a rate
  Y AXb
• In simplest terms, the cost of manufacturing or
  installing a unit should decrease as the number
  of units involved increases.
• As the number of units produced doubles, the cost
  per unit decreases by a fixed percentage
• The concept of learning curves is not new,
  originated in the mid-1930s with T.P. Wright in
  the Journal of Aeronautical Sciences.

45
MethodologiesParametric Learning Curve - Example
Say that the first 100 tasks of an installation
took 10 hours per task and the next 100 averaged
8 hours per task. Thus, the learning curve would
be calculated as follows Learning curve 8
hours per task/10 hours per task 0.8 Implies
an 80 learning curve meaning an improvement of
20 occurred between the first 100 tasks and next
200 tasks
46
Methodologies Rate Impact

• Basic premise The number of units produced in a
  single lot effects the overall cost of producing
  that lot.
• Costco theory that buying in bulk makes the unit
  cost less
• Mathematical equation showing rate impact along
  with learning curve
• Y AXbQr
• Both rate and learning curves can be impacted by
  the following
• Operator turnover rate (new employees do not meet
  expected productivity standards immediately)
• Production reworks
• Material handling and downtime (learning curves
  assume material is ready when needed)
• Engineering rework
• Rework of vendor supplied parts

47
MethodologiesEngineering

• Also referred to as bottoms up or detailed
  method.
• Underlying assumption is that future costs for a
  system can be predicted with a great deal of
  accuracy from historical costs of that system.
• Involves examining separate work segments in
  detail and then synthesizing these detailed
  estimates along with any integration costs into a
  total program estimate.
• Estimate is built up from the lowest level of
  system costs.
• Uses detailed cost element structure (CES).
• Must include all components and functions.
• Can be used during development and production.

48
MethodologiesEngineering (contd)

• Most useful for systems in production.
• Design configuration has stabilized
• Test results are available
• Development cost actuals are available
• Typically broken into functional labor categories
  e.g. engineering, manufacturing, quality control,
  etc.

49
MethodologiesEngineering

• Pros
• Objective
• Reduced uncertainty
• Cons
• Expensive
• Time Consuming
• Not useful early on
• May leave out software integration efforts

50
MethodologiesEngineering Steps

• Understand program requirements,
• Prepare program baseline definition,
• Define ground rules and assumptions,
• Develop detailed cost element structure,
• Develop functional estimates, and
• Use other program history
• Compile estimate data
• Develop rates and factors
• Incorporate supplier/subcontractor prices
• Include integration costs to glue the separate
  components into an integrated system (may need to
  use a CER for this estimate)
• Summarize estimate.

51
MethodologiesEngineering Example

• New IT system can be broken down into Labor and
  Material costs.
• Labor Estimated number of hours
• 250,000 hrs 200/hr 50M
• Material From vendor quotes
• Server 10M
• COTS 3M
• Desktops (50 _at_ 100k) 5M
• Cost 50M10M3M5M 68M

52
MethodologiesActual

• Bases future costs on recent historical costs of
  same system.
• Used later in development or production.
• Preferred method.
• Costs are calibrated to actual development or
  production productivity for your organization

53
MethodologiesActual

• Pros
• Most accurate
• Most objective of the five methodologies
• Cons
• Data not available early on
• Time consuming
• Labor intensive to collect all the data necessary

54
Methodologies Actual - Example

• The development process is nearing completion.
  The material has all been procured at a cost of
  20M.
• The labor cost to date is 30M. According to
  earned value cost performance reports (CPRs) the
  estimate at complete for the remainder of the
  labor is another 20M.
• Cost 20M 30M 20M 70M

55
Methodologies Summary of the Five Cost Estimate
Results

• Expert - 50M
• Analogy - 62M
• Parametric 65M
• Engineering 68M
• Actual - 70M

56
Software Cost Models

• Software costs as a percentage of total system
  costs continues to increase while associated
  hardware costs decrease.
• Accurately capturing software costs can be
  difficult and cost overruns often occur as a
  result of software requirements being difficult
  to estimate and track.
• Software estimating problems occur most often
  because of the
• Inability to accurately size a software project,
• Inability to accurately specify a software
  development and support environment,
• Improper assessment of staffing levels and
  skills, and
• Lack of well-defined requirements for the
  specific software activity being estimated

57
Software Cost Models (contd)

• The requirement to develop cost estimates for
  systems at a time when few details are known has
  led to the development of cost models.
• The need to generate estimates quickly and for
  many alternatives can make the analogy and
  engineering methods impractical due to lack of
  detailed requirements.
• In the early stages of software development, few
  details other than performance requirements and
  some physical constraints are available.
• Software lines of code (SLOC) are often used to
  estimate software size which is the most
  significant driver in determining software costs.
• Programming languages also have a significant
  effect on overall cost.

58
Software Cost Models (contd)

• Software cost models are based on statistically
  derived cost estimating relationships (CERs) and
  various estimating methodologies used to predict
  the cost of a system.
• Models approximate the real world through a
  combination of
• Statistical analysis of historical data,
• Informal/Intuitive analysis of rules of thumb
  based on experience,
• Standard procedures such as learning curves,
  inflation computation, and labor rate/overhead
  applications.
• Models may contain the following features
• Support costs calculations, time-phasing to
  spread development costs by fiscal year,
  inflation calculations to convert base year to
  then-year dollars.

59
Software Cost Models (contd)

• Regardless of how software is programmed, it must
  proceed through certain steps or phases and must
  be supported after development.
• The combination of software development and
  support is known as the software life cycle.
• The Cost Element Structure (CES) discussed
  earlier accounts for the software life cycle
  using the following sub-elements for CES 1.3
  Software Development
• Requirements Definition
  
• Analysis, Design, and Integration
• Quality Assurance Program
• Configuration Management
• Software Design and Development
• HW/SW Integration, Assembly, Test and Checkout
• System Operational Test and Evaluation
• System Independent Software Verification and
  Validation
• Site Acceptance Testing
• Independent Operational Test and Evaluation
• Program Management
• Installation and Checkout
• Corrective Maintenance

60
Software Cost Models (contd)Examples of
Commercially Available Models

• COCOMO
• COSTXPERT
• SLIM
• SEER
• Costar, REVIC, etc.

61
Software Cost Models (contd)COCOMO

• Constructive Cost Model (COCOMO) is one of the
  earliest cost models widely used by the cost
  estimating community.
• COCOMO was originally published in Software
  Engineering Economics by Dr. Barry Boehm in 1981.
• COCOMO is a regression-based model that considers
  various historical programs software size and
  multipliers.

62
Software Cost Models (contd)COCOMO

• COCOMOs most fundamental calculation is the use
  of the Effort Equation to estimate the number of
  Person-Months required to develop a project.
• Example of person months loaded labor rate
  Estimated Cost
• Most of the other estimates (requirements,
  maintenance, etc) are derived from this quantity.

63
Software Cost Models (contd)COCOMO

• COCOMO requires as input the projects estimated
  size in Source Lines of Code (SLOC).
• In addition to SLOC, COCOMO has scale factors
  which allow you to tailor conditions to your
  project.
• Scale factors are used to calibrate or adjust the
  model that is based on data which is not
  necessarily representative of the system being
  estimated
• Intent of calibrating is to allow for general
  application of a model developed from a specific
  database
• Based on the assumption that any differences
  between the predicted cost and the historical
  cost not explained by the model are due to
• Differences in the type of systems and
  acquisition environment represented in the two
  data sets

64
Software Cost Models (contd)COCOMO

• Scale factors include
• Development flexibility, architecture, risk, team
  cohesion, and process maturity among others.
• Additionally COCOMO uses Cost Drivers grouped
  in broad categories such as personnel, product,
  platform, project etc.
• Product Cost drivers include
• required reliability and product complexity.
• Personnel cost drivers include
• language, platform, and tool experience

65
Software Cost Models (contd)COCOMO

• COCOMO defaults to a nominal value for all
  factors when model is first used.
• Cost estimators must calibrate the model to the
  program being estimated to their specific
  development environment and productivity level of
  staff.
• COCOMO uses analogy and parametric methods based
  pm a historical, proprietary database of data
  submitted by consortium members.
• Allows COCOMO to be compatible with current
  design methods and evolving higher order
  languages.

66
Software Cost Models (contd)COCOMO

• The model makes its estimates of required effort
  (measured in Person-Months) based primarily on
  the projects estimate of the software size (as
  measured in thousands of SLOC, KSLOC))
• Effort 2.94 EAF (KSLOC)E
• Where EAF is the Effort Adjustment Factor derived
  from the Cost Drivers and E is an exponent
  derived from the five Scale Drivers.

67
Software Cost Models (contd)COCOMO

• For example assuming a project with all nominal
  cost and scale drivers, EAF would have a value of
  1.00 and exponent, E, of 1.0997.
• Assuming that the project consists of 15,000
  source lines of code, COCOMO estimates that 57.77
  Person-Months of effort is required to complete
  it.
• Effort 2.94 (1.0) (15)1.0997 57.77
  Person-Months.
• Person-months multiplied by a loaded labor rate
  (includes direct and overhead costs) would yield
  the cost estimate
• 57.77 person-months 15K per person-month
  866,000

68
Software Cost Models (contd)COCOMO Model output
example
69
Software Cost Models (contd)COCOMO
Example of how model defaults to nominal scale
factors. These should be adjusted for every
estimate to approximate the new, unique
development effort.
70
Software Cost Models (contd)COCOMO outputs by
phase
71
Software Cost Models (contd) COCOMO outputs by
phase
72
Software Cost Models (contd)COCOMO overall
estimate by phase
73
Software Cost Models (contd)COCOMO Summary

• COCOMO provides the following information
• Software development, maintenance cost and
  schedule estimates.
• Cost, schedule distribution by phase, activity,
  and increment.
• A free updated version of COCOMO is available at
• http//sunset.usc.edu/research/COCO
  MOII/

74
Software Cost Models (contd)

• COCOMO
• COSTXPERT
• SLIM
• SEER
• REVIC
• Costar, Price S, etc.

75
Software Cost Models (contd)CostXpert

• CostXpert builds on COCOMO by having a database
  of actual projects that is constantly being
  updated for various types of systems (commercial,
  military, etc.).
• When you enter project specific information,
  CostXpert maps it to the closest projects in its
  database and gives you results tailored to your
  project.
• CostXpert uses both parametric and analogy
  estimating methods.
• CostXpert can accommodate a variety of software
  sizing elements such as SLOC, function points,
  object metrics, GUI metrics, and feature points.

76
Software Cost Models (contd)CostXpert

• CostXpert also allows for customization
  specifically to your work environment.
• In addition, CostXpert creates a detailed cost
  element structure (CES), conducts what if
  scenarios, risk analysis, and provides a wide
  variety of reports to include
• Cost allocations by task list,
• Labor loading by phase and functional categories,
• Maintenance and life-cycle costs,
• Software defects estimates, and
• Documentation deliverables
• GAO uses CostXpert during various cost analysis
  assessments to check the reasonableness of both
  the software cost estimate and accompanying
  schedule for delivery.
• Evaluation copies can be downloaded at
  http//www.costxpert.com/products/downloads.html

77
Software Cost Models (contd)CostXpert example
of input screen
78
Software Cost Models (contd)

• COCOMO
• COSTXPERT
• SLIM
• SEER
• REVIC
• Costar, Price S, etc.

79
Software Cost Models (contd)SLIM

• The Software Life Cycle Model (SLIM) is marketed
  by Quantitative Software (QSM).
• Set of tools include SLIM-Estimator, SLIM-Data
  Manager, SLIM-Control, SLIM-Metrics and
  SLIM-Master Plan.
• Input metrics include SLOC, Function Points,
  Objects, Windows, Screens, or Diagrams
• Historical productivity and complexity are also
  inputs to the model
• Quick start wizard enable rapid generation of
  estimates and details which can then be tailored
  for the specific project.

80
Software Cost Models (contd)SLIM

• Results are presented at the 50 probability and
  can be adjusted by using sliders for various
  parameters.
• Estimates can readily be compared to historical
  data and model allows for analysis of what if
  scenarios.
• A Productivity Index (PI) is calculated for your
  project and updates as new data is available.

81
Software Cost Models (contd)SLIM

• As described in the users manual, the PI is a
  macro measure of the total development
  environment. It incorporates many factors in
  software development, including
• Management influence, development methods, tool,
  techniques, skill and experience.
• Values for PI range from .1 to 40.
• Low values generally are associated with poor
  environments, process, tools and complex systems.
• High values are associated with mature processes,
  and capable management and well-understood,
  straightforward projects.
• Outputs of the model include project
  description, estimation analysis views, schedule,
  risk analysis, staffing and skill breakout,
  effort and cost, reliability estimates, and
  documentation sections.
• Demo copies of SLIM can be downloaded at
  http//www.qsm.com/ .

82
Software Cost Models (contd)

• COCOMO
• COSTXPERT
• SLIM
• SEER
• REVIC
• Costar, Price S, etc.

83
Software Cost Models (contd)SEER

• This set of tools provides both software and
  hardware estimation capabilities.
• Software Project ControlSoftware Estimation,
  Planning and Project Control
• Hardware Project ControlHardware Estimation,
  Planning, Project Control and Life-Cycle Cost
  Analysis, IncludingOperations and Support
• Design for ManufacturabilityCost Designer for
  Parts, Process and Assembly

84
Software Cost Models (contd)SEER

• The toolset includes
• SEER-SEM for evaluating software development.
• SEER-H for estimating hardware and hardware
  Operations and Support costs.
• SEER-DFM for manufacturability analysis.
• SEER-IC for estimating foundry costs.
• SEER-SSM for software sizing.
• Outputs of the model include
• Labor estimates by function,
• A quick estimate providing a snapshot of size,
  effort, and schedule, and
• Staffing by month, cost by activity,
  person-months by activity, delivered defects, and
  SEI maturity rating.
• Demo copies of SER-SEM and other tools can be
  downloaded at http//www.galorath.com/tools_overvw
  .shtm

85
Software Cost Models (contd)

• COCOMO
• COSTXPERT
• SLIM
• SEER
• REVIC
• Costar, Price S, etc.

86
Software Cost Models (contd)REVIC

• The Revised Enhanced Version of Intermediate
  COCOMO (REVIC) model was developed by Mr. Raymond
  L. Kile formerly of Hughes Aerospace.
• The main difference between REVIC and COCOMO is
  the coefficients used in the effort equations.
  REVIC changed the coefficients based on a
  database of recently completed DOD projects.
• REVIC's schedule estimates are often considered
  lengthy because it assumes that a project's
  documentation and reviews comply with the full
  requirements of DOD-STD-2167A/498.

87
Software Cost Models (contd)

• COCOMO
• COSTXPERT
• SLIM
• SEER
• REVIC
• Costar, Price S, etc.

88
Software Cost Models (contd)Costar, Price S,
etc.

• There are many other models available.
• Most are based on COCOMO.
• Almost all need to be calibrated with project
  specific data.
• Using multiple models and comparing results
  provides better confidence.
• Crosschecking estimates leads to better cost
  estimate validation
• The International Council on Systems Engineering
  (INCOSE) has a list of software tools that may be
  of use.
• http//www.incose.org/tools/tooltax/costest_tools.
  html

89
Cross-checks and Validation

• After an estimate has been created, the next step
  involves validating the estimate by
  cross-checking.
• Cross-checking means using a different approach
  to create the estimate.
• If both estimates are close, the target estimate
  has some validity.
• If both estimates are very different.
• This increases the level of uncertainty which
  must be reflected in a risk analysis.
• This may lead to another estimating method to
  increase cost estimate confidence.
• It is a good practice to cross-check major cost
  drivers.
• If time is available, cross-checking other cost
  elements can further validate the entire
  estimate.
• Factors tend to be the most common approach for
  top level checks.
• Software cost models play an important role
  during later stages of development phase.
• Can be used to cross-check the reasonableness of
  costs forecasted using either the engineering or
  actual cost methods.

90
Cross-checks and Validation (contd)

• Validation is the process of demonstrating that
  either a CER or a cost model accurately predicts
  past history.
• Validation also includes a demonstration that
• The data relationships are logical,
• The data used are credible,
• Strong data relationships exist, and
• The CER or cost model is a good predictor of
  costs.
• This can be done using independent test data (not
  included in models calibration).
• The models cost output would then be compared to
  the test points known value to provide an
  accuracy benchmark.

91
Cross-checks and Validation (contd)

• Validation also includes a demonstration that
• Model users have sufficient experience and
  training,
• Calibration processes are thoroughly documented,
• Formal estimating policies and procedures are
  established, and
• When applicable, information system controls are
  maintained to ensure the integrity of the models
  being used.

92
Risk and Sensitivity AnalysisRisk Analysis
Definition

• Risk analysis is a process that uses qualitative
  and quantitative techniques for analyzing,
  quantifying and reducing uncertainty associated
  with cost goals.
• By nature, all cost estimates have some
  uncertainty.
• Earlier in development that uncertainty is
  higher.
• As the project matures these uncertainties
  decrease due to greater design definition, actual
  experience, and less opportunity for change.
• Errors can also occur from historical data
  inconsistencies.
• Cost risk analysis aims to achieve the following
  objectives
• Identify program level confidence for development
  schedule,
• Provide credibility to the target estimate, and
• Identify technical, schedule, and cost estimating
  risk drivers for use in risk management.

93
Risk and Sensitivity AnalysisRisk Analysis
Definition (contd)

• Risk is defined as a situation in which the
  outcome is subject to an uncontrollable random
  event stemming from a known probability
  distribution.
• Roll of two dice is an example since the roll can
  result in one of 11 possible outcomes.
• Uncertainty is defined as a situation in which
  the outcome is subject to an uncontrollable
  random event stemming from an unknown probability
  distribution.
• Example would be will it rain two weeks from
  today?
• Cost estimating falls more into the range of
  uncertainty than risk, but most managers use the
  term risk analysis.

94
Risk and Sensitivity AnalysisSensitivity
Analysis Definition (contd)

• Sensitivity analysis answers the question
• What happens if the assumptions change?
• Changes to some assumptions can have profound
  impact, while huge changes to other assumptions
  have little effect on results.
• Sensitivity examines the effect of primary cost
  estimating outputs to changes on individual CES
  inputs.
• Sensitivity analysis highlights the factors that
  have the strongest impact on the overall cost
  estimate.
• Points to management which factors deserve the
  most attention.
• Narrows down the number of lower level cost
  elements that should be examined using risk
  analysis techniques.

95
Risk and Sensitivity AnalysisSources of Cost
Risks

• Schedule and Technical risks
• Unexpected design changes
• Project team experience
• Number of business units impacted
• Requirements changes
• Integration considerations
• Technical difficulties or maturity issues
• Revised project or acquisition plans
• Quantity changes
• New labor rates
• Higher inflation
• Cost estimating risks
• Imprecision associated with the estimating
  techniques used, errors, or oversights

96
Risk and Sensitivity AnalysisRisk Analysis
Techniques

• Risk factor approach
• Adds cost to the overall point estimate (most
  likely estimate) to cover risks by way of a
  factor.
• This factor is usually a percentage derived from
  past data and experience.
• Often applied to the estimate as a whole versus
  lower level cost elements.
• Monte Carlo Simulation
• Automatically analyzes the effect of varying
  inputs on outputs of a cost model using
  spreadsheet risk analysis.
• Randomly generates values for uncertain variables
  over and over to simulate a model.

97
Risk and Sensitivity AnalysisRisk Analysis
Techniques (contd)

• Monte Carlo Simulation Steps
• Analyst obtains cost distribution for each
  element identified as a major cost driver either
  through experience or sensitivity analysis.
• Cost distribution is usually triangular in the
  form of optimistic, most likely, and pessimistic
  cost estimates.
• These cost ranges are usually obtained through
  expert judgment (engineer or technical specialist
  interviews) or using a Delphi technique
• After all cost elements have been identified by a
  distribution, the simulation is run many times
  (1,000 10,000 times).
• Simulation calculates multiple scenarios of the
  cost model by repeatedly sampling values from the
  probability distributions assigned to the various
  cost elements.
• While the simulation runs, the forecasts
  stabilize towards a smooth frequency distribution
  called a cumulative frequency distribution or CDF
• After thousands of trials, statistics of the
  results and the certainty of any outcome can be
  obtained from the CDF.

98
Risk and Sensitivity AnalysisRisk Analysis
Techniques (contd)

• Monte Carlo Simulation Results
• Reveal not only the result values for each
  forecast, but also the probability of any value
  occurring.
• This is helpful to management because it can show
  the level of certainty of achieving a cost
  objective.
• For example, a simulation can show there is a 10
  chance of the project finishing for 50 million,
  a 50 chance of it costing 70 million, and a
  90 chance of developing the project for 100
  million or less.
• Decision makers can use these results to decide
  which projects will continue funding based on
  quantifiable risk parameters.

99
Risk and Sensitivity AnalysisTypes of Risk
Simulation Tools

• US GAO uses Crystal Ball to perform Monte Carlo
  Simulations
• Crystal Ball is an simulation program that helps
  you analyze the risks and uncertainties
  associated with Microsoft Excel spreadsheet based
  models.
• http//www.decisioneering.com/crystal_ball/
• Other tools are available, such as
• _at_Risk (http//www.palisade.com/html/risk.html)
• RiskEase (http//www.riskease.com/)
• Pertmaster (http//www.pertmaster.com/)

100
Documentation

• After cross-checking the estimate major cost
  drivers, the next step, and most important one of
  all, involves documenting the entire estimate
  process.
• Although this is a difficult and time-consuming
  procedure, the level of detail and attention will
  pay big dividends when it comes time to
  re-estimate
• May also help data collection of actual analogous
  costs.
• Important to document the estimate as it is being
  developed (i.e., document as you go).
• Hard to remember rationale and judgments for
  adjusting data months later when it needs