Advanced Training Module Cost Estimation

1
Advanced Training ModuleCost Estimation

• Status
• 10 September 2003

2
Advanced Training ModuleBackground

• Task assigned to SAI USA at 11th meeting of the
  INTOSAI Standing Committee on IT Audit in New
  Delhi, India.
• SAI USA had previously submitted a paper for the
  INTO IT Journal on cost estimating
• SAI USA has been using cost estimating
  methodologies as a part of value for money and
  technical audits

3
Advanced Training ModuleStatus

• SAI USA has completed development of the Cost
  Estimation and Analysis training module
• Will provide a brief overview at this time
• Will be using it internally at the GAO to train
  personnel
• Will also be developing a cost estimation and
  analysis guide in the next year
• Would like to thank the committee for giving us
  the opportunity to work on this effort

4
Advanced Training ModuleOverview

• Karen Richey
• Senior Cost Analyst

5
Advanced Training ModuleCost Estimation
Analysis

• Outline
• Introduction to Cost Estimating
• Life Cycle Costs
• Data Collection
• Data Analysis
• Cost Estimating Methodologies
• Expert Opinion, Analogy, Parametric, Engineering,
  and Actual
• Software Cost Modeling
• Cross-checks and Validation
• Risk and Sensitivity Analysis
• Documentation Requirements
• Cost Estimating Challenges
• Cost Estimating Auditor Checklists

6
Advanced Training Module

• Introduction to Cost Estimating
• Basic introduction to cost estimating
• Links to places where additional material can be
  found
• Limitations on what cost estimating can and
  cannot do in an audit
• Types of estimation methodologies

7
Advanced Training Module

• Life Cycle Costs
• Definition and discussion of the scope of an
  estimate
• Discussion on how a life cycle cost can be used
• Introduction of a Cost Element Structure (details
  later) used to collect various costs for a
  project
• Example CES

8
Advanced Training Module

• Example CES
• 1.0 Investment Phase 2.0 System Operations
  Support Phase
• 1.1 Program Management 2.1 System Management
• 1.2 Concept Exploration 2.2 Annual Operations
  (supplies/spares)
• 1.3 System Development 2.3 Hardware
  Maintenance
• 1.3.1 System Design Specification 2.4
  Software Maintenance
• 1.3.2 Development Prototype and Test Site
  Investment 2.5 Outsource Provider Support
• 1.3.3 Software Development 2.6 Data
  Maintenance
• 1.3.4Training 2.7 Site Operations
  (personnel, training,etc.)
• 1.3.6 Facilities
• 1.4 System Procurement
• 1.4.1 Deployment Hardware
• 1.4.2 Deployment Software
• 1.4.3 Initial Documentation
• 1.4.4Logistics Support Equipment
• 1.4.5 Initial Spares
• 1.4.6 Warranties
• 1.5 Outsource Provider Investment
• 1.6 System Initiation, Implementation Fielding

9
Advanced Training Module

• Data Required
• Discussion of data required to generate a cost
  estimates
• Direct (engineering, materials) vs. indirect
  (support, rent, insurance) costs
• Data sources primary and secondary (reports,
  contractor site visits)
• Data validity and integrity
• Normalizing data for inflation

10
Advanced Training Module

• Cost Estimating Methodologies
• Expert Opinion,
• Analogy,
• Parametric,
• Engineering, and
• Actual

11
Advanced Training Module

• Considerations
• Choice of methodology is dependent upon
• Type of system
• Software, hardware, etc
• Phase of program
• Development, Production, Support
• Available data
• Historical data points from earlier system
  versions or similar system
• Technical parameters of system

12
Advanced Training Module

• Detailed Description for Each Methodology
• Overview
• Strengths and limitations (Pros Cons)
• Steps required for each methodology
• Simple case study

13
Advanced Training ModuleMethod Example -
Parametric

• Utilizes statistical techniques called Cost
  Estimating Relationships (CER).
• Relates a dependent variable (cost) to one or
  more independent variables
• Based on specific factors that have a high
  correlation to total cost
• Number of software lines of code (SLOC) or
  function points,
• Square feet for office floor space,
• Number of floors in a high rise building for
  cabling estimates,
• Database size, etc.
• Can be used prior to development.

14
Advanced Training ModuleMethod Example -
Parametric

• Pros
• Can be excellent predictors when implemented
  correctly
• Once created, CERs are fast and simple to use
• Easily changed
• Useful early on in a program
• Objective
• Cons
• Often lack of data on software intensive
  systems for statistically
• significant CER
• Does not provide access to subtle changes
• Top level lower level may be not visible
• Need to be properly validated and relevant to
  system

15
Advanced Training ModuleMethod Example -
Parametric

• Historical data has enabled the organization to
  have a previously developed CER to estimate a new
  IT system based on SLOC.
• Cost SLOC 25 /SLOC
• The CER is based on systems ranging from
  1,000,000 to 3,000,000 SLOC.
• You have estimated 2,600,000 SLOC for new system
• Cost 2,600,000 25 65M

16
Advanced Training ModuleMethod Example -
Parametric

• Cost estimators can develop their own CERs or
  they can use existing commercial cost models.
• Various Software cost estimating models will be
  discussed next
• Learning curves specialized type of CER
• CERs can be cost to cost or cost to non-cost
• Cost to Cost e.g., Manufacturing costs are 1.5
  times Quality Assurance costs
• Cost to Non-Cost /pound, or engineering
  hours/ of engineering drawings yields
  hours/drawing metric that can be applied to new
  program
• Factors and ratios are also examples of
  parametric estimating.

17
Advanced Training ModuleMethod Example -
Parametric

• Reliable, normalized data is most important for
  CER development.
• Must determine range of data for which the CER is
  valid.
• Useful at any stage in a program.
• Typically CERs are the main cost estimating
  methodology in early stages of a program.
• In later stages of a program, CERs serve as a
  cross check to other methods
• Must be logically sound as well as statistically
  sound.
• High correlation (r2 0.75 or higher) for
  goodness of fit test

18
Advanced Training ModuleMethod Example -
Parametric

• Different statistical techniques may be used to
  judge the quality of the CER.
• Least squares best fit (regression analysis, or
  the ability to predict one variable on the basis
  of the knowledge of another variable)
• Multiple regression (a change in the dependent
  variable can be explained by more than one
  independent variable)
• Curvilinear regression (relationship between
  dependent and independent variable is not liner,
  but based on a curve)
• Learning curve (describe how costs decrease as
  the quantity of an item increases)

19
Advanced Training ModuleMethod Example
Parametric (Learning Curve)

• The first 100 tasks of an installation took 10
  hours per task and the next 100 averaged 8 hours
  per task. Thus, the learning curve would be
  calculated as follows
• Learning curve 8 hours per task/10 hours per
  task 0.8
• Implies an 80 learning curve, meaning an
  improvement of 20 occurred between the first 100
  tasks and next 200 tasks

20
Advanced Training ModuleMethod Example
Parametric (Software Cost Models)

• Software costs as a percentage of total system
  costs continues to increase while associated
  hardware costs decrease.
• Accurately capturing software costs can be
  difficult and cost overruns often occur as a
  result of software requirements being difficult
  to estimate and track.

21
Advanced Training ModuleMethod Example
Parametric (Software Cost Models)

• Software estimating problems occur most often
  because of the
• Inability to accurately size a software project,
• Inability to accurately specify a software
  development and support environment,
• Improper assessment of staffing levels and
  skills, and
• Lack of well-defined requirements for the
  specific software activity being estimated

22
Advanced Training ModuleMethod Example -
Parametric (Software Cost Models)

• There are numerous software cost models
• COCOMO
• COSTXPERT
• SLIM
• SEER
• Costar, REVIC, etc.

23
Advanced Training ModuleMethod Example -
Parametric (Software Cost Models)

• Software cost models typically
• Require software lines of code (SLOC) to estimate
  cost
• Can be used to estimate size given typical
  applications (business, aerospace, real-time
  control, etc)
• Have various parameters which modify cost,
• For example, skill, tool set, programming
  language, operating system, processes used for
  development, etc.
• Approximate the real world through a combination
  of
• Statistical analysis of historical data,
• Informal/Intuitive analysis of rules of thumb
  based on experience
• Contain the following features
• Development and Support cost calculations,
  time-phasing to spread estimates by year, and
  inflation to convert to budget year dollars

24
Advanced Training Module

• Advanced Training concludes with brief discussion
  on
• Risk and Sensitivity Analysis
• Documentation Requirements
• Cost Estimating Challenges
• Cost Estimating Auditor Checklists

25
Advanced Training Module

• Thank You