Title: Software Cost Estimation at GSFC
1Software Cost Estimation at GSFC
2GSFC Cost EstimationAgenda
- Overview
- Examples of project cost data
- Mission Critical Software cost growth study
- Flight Software Approach
- Ground Science Software Approach
- Wide band Delphi method a team approach to cost
estimation - Cost estimates and the software management plan
(product plan).
3Cost EstimationExamples from missions
- Provided by experienced software managers
- Some examples
- Flight MAP was 75 staff years, EO-1 20
- Ground
- Total including science operations Great
Observatory 60-80 staff years, Small Explorer
25-35 - Mission data system range of 10 (SWIFT)-20 (MAP)
staff years through launch - Where does the money go?
- (MAP flight SW example)
4Study Of Why SW Cost Grows
- Results from Mission Critical Software Look
Back study - Based on similar study at JPL
- Estimates are made by manager of 7 ground
software and 7 flight software projects What is
the size of cost growth? What causes the cost
growth? - Size of cost growth
5Causes Of Cost Growth From Study
- Contribution to Overall Cost Growth
- (add to 100, w/ round off)
6Causes Of Flight SW Cost Growth
- Planning
- Specific Project pressures to reduce cost
estimates due to cost constraints, in some
instances never allowing a true estimate to be
baselined - Post-PDR mandatory cost reductions across the
board - Unrealistic dependencies within delivery schedule
(e.g., requirements not specified as scheduled,
hardware not available as scheduled) - Requirements
- New requirements often levied in late mission
design or IT phase, as systems personnel become
more knowledgeable - Incomplete requirements
- Development and test ramifications are not fully
understood - Reduction of Perceived Overhead
- Minimized production or review of documentation
- Minimized formal and informal review activities
7Causes Of Ground SW Cost Growth
- Planning
- Project pressure to reduce overall cost estimates
- COTS capabilities not functioning as
planned/advertised - Requirements
- Maturity of requirements limits accuracy of
preliminary cost estimates - Disparity in completeness of functional
requirements (e.g., telemetry and command
requirements are often well understood, while
science processing requirements are left
incomplete until late in mission lifecycle) - Testing
- Insufficient emphasis on test plan completeness
- Application of additional testing as primary risk
mitigation - Staffing
- Launch delays result in funding and maintaining
marching army - Lack of adequate dedicated staffing early in
project lifecycle
8Implications
- It is tempting to cut back estimated costs, but
- This implies implementing fewer features, or
- This implies jettisoning proven engineering
practices - THIS WILL COST MORE DURING TESTING!
- Requirements always change, plan a reserve!
- Provide best case, most likely case, and worst
case - Measure development costs for
- Tracking against plan
- Refining estimation models for future projects
9Cost EstimationHow Do We Estimate Costs?
- General approach
- Comparison with previous missions
- Adjust based on technical management risk
factors - Example Flight Software Branch
feature-by-feature estimation - Break into subsystems
- Estimates are made for each subsystem
- Experienced developers rate complexity of each
feature using previous missions as guide - Rationale for each rating should be documented
- Points assigned to complexity correlate with
effort - Subsystem efforts are totaled
- Adjust total for project risk
10Flight SW Cost EstimationACS subsystem
development
- Estimator compares each required feature with
prior missions creates a new rating (last
column) - Scores are added to get overall ACS complexity
- When 89 features are rated for ACS, estimation
errors tend to offset each other - Linear regression or other technique is used to
relate estimated cost to complexity estimates,
e.g. - Effort (staff years) 0.2points 6
11Flight SW Cost EstimationFlight Software Base
Cost
...
ACS Estimate
PSE Estimate
C DH Estimate
Instrument Estimate
Project Management Estimate
Dev. Test Environment Estimate
Subsystem Development Estimate
System Testing Estimate
Project HW Subsystem Sup. Estimate
Sustaining Engineering Estimate
FSW BASE COST ESTIMATE
12Flight SW Cost EstimationAdjusting for Risk
- For a set of risk factors, rate each risk as
- Extremely high
- High
- Nominal
- Low
- Extremely low
- Based on these ratings, adjust cost estimate
- Models such as COCOMO have tables of multipliers
for scale or exponent, corresponding to these
ratings - Flight Software Branch uses expert judgment
13Flight SW Cost EstimationRisk factors to use
- Flight Data System Hardware selection
- Distributed system or not
- Mission Hardware selections and redundancies
- Ground System Selections for FSW Test, IT and
Operations - FSW Testbeds
- Operations Concepts
- Ground Contact Opportunities
- Failure Detection and Response Philosophy
- Deployment and in-orbit insertion plan
- Experience of team lead
- Quality of Tools available to the FSW Team
- Schedule compression
14Ground Systems Costing Practices
- Lead Engineer costs all ground systems
components, including component functional domain
experts as necessary - Most Recent Missions Use Heritage Systems or COTS
Solutions, e.g., - ITOS on Triana
- Epoch 2000 on NGST
15Science Systems Cost Estimation
- Systems Costed by Developing or Operational
Organizations - Most Recent Mission Science Data Systems are
Evolutions of Existing Data Processing Systems,
e.g., - GPMs PPS is an evolution of TRMMs TSDIS
- In Some Cases, a Level of Effort is Provided to
Science Labs for System Development - Process is Mostly Informal
16HW/SW Maintenance CostCOTS Model of EOSDIS
Growth
- EOSDIS is a long life system (to support the
goal of a 15 year consistent climate data set) - Spreadsheet model for phased computational and
I/O throughput and archive volume capacity based
on - Mission launch schedules (Landsat, Terra, Aqua,
Aura, et.al.) - Reprocessing and product deletion phasing
assumptions - Assumptions about end user demand for product
distribution - Translated into projected COTS (and COTS
maintenance) costs and support capacity
requirements - Based on current design and calibrated to recent
COTS purchases - Incorporating estimated future price/performance
curves for several equipment categories (e.g.
CPU, disk, archive media) - Technology refresh trade-offs also included (vs.
maintenance)
17Wide-band Delphi Cost EstimationOverview
- Iterative method multiple experts create
estimates - Many variations of Delphi methods exist.
- Estimators meet to analyze discrepancies between
estimates - Meeting notes capture varying assumptions
- Agree on assumptions for next round
- Estimators create new estimates based on new
assumptions - Process is repeated until estimates converge
- Moderator Preparation
- Provide materials
- Provide initial assumptions
- Select team
- Conduct initial meeting
Round 1 by individuals Produce independent
estimates
- Moderator produce estimatesummary
- Individual median estimates
- New assumptions
Round 2 Reconsider based on all infor-mation in
estimate summary
Consensus?
Meet to reachconsensus
Done!
18Wide-band Delphi Cost EstimationTypical
assumptions are about
- Development team experience
- Requirements quality
- Completeness
- Clarity
- Stability
- Available tools
- Reuse approach
- Imposed constraints
- Cost and/or schedule
- Tools, programming language,
19Wide-band Delphi Cost Estimation Estimation Team
- Team must be
- Experienced
- In type of mission software
- With GSFC software development processes
- Representative experts in
- Development
- Testing
- Software project management
- Maintenance
20Wide-band Delphi Cost EstimationEstimation by
individuals
- Initial phase
- No collaboration between estimators
- Make best case, most likely case, and worst case
estimate (can apply methods taught in class) - Document issues, questions assumptions
- Revision phase
- Moderator provides Estimation Summary
- (anonymous) individual and median estimates
- Summary of issues, questions assumptions
- Individuals revise based on Estimation Summary
21Software Management PlanCost Estimation in
context
Example from Flight SW Product Plan Template
22GSFC Cost Estimation Summary
- Where are we?
- Information Systems Cost Estimates Rely on the
Knowledge and Experience of a Few Experts - Estimates are as good as the expert
- It is sometimes Difficult to Ascertain Actual
Past Missions Costs - Where are we going?
- Efforts are Underway to Document/Formalize Cost
Estimation Practices - Some of what you are seeing in this course is
likely to show up in the documented processes
later