Joint ADL CoLab BAA COSCOMO Prototype Project

1
Joint ADL Co-Lab BAACOSCOMO Prototype Project
Contract N61339-05-C-0115

• Executive Overview

2
Points of Contact
Mike Garnsey Mike.Garnsey_at_sparta.com Lacey
Edwards Lacey.Edwards_at_sparta.com
Dean Marvin Dean.Marvin2_at_us.army.mil Susan
Marshall Susan.Marshall1_at_us.army.mil
Kelly Ward Kelly.Ward_at_gdit.com
Roger Smith Roger.Smith14_at_us.army.mil
3
COSCOMO Prototype Project Final Review Agenda

• Project Overview
• Management Review Material
• Technical Project Essentials
• COCOMO Background
• COSCOMO Model Derivation
• How the COSCOMO Model Works
• The COSCOMO Prototype Tool
• live Demo
• Model Calibration and Validation
• Initial Model Calibration
• Historical Project Data Collection
• Model Validation Tests
• Wrap-up / Discussion

4
Project Overview
5
COSCOMO Prototype Project Concept
It is challenging for both sponsors and
developers to estimate the expected level of
effort, duration, and cost of developing
web-based SCORM conformant courseware.

• Project Goal create an interactive project
  estimation tool COSCOMO for ISD/SCORM content
• Domain focus ADL Sharable Content Object
  Reference Model (SCORM) conformant content
• ISD methodology Analysis, Design, Development
  Implementation, Evaluation (ADDIE) model
• Algorithmic foundation COCOMO II model for
  software project estimation

6
COSCOMO Prototype Project Description

• Create a COSCOMO algorithm to estimate the
  person-months required to create a SCORM
  conformant learning product
• Collect historical project data expert
  knowledge
• Calibrate and Validate COSCOMO Model
• Calibrate COSCOMO model to fit historical data
  and expert knowledge
• Validate the algorithm for various sets of data
  within the bounded conditions
• Develop a prototype tool for the COSCOMO cost
  model
• Build a community of interest around the method
  tool
• Training 2006 conference
• COSYSMO conference
• INCOSE 2006 symposium
• Implementation Fest 2006
• I/ITSEC 2006 conference

7
Applicability Value to Community

• Consistent, objective, and reliable estimation
  tool for SCORM content and projects
• First step in formalizing an estimation method in
  the ADL community
• Create a tool that other projects can apply,
  modify, and mature
• COCOMO II has been evolving for 25 years.
• COSCOMO prototype from this project will be the
  first step in the long evolution and improvement
  of a tool for this community

8
COCOMO Background
9
COCOMO II Tool Example
Users Perspective
10
COCOMO II Drivers(29 Input Variables)

• Size
• Source Lines of Code (SLOC)
• Design Modification (DM)
• Code Modification (CM)
• Integration (IM)
• Assessment (AA)
• Understanding (SU)
• Unfamiliarity (UNFAM)
• Requirements Evolution (REVL)
• Product Effort Multipliers (EM)
• Required Reliability (RELY)
• Database Size (DATA)
• Product Complexity (CPLX)
• Required Reuse (RUSE)
• Documentation (DOCU)
• Platform EM
• Execution Time Constraints (TIME)
• Main Storage Constraints (STORE)
• Platform Volatility (PVOL)

• Personnel EM
• Analyst Capability (ACAP)
• Programmer Capability (PCAP)
• Personnel Continuity (PCON)
• Applications Experience (APEX)
• Platform Experience (PLEX)
• Language/Toolset Experience (LTEX)
• Project EM
• Use of Software Tools (TOOL)
• Multisite Development (SITE)
• Required Development Schedule (SCED)
• Scale Drivers
• Development Flexibility (FLEX)
• Process Maturity (PMAT)
• Precedentedness (PREC)
• Arch/Risk Resolution (RESL)
• Team Cohesion (TEAM)

11
COCOMO II Model Form
17
PM A(Size)E P EMi
i1
5
E B .01S SFj
j1
Where PM effort in Person Months A
calibration constant derived from historical
project data Size Adjust Source Lines of Code
E represents diseconomy of scale (composed
of 5 scale factors) EM effort multiplier for
the ith cost driver. The geometric product
results in an overall effort adjustment factor to
the nominal effort. B calibration constant
derived from historical project data SF scale
factor for the jth cost driver. Provides
organization specific adjustments to the size of
the project.
3
EM1 x EM2 x EM3
P EMi
i1
12
Reliability PRED(30)

• Reliability of COCOMO family of models is often
  measured by the percentage of test cases that it
  will estimate within 30 of the actual project
  costs
• e.g. If a project requires 300 person-months to
  complete, then its PRED(30) range would be (210
  to 390)
• If the model estimates 70 of its test cases
  within this range then the models PRED(30) 70
• COCOMO Family Model Levels
• COCOMO II (2000) PRED(30) 69
• COSYSMO PRED(30) 56

13
COSCOMO Model Derivation
14
COCOMO II Research Approach

• Studied the definitive source on COCOMO II
  Software Cost Estimation with COCOMO II by Barry
  Boehm et.al.
• Model form and variables
• Methodology used to create the COCOMO II model
• Researched how related models (COSYSMO, COPSEMO,
  etc.) modified COCOMO II for their purposes
• COSYSMO workshop
• Relationship with Dr. Ricardo Valerdi
• Surveyed software implementations of COCOMO II

15
COCOMO Model Family
Other Independent Estimation Models
Software Cost Models
DBA COCOMO 2004
?
?
COCOMO 81 1981
COCOTS 2000
COSYSMO 2002
COCOMO II 2000
?
?
COINCOMO 2004
COSoSIMO 2004
Costing Secure System 2004
?
?
iDAVE 2003
COPLIMO 2003
COPSEMO 1998
Security Extension 2004
COQUALMO 1998
?
COPROMO 1998
CORADMO 1999
COSCOMO 2006
?
Software Extensions
Legend Model has been calibrated with historical
project data and expert (Delphi) data Model is
derived from COCOMO II Model has been calibrated
with expert (Delphi) data
Dates indicate the time that the first paper was
published for the model
16
COCOMO II Modeling Methodology
17
COSCOMO Mods to the COCOMO II Input Variable Set

• Size
• Source Lines of Code (SLOC)
• Design Modification (DM)
• Code Modification (CM)
• Integration (IM)
• Assessment (AA)
• Understanding (SU)
• Unfamiliarity (UNFAM)
• Requirements Evolution (REVL)
• Product Effort Multipliers (EM)
• Required Reliability (RELY)
• Database Size (DATA)
• Product Complexity (CPLX)
• Required Reuse (RUSE)
• Documentation (DOCU)
• Platform EM
• Execution Time Constraints (TIME)
• Main Storage Constraints (STORE)
• Platform Volatility (PVOL)

• Personnel EM
• Analyst Capability (ACAP)
• Programmer Capability (PCAP)
• Personnel Continuity (PCON)
• Applications Experience (APEX)
• Platform Experience (PLEX)
• Language/Toolset Experience (LTEX)
• Project EM
• Use of Software Tools (TOOL)
• Multisite Development (SITE)
• Required Development Schedule (SCED)
• Scale Drivers
• Development Flexibility (FLEX)
• Process Maturity (PMAT)
• Precedentedness (PREC)
• Arch/Risk Resolution (RESL)
• Team Cohesion (TEAM)

No
No
No
No
No
No
No
No
No
No
18
More COSCOMO Mods to the COCOMO II Input Variable
Set

• Size of Product
• Hours of Courseware
• Level of Instruction
• New, Reused, Modified Courseware
• Requirements Evolution (Content Discarded)
• Product Effort Multipliers (EM)
• Required Reliability (RELY)
• Product Complexity (CPLX)
• Required Reuse (RUSE)
• Documentation (DOCU)
• Platform EM
• Bandwidth (BAND)
• Platform Volatility (PVOL)

• Personnel EM
• Senior Capability (SCAP)
• Developer Capability (DCAP)
• Personnel Continuity (PCON)
• Applications Experience (APEX)
• Platform Experience (PLEX)
• Development Tools Experience (DTEX)
• Project EM
• Lifecycle Tools (LIFE)
• Multisite Development (SITE)
• Required Development Schedule (SCED)
• Scale Factors
• Precedentedness (PREC)
• Development Flexibility (FLEX)
• Arch/Risk Resolution (RESL)
• Team Cohesion (TEAM)
• Process Maturity (PMAT)

Change
Change
Change
New
Change
Change
Change
New
19
How the COSCOMO Model Works
20
Project Cost Estimation
Dollar Costs
Project Requirements
Quantitative Input
Qualitative Adjustments
Reuse
Person Months to Develop
Level of Instruction
Course Hours
Time to Develop
21
COSCOMO Algorithm
COCOMO II equation form
COSCOMO algorithm
Note in COSCOMO, Size adjusted of courseware
hours (vs SLOC)
22
Effort Multipliers Effect on Project Cost
Estimates
15 Effort Multipliers for Nominal Project
Course Hours
Project Cost
Project Cost A EMi
Course Hours (Person Months) (Constant)
(Modifiers) (Student Hours)
Multiplier Form
23
Effort Multipliers Effect on Project Cost
Estimates (cont)
One use of the COSCOMO tool output graph is to
aid in identifying which variables are driving
the estimate. As shown is this notional graph,
high senior team capability (SCAP) and high
development team capability (DCAP) are
contributing significantly to reducing the
project cost. (increasing the experience and
capability of SCAP and DCAP by one level each
decrease the estimate by 37)
24
Scale Factors Effect on Project Cost Estimates
on Project Cost
when the input values for the five scale factors
are at their default Nominal levels, the scale
factors have no impact on the estimated cost value
higher than nominal scale factor value levels
reduce the estimated cost lower than nominal
scale factor value levels increase the estimated
cost. This graph depicts a notional real world
project, with Very High development
flexibility Low Process Maturity Nominal
precedentedness Very Low architecture/ risk
resolution Very Low team cohesion
25
Scale Factors Maximum /-Effects on Estimated
Cost
All SFs Very Low PREC 1 FLEX 1 RESL
1 TEAM 1 PMAT 1
Impact 14.8 Higher Cost Estimate
All SFs Extra High PREC 6 FLEX 6 RESL
6 TEAM 6 PMAT 6
Impact 7.1 Lower Cost Estimate
26
COSCOMO Prototype
27
Prototype Tool - Overview

• The COSCOMO tool GUI prototype was developed in
  Microsoft Excel. Why?
• Spreadsheets are great for Cost Model
  implementations
• Easy to deploy (just about everyone has Microsoft
  Excel!)
• This is an RD project (not a shrink-wrap s/w
  devel contract)
• Specifically, the prototype tool was developed as
  a custom Microsoft Excel Add-in
• Look and feel more like a custom application vs
  spreadsheet
• Easy to constrain and simplify user input via
  user forms
• Decent GUI rapid prototyping environment widget
  set
• Protection from all sorts of undesirable user
  actions
• resizing / hiding / deleting columns
• deleting cells that contain important
  formulas/parameters
• Etc
• Enables password protection of the source code

28
COSCOMO Prototype Demo
29
COSCOMO Tool Prototype - Screenshot 1
30
COSCOMO Tool Prototype - Screenshot 2
31
COSCOMO Tool Prototype - Screenshot 3
32
Initial Model Calibration
33
Delphi Calibration Survey

• Using Delphi Method to gather independent
  opinions about the level of effort associated
  with each setting on each variable.
• Collect individual responses
• Calculate group average
• Present averages to each person and allow them to
  change their responses if they would like
• Convene group work session to settle on
  acceptable values
• Round 1 of Delphi Conducted Via Private Survey
• Round 2 3 Conducted in Seminar Form

34
Delphi Variable Survey Excerpt
The center value is "Nominal" or "Standard."
Estimate the relative amount of effort required
for each of the other levels. For example, if
you believe that a project ranked as "Very High"
would take 2 times the effort of a nominal
project, then enter 2.0 in the column for "Very
High."
Very Low
Low
Nominal
High
Very High
Extremely High
actual ratings from the 5 survey experts after
Delphi round 2
35
COSCOMO Whats Next?
36

• This project is the first step in formalizing a
  cost estimation method in the ADL community
• COSCOMO tool prototype is the first step in the
  long evolution and improvement of a tool for the
  ADL community
• keep in mind COCOMO II has been evolving for 25
  years
• Historical project data collection is essential,
  but it is also very difficult to get access and
  cooperation from the people with this information
  
• Have currently collected data on 9 projects
• 40 projects needed to calibrate the model
  appropriately
• GUI prototype of the COSCOMO tool is ready for
  early adoption by the ADL community
• Not a polished, shrink-wrapped product, but
  more refined and user friendly than a raw
  spreadsheet