newt nostromo entry writing tool a delphi polling tool

1
Schopenhauer's Proof For Software Pessimistic
Bias In the NOSTROMO Tool (U)

• Dan Strickland
• Dynetics Program Software Support
• daniel.strickland_at_dynetics.com

Ashley Mathis THAAD Project Office System
Software Engineering ashley.mathis_at_mda.mil

2
Overview (U)

• Background
• Pessimism in Code Size
• NOSTROMO Tool
• NOSTROMO Demo

3
(No Transcript)
4
Background (U)

• In 2003, TPO and Dynetics presented a methodology
  on using Monte Carlo simulation and uncertainty
  in COCOMO II to estimate risk in estimates
• In 2004, TPO and Dynetics presented updated
  methodologies to include Auto-Generated Code and
  other updates to NOSTROMO
• A new methodology for estimating pessimistic size
  growth has been incorporated into NOSTROMO

5
Boehm Horn Curve (U)

• Size (and effort) should converge over time
  relative to development phase
• During Concept of Operation, estimates can be
  50 to 200 of the actual final size
• Some types of software are always on the
  bottom part of the curve always growing in
  size

Ref (1) - SOFTWARE COST ESTIMATION WITH COCOMO
II (P 10) - BOEHM 2000
6
Code Size Growth (U)

• Code size estimates (and subsequent cost/schedule
  estimates) are required long before software
  requirements are baselined and finalized
• Code estimates grow to any number of factors
• Requirements Volatility
• Auto-Generated Code
• Poor Estimating Techniques
• Many large, complex programs exhibit constant
  code size growth outside the boundaries of the
  horn curve
• We define pessimism as extending the boundaries
  of the expected code size growth to historical
  limits with only growth potential things can
  only get worse

7
Three Historical Projects Size Growth (U)

• Three projects that have completed development or
  are very near completion
• Graphs represent code size estimates over time
  for new code and subsequent Effective SLOC
• Graphs show growth realization in new and ESLOC
  growth over time

8
Three Historical Projects Percentage Size
Growth (U)

• Projects new code size grew 90-180 from original
  estimates
• Projects ESLOC size grew 40-120 from original
  estimates
• Programs exhibited extensive growth after
  Critical Design Review (CDR) during Coding and
  Unit Testing
• Percentage growth is outside of the boundaries
  expected by the horn curve

9
Pessimism Curve (U)

• Plot the data points of the three projects in the
  same manner as the relative size of the horn
  curve
• Assume that the ceiling for Pessimism is the
  final size x and the floor encompasses the most
  extreme data points
• Where data is unavailable, use the horn curve
  values and some extrapolation if necessary

Value taken from horn curve no like value in
data Value extrapolated average of previous
and subsequent phases
10
Smoothed Pessimism Curve (U)

• Mapping the points of the pessimism curve against
  the horn curve show an exponential curve of
  greater growth potential
• Using MS_Excel, we added an exponential trendline
  to smooth the pessimism curve and identify a
  formula for use (R-squared value 0.9666)
• Using the smoothed curve, we can calculate the
  ranges for our pessimism curve as tied to
  development phase
• We used this new range (inverted) as boundary
  conditions for size in NOSTROMO
• NOSTROMO uses a triangle distribution between
  high and low boundaries for size

11
Original NOSTROMO Concept (U)
Notional Obscurity STatistical Risk Observation
MOdel
DATA PAGE
NEWT
NOSTROMO
REPORTS

• NOSTROMO first takes normal COCOMO II inputs
• NOSTROMO accounts for uncertainty in the
  settings
• NOSTROMO uses Monte Carlo to simulate hundreds
  of COCOMO II estimates using the inputs and
  uncertainties
• NEWT NOSTROMO Entry Writing Tool a Delphi
  polling tool that captures Uncertainty

12
New NOSTROMO Concept (U)
PESSIMISM
DATA PAGE
AGC CALCULATOR
REPORTS
NEWT
NOSTROMO
DEFAULT UNCERTAINTY
COMBINATION

• NOSTROMO 0.3.15 offers
• multiple methods for inputting Uncertainty
• combination of subcomponents
• more reporting capability
• AGC methodology
• Pessimistic SLOC estimation

13
Distributions of Uncertainty (U)
Level 1 - Certain
NOSTROMO assumes ceiling and floor limits of the
highest and lowest default values for each COCOMO
II Scale Factor and Cost Driver
Level 2 - Low Uncertainty - Normal
Level 3 - Medium Uncertainty - Triangle
Level 4 - High Uncertainty - Uniform
Setting n-1
Setting n
Setting n1
NOSTROMO changes the distribution and sets the
high and low points based on the level of
uncertainty with each factor
14
NOSTROMO Application (U)

• Microsoft Excel 2002 worksheet with attached
  macros and Visual Basic code
• Currently on version 0.3.15 (as of 10/19/06)
• Does not use outside applications for Monte Carlo
  simulation (standard random number generation
  from Visual basic)
• Limited testing performed on a proof-of-concept
  application
• Suggest limiting Monte Carlo run size to 500-2000
  runs
• Generates two pages of output Data and Charts
• Data page contains all outputs from Monte Carlo
  simulation, charting data, and histograms
• Charts page contains Confidence Intervals for
  output of COCOMO II and Putnam models, charts,
  and histograms
• NOSTROMO Data Sheet and CM Control Sheet are
  Developer pages and should not be removed

15
NOSTROMO Input Sheet Notional Data (U)
Software Item Name
Scale Factors
Function Buttons
Code Size
Pessimism Toggle
Conditional Formatting
Cost Drivers
Pessimism toggle automatically sets the
Requirements Evolution and Volatility to 0 to
avoid double counting
Iterations
Putnam Productivity
16
NOSTROMO Pessimism Worksheet (U)
Allows the user to develop their own relative
size numbers for Pessimism and use in NOSTROMO
17
NOSTROMO Outputs 5000 SLOC (U)
5000 SLOC Pessimism On
5000 SLOC Nominal REVL No Pessimism
Pessimism accounts for an 18 increase in Size in
the 50th Percentile NOSTROMO Run
18
NOSTROMO Outputs 500000 SLOC (U)
500 KSLOC Pessimism On
500 KSLOC Nominal REVL No Pessimism
Pessimism accounts for a 20 increase in Size in
the 50th Percentile NOSTROMO Run
19

• Demonstration

20
Future NOSTROMO Additions and Efforts (U)

• Addition of Exception Conditions from Ray
  Madachys Expert COCOMO
• Transition to an application environment
  (MS-Access or .NET)
• Expansion of the NOSTROMO tool to address other
  COCOMO Family models with uncertainties
• COSYSMO especially relevant in development of
  complex DoD systems
• COSOSIMO System of Systems models are becoming
  increasingly prevalent in DoD efforts
• Expansion of the NOSTROMO methodology and tool
  into Readiness Level models
• Software Readiness Levels (SWRL)
• Technology Program Management Model (TPMM)

21
Conclusion (U)

• Software size in many development environments
  has a tendency to grow, even late into the
  project development
• Software estimates need to address any
  pessimistic growth potentials and software
  estimators have an obligation to report the risks
  associated with developments that continue to
  grow in size
• New NOSTROMO methodology incorporates pessimistic
  sizing data to predict the worst-case scenario

Analysis of size growth potential gives valuable
insight into the true risks of software
development