Uncertainty surrounding the Cone of Uncertainty

1
Uncertainty surrounding the Cone of Uncertainty

• Todd Little

Its tough to make predictions, especially about
the future. Yogi Berra
2
IEEE Software, May/June 2006
3
Managing the Coming Storm Inside the Tornado
Project Kickoff
Team Unity
4
Were not in Kansas Anymore
Developer Hero
Reorg
Testing
5
Hurricane Rita
6
About Landmark

• Commercial Supplier of Oil and Gas Exploration
  and Production Software
• Users are Geophysicists, Geologists, Engineers
• Subsidiary of Halliburton Energy Services
• Integrated suite of 60 Products
• 50 Million lines of code
• Some products 20 years old

7
Landmark Product Suite
Geophysics
Engineering
Geology
8
Data in the Portfolio

• 3 years of data (1999-2002)
• 570 projects
• 106 valid (Shipped commercial product)
• Remainder Currently active, placeholder
  projects, internal projects, non-commercial
  releases, deferred projects, etc.
• Relatively Unbiased.
• Each week the Program Manager recorded the state
  of the project and the current release estimate.
• No improvement goal bias

9
Data from LGC
Developing Products in Twice the Time
10
Data from Tom DeMarco
Its déjà vu all over again
11
Cumulative Distribution Curve for Actual/Estimate
(DeMarco)
12
CDF Distribution Curve (LGC)
13
Probability Distribution Curve
14
Its tough to make predictions, especially about
the future. Yogi Berra

• That idea is so damned nonsensical and impossible
  that I'm willing to stand on the bridge of a
  battleship while that nitwit tries to hit it from
  the air.
• Newton Baker, U.S. secretary of war in 1921,
  reacting to the claim of Billy Mitchell (later
  Brigadier General Mitchell) that airplanes could
  sink battleships by dropping bombs on them.
• Heavier-than-air flying machines are impossible.
• Lord Kelvin, British mathematician, physicist,
  and president of the British Royal Society,
  spoken in 1895.

15
How does Estimation Accuracy Improve Over Time?

• At the end of each phase, compare the most
  current estimate with the resulting end date.
• Envisioning
• Planning
• Developing

16
Estimation Accuracy (Boehm)
2
0.5
17
So what does LGC data look like?
18
Landmark Cone of Uncertainty
19
Cumulative Distribution (CDF) Curve
20
But is Uncertainty Really Reduced?
Take away an ordinary persons illusions and you
take away happiness at the same time. Henrik
Ibsen--Villanden
21
Remaining Uncertainty
22
The Pipe of Uncertainty
2
0.5
23
Does Landmark Suck at Estimation?

• A severe depression like that of 1920-21 is
  outside the range of probability.
• Harvard Economic Society, Weekly Letter,
  November 16, 1929.
• I think there is a world market for about five
  computers.
• Thomas J. Watson, chairman of IBM, 1943.
• They couldn't hit an elephant at this dist
• General John B. Sedgwick, Union Army Civil War
  officer's last words, uttered during the Battle
  of Spotsylvania, 1864

24
Estimation Quality Factor (EQF)
Actual Value
Blue Area Red Area
Value to be Estimated
EQF
Initial Estimate
Actual End Date
Elapsed Time
Link to article by Tim Lister
25
EQF from Lister/DeMarco

• An EQF of 5 is pretty good (i.e. averaging about
  1/5 or 20 percent off.)
• The median for schedule estimating is about a 4,
  with the highest sustained scores at 8 to 9.
• Lister and DeMarco have never known anybody to
  sustain a 10 (just 10 percent off).
• Typical disaster project is 1.8

26
EQF Distribution Curve (LGC)
EQF for duration has a theoretical minimum of 2.0
27
We slip one day at a time, EQF2
Actual Value
Blue Area Red Area
Value to be Estimated
EQF
Initial Estimate
Actual End Date
Elapsed Time
28
(EQF-2) Distribution Curve (LGC data)
29
LGC Estimation Quality

• LGCs EQF measurement is pretty good.
• Our p(50) is 4.8, versus an industry average
  around 4 and a best sustained in the 8-10.
• Our p(10) is 2.8, which is not bad.

30
Uncertainty
Know that we know
Know that we dont know
Knowable
Unknowable
Dont know that we dont know
Dont know that we know
31
Uncertainty
Know that we know
Know that we dont know
Planning
Risk Management
p10
p50
Knowable
Unknowable
Dont know that we dont know
Dont know that we know
Uncertainty Management
p90
32
The Cone of Uncertainty
33
Successful Projects?