Results from Surveys on Software Effort Estimation

1
Results from Surveys on Software Effort Estimation

• Kjetil Moløkken
• Magne Jørgensen

2
Topics of interest

• For researchers
• Obtain a summary of available results.
• Focus on methodological aspects for further
  research.
• For practitioners
• Ability to benchmark own organization.
• Examples on how to critically interpret research
  results.

3
Content

• What is an estimate?
• Measuring estimation performance.
• Previous surveys
• Methodological aspects.
• Own survey.

4
What is an estimate?

• Who is it for?
• Internal estimates. The most likely?
• External estimates. Communicated to
  managers/customers etc. Affected by price-to-win?
• At what stage?
• Bidding process or equivalent.
• Planning phase.
• Re-estimation(s).

5
How do previous surveys define an estimate?

• Poor or no definitions of either estimate type or
  estimation stage.
• E.g. price-to-win is defined as a method.

6
Estimation accuracy vs. estimation bias

• Ex. 10 projects which finish 50 over estimated
  effort, and 10 projects which finish 50 under
  estimated effort.
• Two average properties
• Average accuracy deviation of 50.
• Average estimation bias of 0!

7
Previous Surveys (n10)

• Response rates from 23 to 51.
• Only one used interviews, the rest mailed
  questionnaires.
• None had random samples.
• Only similar western countries.

8
RQ 1 To what extent do software development
projects deviate from the original plan, with
regard to cost, schedule and functionality?
9
RQ 2 Which methods are used to estimate software
effort, and do these systematically differ in
accuracy? (1)
10
RQ 2 Which methods are used to estimate software
effort, and do these systematically differ in
accuracy? (2)

• Heemstra and Kusters
• Found that estimation accuracy did not improve
  when formal models were used.
• Projects estimated with Function Point Analysis
  had larger overruns than the other projects.
• Bergeron and St-Arnaud
• Price-to-win methods and algorithmic models were
  associated with less accurate estimates.
• The methods associated with the most accurate
  estimates were based on analogy and expert
  opinion.
• However, this may be coincidental since few
  respondents used price-to-win and algorithmic
  methods.

11
RQ 3 How important is accurate effort estimation
perceived to be, and to what extent is the level
of accuracy considered a problem in the software
industry?

• Lederer and Prasad On a five point Likert scale,
  the average importance rating reported by
  managers was 4.17.
• Moores and Edwards Found that 91 of the
  responding managers answered yes to the
  question do you see estimation as a problem?,
  while only 9 answered no. An interesting
  finding is that the accepted level of estimation
  accuracy was typically /- 20.

12
Methodological aspects

• Uncertain measures.
• Non-random samples.
• No differentiation of estimation bias and
  estimation accuracy.

13
Example Standish
14
Summary

• 60-80 of software projects encounter overruns.
• Overruns are typically 30-40 over budget and/or
  schedule.
• Expert judgment is the most common estimation
  method.
• There exists no conclusive evidence that
  estimation models are more accurate than expert
  based methods.

15
Own survey approach

• Eighteen companies visited
• Internal development (massmarket and internal
  use).
• Contraction (public and private).
• Stratified random sample.
• Personal interviews with senior management and
  two to four project managers at each company.
• 38 projects have currently been analyzed.

16
Estimation method (n38)

• 31 projects used pure expert estimation.
• 7 used combination of expert and model.
• Six use-case based.
• One company specific model.

17
Impact from choice of estimation method

• No observed impact on either estimation accuracy
  and bias.

18
Expert estimation approach

• 100 use bottom up-estimation
• About 30 use of WBS.
• 20 use of database with previous projects
• 20 use of checklists
• 90-95 use of group based estimates.

19
Choice of development method

• Differentiation between two main groups
• Incremental/evolutionary (incl. XP) approach
• Waterfall based development approach

20
Impact of development method

• Incremental/evolutionary projects were more
  accurate
• Increased accuracy
• Reduced bias
• (Paper accepted for Profes 2004 based on n22)

21
Possible effect?

• Amount of customer contact more important than
  development method reported?

22
Summary

• 70-75 (60-80) of software projects encounter
  overruns.
• Overruns are typically 27 (30-40) over budget.
• Expert judgment is the most common estimation
  method.
• There exists no conclusive evidence that
  combination of estimation methods are more
  accurate than expert based methods.

23
Lessons learned?

• The magnitude of overruns is not necessarily as
  catastrophic as media and consultants often
  depict.
• The most common estimation method has received
  relatively little attention from the scientific
  community.