CHAPTER 19: HOW ACCURATE IS CBA?

1
CHAPTER 19 HOW ACCURATE IS CBA?

• Purpose To evaluate the accuracy of CBA

2
CBA depends on accuracy

• One way of assessing accuracy is by performing
  analyses of the same project at different times
  and to compare the results.
• There are three temporal CBA types
• Ex ante (EA) CBA prior to project t year0 T
  the end of the project
• Ex post (EP) CBA after the project tgtT
• In media res (IMR) CBA during the project 0lttltT

3
SOURCES OF ERROR IN CBA STUDIES

• Errors may arise for a number of reasons,
  including managers bureaucratic lens
  (organizational perspective/self-interest).
• There is evidence that managers generally tend to
  overestimate benefits and underestimate costs. It
  is possible to somewhat offset such biases with
  the use of an independent analyst.

4
Most error arises from four causes

• Omission errors.
• Forecast errors.
• Valuation errors.
• Measurement errors.

5
Omission Errors

• Exclusion of an effect category due to the low
  probability of occurrence, highly technical
  nature of the project, scientific disagreements
  over the physical effects, etc.
• Double counting. Often from counting an effect
  in both the primary and secondary market. This
  is likely to be less of a problem as time (and
  the project) goes on.

6
Forecasting Errors

• Can occur from inherent difficulties, such as
  difficulty in predicting technological change.
• They can also arise from cognitive biases,
  changing project specifications, or for strategic
  reasons

7
Difficulty of accurate forecasting

• Increases as projects are more complex, unique,
  further in future, and involve unknown
  cause-and-effect relationships.
• Unknown cause-and-effect usually occurs when
  there is uncertainty in the underlying scientific
  relationships, or when the project is new and
  untried.

8
Difficulty of accurate forecasting

• Cognitive biases are endemic. People
  systematically underweight low probability bad
  events and systematically overweight low
  probability good events.
• Selection bias
• Changing project specifications. Large and
  complex projects are often modified when
  underway when modified, EA/EP comparisons may
  end up comparing apples and oranges.

9
Measurement Errors

• The extent of measurement error depends on the
  quality of measurement equipment, e.g., the
  accounting system.
• Accurate monetary estimates of the value of some
  effects (i.e., time saved or lives saved) have
  been scarce
• Unanticipated relative price changes (can also be
  seen as a forecasting error) can have significant
  effects, especially on big infrastructure
  development projects.

10
DISTRIBUTION OF NET BENEFIT OVER TIME

• Net benefit (NB) has a probability density
  function
• Ft (NBt, ???????
• Where???????mean???????varience?????????Expected
  value of NB

11
(No Transcript)
12
The effects of time on the four kinds of CBA
errors

• Omission errors these errors decline over time.
• Forecasting errors these errors decrease as the
  effects occur. Errors could still be a problem
  with the counterfactual.
• Measurement errors these are not really
  affected by the passage of time.
• Valuation errors these decrease over time, but
  could still be present at T.

13
SUMMARY OF THE CBAs OF THE COQUIHALLA HIGHWAY

• The Coquihalla highway is a toll road that was
  actually built in British Columbia in three
  phases (I 1986 II 1987 III -1990)

14
(No Transcript)
15
Three CBAs of the Coquihalla

• Waters Meyers (WM) 1986 EA
• Mallery (MLY) 1987 IMR.
• Boardman, Mallery, Vining (BMV) 1993 more
  EP.

16
Analysis of the Differences among the CBAs
17
Omission Problems Differences

• None of the studies explicitly included the
  opportunity cost of land (minimal overall
  effect), environmental costs (probably minimal
  since several environmental costs were
  incorporated into construction costs), or
  regional development benefits (possibly related
  to positive externalities)

18
Forecasting Differences 1

• Traffic volume data forecasting is of most
  interest because it is crucial to the accuracy of
  a highway CBA)
• WM This CBA slightly overestimated initial use
  and seriously underestimated future use.
• MLY This CBA estimated 3 annual traffic growth
  (5 actual) and predicted a 20 increase after
  Phase III (40 actual).
• BMV This CBA used actual data through 1990 and
  then projected 5 annual growth rates (doesn't
  say how accurate it was).

19
Forecasting Differences 2

• Time and distance savings estimates varied due to
  changes in the final design and different sources
  used to measure distance (time, therefore, also
  varied since its based on distance).
• Also, the three studies used different methods
  to estimate congestion savings on alternate
  routes.
• MW and MLY used a flat 20-minute time savings
  during congested periods (less for local
  traffic).
• BMV used a kph savings for the alternate routes.

20
Forecasting Differences 3

• Accident rate estimates varied. Safety benefits
  accrue from both the shorter distance traveled
  and from driving on a safer highway.
• MW and MLY estimated the benefits by multiplying
  the distance (either total distance for a safer
  road or a shorter distance for the shorter road)
  by an estimate for accident rates for that type
  of highway. Both MW and MLY underestimated the
  benefits.
• BMV used actual data on accident rates.

21
Estimation/Measurement Differences

• There were difficulties in accurately measuring
  maintenance and construction costs
• Maintenance expenses Even EP studies can suffer
  from measurement errors.
• Construction costs These can be difficult to
  measure even after the project is completed.

22
Valuation Differences

• These differences can arise from differing
  estimates of any relevant parameter.
• Studies similar in valuing time savings, but
  different in estimation of gross wage rates and
  in vehicle operating costs.
• Value of safety benefits. WM and MLY used
  similar value of life estimates BMV used a much
  higher value for an avoided fatality in
  accordance to recent research.
• Terminal value. Used 75-85 of initial
  construction cost as the terminal value

23
Conclusions arising from the IMR, EP versus EA
Comparisons
24
Did EA and IMR Analyses Have Overall Predictive
Capability?

• Both the EA analysis and the IMR analysis showed
  significantly less NPV than the EP study.
• Actual differences could also be understated by
  some differences in costs and benefits
  estimations canceling each other out or
  offsetting each other within the same category.
• A single study cant answer all the questions on
  the accuracy of a CBA.

25
Other Lessons from this Comparison Study

• Errors can compound one another (i.e., when one
  estimate with error depends on other estimates
  with error, the error becomes compounded).
• One should attempt to generate at least rough
  independent estimates.
• Strategic bias can affect alternatives as well as
  favored projects, i.e., they could overestimate
  costs and underestimate benefits on the
  alternatives.