Analysing Forecast Adjustment Behaviour

1
Analysing Forecast Adjustment Behaviour

• Michael Lawrence
• UNSW, Sydney, Australia
• Robert Fildes
• University of Lancaster, UK
• Paul Goodwin
• University of Bath UK

An EPSRC Research Project A collaboration between
Lancaster Centre for Forecasting,, and 10
companies, including McBride, Interbrew, Heinz
2
Information affecting the supply chain
Forecast object
The basic model Forecast of Orders f(past
orders) judgemental estimates of promotions
etc. computer forecast judgemental
adjustment
3
Background theory

• Influence of biases and heuristics
• Tversky and Kahneman Many lab field studies
• Anchor and adjustment heuristic implies anchoring
  on the statistical forecast and a too small
  adjustment
• Lawrence and OConnor Lab studies
• In time series forecasting adjustment generally
  too small not too large.
• Hypothesis Adjustment too big so biased high
  (up adjust) or low (down adjust).

4

• Mathews Diamantopolous
• Studied large warehouse operation where forecasts
  routinely adjusted. Found adjustments beneficial
  (altho often only marginal) but biased.
• Lawrence, OConnor and Edmundson
• Studied forecasting practice in 13 large
  multinationals all using only judgement.
• Forecasts biased and inefficient some
  organisations worse accuracy than naïve forecast,
  others good.
• Optimism Bias
• Most widely studied and observed bias
• Benefits overestimated and costs underestimated.
• Hypothesis Positive benefits (e.g. promotion)
  over-estimated negative impacts underestimated.
• Forecasts biased and inefficient.

5
Database of forecasts by company
6
Analysis Methodology

• Analyse by up and down adjustments
• Error measures
• Absolute Percentage Error
• APE act fcst/act
• Forecast improvement
• FCIMP (act - systfc - act finalfc)/act
• Note When FCIMP is ve adjustment has improved
  the forecast.
• Measures of adjustment impact on accuracy
• 1. Comparison of median PEs and APEs
• 2. Forecast improvement - FCIMP

7
Analysis Methodology

• 3. Validity of decision to adjust.
• Conjectured steps in adjustment. (From
  observations of currency forecasting.)
• (a) Decide if statistical computer forecast is
  too low or too high.
• This results in a decision to adjust and a
  direction for the adjustment.
• (b) Decide by how much to adjust the forecast.
• Hence most basic adjustment measure
• How often is the direction for the adjustment
  correct?
• Pool companies A-C as statistical properties
  similar.

8
Results
Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error Median percent error and absolute percent error
N Naïve error Systm fcast error Final fcast error Naive Mdape Systm fcast Mdape Final fcast Mdape FCIMP
No adjust. 3174 2.9 -1.4 -1.4 22 13.6 13.6 n.a.
Up adjust. 4013 2.5 9.8 -9.4 26 20.2 17.6 0.1
Down adjust 3392 1.0 -14.0 2.2 25 20.7 15.7 3.9
9
Observations

• System forecasts much better than naïve and so
  form good anchor point for adjustments.
• Overall, the groups selected for adjustment need
  it (indicated by bias of system forecast.
• UP adjustments biased and make little difference
  to accuracy. (validated by t-test)
• DOWN adjustments unbiased and make considerable
  improvement to accuracy.

10
Impact of wrong side adjustment
Adjustment direction Adjustment right/wrong Systm Forecast. MdAPE Final Forecast. MdAPE FCIMP
NO - 13.6 13.6 -
UP RIGHT 19.8 11.1 7.1
UP WRONG 17.3 37.0 -13.2
DOWN RIGHT 26.4 12.8 -11.7
DOWN WRONG 13.9 27.8 -9.6
11
What percent of adjustments are in the right or
wrong direction?
Direction of adjustment Right Direction Right by adjustment size Right by adjustment size FCIMP
Direction of adjustment Right Direction Smaller adj. Larger adj. FCIMP
NO adj 31 - - - -
UP adj 37 66 55 -0.4 74 4.2 0.06
DOWN adj 31 71 63 1.2 79 20.3 3.9
12
Why is wrong direction picked so often?

• Alternatives
• 1. No valid basis for adjustment just an
  illusion of control affect. (Note that if the
  direction is picked at random, 50 will be
  wrong.)
• 2. Timing effect. E.g. influence of promotion
  anticipated too soon.
• Assume timing effect (or possibly learning
  effect?) If an adjustment is made in the period
  following a wrong sided adjustment, it should be
  more accurate.

13
WRONG FCIMP
45 WRONG -4.0 __ __ 33 WRONG 2.2
Wrong up adjustment (34)
53 UP
17 NO
30 DOWN
Period t t1
After a wrong sided adjustment the accuracy of
judgement is worse.
14
WRONG FCIMP 39 -0.13 _ _ 36 1.6
35 UP
Wrong down adjustment (29)
13NO
51 DOWN
Period t t1
15
Conclusions on wrong direction.

• In period following a wrong adjustment
• Over 80 of forecasts adjusted
• An UP adjustment worsens forecast.
• A DOWN adjustment improves forecast.
• Direction error rates generally show increase.
• Timing effect does not appear responsible for
  wrong direction.
• Illusion of control seems best explanation for
  increasing error rates on adjustment.
• Evidence for optimism bias.
• After two wrong adjustments the following period
  shows a similar pattern.

16
How to improve adjustments

• Incorporate restrictiveness, guidance and better
  feedback into software
• Prevent small adjustments to system forecasts.
• Prevent an UP adjustment following a wrong sided
  adjustment unless special reason is given.
• Provide systematic feedback on results of
  adjustment activity.
• Schedule periodic meetings to review impact of
  adjustments on accuracy and chart progress.
• While the software can play a role, the major
  changes needed are organisational.