Session 6: A guide to choosing forecast models

1
Session 6 A guide to choosing forecast models

• Demand Forecasting and
• Planning in Crisis
• 30-31 July, Shanghai
• Joseph Ogrodowczyk, Ph.D.

2
A guide to choosing forecasting models

• Session agenda
• Judgment modeling Using expert knowledge to
  provide forecasts
• Mixed methods Constructing forecasting
  frameworks from expert knowledge
• Quantitative modeling Using statistical
  techniques to provide forecasts
• Criteria for combining or adjusting forecasts

3
A guide to choosing forecasting models

• Review
• The last session explained some short run
  forecasting methods
• These techniques are used to quickly obtain
  forecasts
• How do we produce more accurate forecasts for
  longer time horizons?
• We need more sophisticated models
• What are the different types of forecasting
  models?
• What are the criteria for choosing which model to
  use in a particular instance?

4
A guide to choosing forecasting models

• Judgment Experts are providing forecasts
• Mixed Applying forecasting guidelines to
  statistically produced forecasts
• Quantitative Using statistical techniques to
  generate forecasts
• (Armstrong and Green 2009)

5
A guide to choosing forecasting models

• Judgment model types Experts (in forecasting or
  a product line) are producing the forecasts via
• Unaided judgment
• Expert forecasting
• Decomposition
• Conjoint analysis
• Intentions / Expectations
• Role playing / Simulated interaction
• Structured analogies

6
A guide to choosing forecasting models

• Judgment models
• Unaided judgment Forecasts made without the use
  of formal forecasting methods
• Conditions of use
• Large changes are not expected
• Forecasts are not used for policy analysis
• Highly predictable/repetitive atmosphere
• Advantages
• Quick to produce forecasts
• Inexpensive if only a few forecasters are needed
• Accuracy can be improved when forecaster obtains
  rapid feedback

7
A guide to choosing forecasting models

• Judgment models
• Expert forecasting Experts are asked to provide
  forecasts
• Conditions of use can vary
• Easy access to experts
• Motivated and knowledgeable experts
• Need for confidentiality
• Low dispersal of knowledge
• Limited time for forecast production

8
A guide to choosing forecasting models

• Judgment models
• Expert forecasting
• Nominal group technique A one-round survey for
  forecasts in which experts possess similar
  information
• Estimate-talk-estimate A three-round survey
  where experts possess different information.
  Between forecast estimations, the participants
  are asked to have a discussion
• Delphi method At least two survey rounds with
  results of the previous round summarized for
  participants
• Prediction markets Incentive-based arrangements
  that use markets to aggregate, in the form of
  prices, information that is dispersed among
  participants.

9
A guide to choosing forecasting models

• Judgment models
• Decomposition Breaking down the estimation task
  into a set of components to produce a target
  forecast
• Method is to ask experts to forecast parts of the
  whole and then aggregate for the whole forecast
• Conditions of use can vary
• Forecasting a highly complex system
• Forecasting in an unfamiliar metric/market/custome
  r
• Higher confidence in component forecasts than in
  the target forecast
• Decomposition can be additive or multiplicative

10
A guide to choosing forecasting models

• Judgment models
• Conjoint analysis Survey method based on
  characteristics of a product
• Decomposition used pieces of the target forecast.
  Conjoint uses characteristics of the forecasted
  item
• Sometimes used to construct a forecast manually
  but often used as the basis for a regression type
  analysis (mixed method)
• Conditions of use
• Large changes in demand expected
• To be used for policy analysis
• Survey users of the forecasted item instead of
  experts with market / forecasting knowledge

11
A guide to choosing forecasting models

• Judgment models
• Conjoint analysis
• Popular in marketing
• Consumers are asked to rank / assign a value
  trade-offs between multi-dimensional alternatives
• Automobiles, soft drinks, computers, checking
  accounts, hotel accommodations, etc.
• Results can be plausible
• Major drawback is the potential hypothetical
  nature of the survey
• Consumers are not deciding between actual goods
  and services

12
A guide to choosing forecasting models

• Judgment models
• Intentions / expectations Survey method based
  on intended future behavior
• Conjoint asked consumers for preferences on
  specific item characteristics. Intentions asked
  consumers for anticipated future behavior
• Conditions of use
• Large changes in demand are expected
• Relatively little conflicts among forecasters
• Forecasts are not used for policy analysis (a
  need to choose between different courses of
  action)
• Disadvantages
• Research shows that intentions are biased as
  measures for prediction
• Research is inconclusive about how best to
  measure intentions

13
A guide to choosing forecasting models

• Judgment models
• Role playing / Simulated interaction Customers
  are asked to act out prospective interactions in
  a realistic manner
• Conditions of use
• Important as a tool in forecasting within
  conflicts (threats of striking workers, jury
  reactions, assessing outcomes from different
  strategies)
• Small number of parties interacting
• Need to predict in situations involving large
  changes
• Advantages
• Decisions are often difficult to forecast if they
  are the result of a series of actions
• Role playing can be used to simulate the actions
  and reactions between the parties

14
A guide to choosing forecasting models

• Judgment models
• Structured analogies Surveying experts to
  compare analogous situations and using the
  outcomes of the analogies as the forecast for
  target
• Conditions of use
• Large changes are expected
• Difference among forecasters
• Similar cases exist
• Methodology
• Describe the target situation
• Select experts
• Identify and describe analogies
• Ask the experts to describe as many analogies as
  they can without considering the extent of the
  similarity to the target situation

15
A guide to choosing forecasting models

• Judgment models
• Structured analogies (Green and Armstrong 2007)
• Methodology
• Rate similarity
• Ask the experts to list similarities and
  differences between their analogies and the
  target situation, and then to rate the similarity
  of each analogy to the target
• Ask them to match their analogies outcomes with
  target outcomes
• Derive forecasts
• Set up the rule system (criteria) for choosing
  the analogy to use for the target forecast before
  interviewing experts
• Many rules are reasonable
• For example, one could select the analogy that
  the expert rated as most similar to the target
  and adopt the outcome implied by that analogy as
  the forecast

16
A guide to choosing forecasting models
17
A guide to choosing forecasting models

• Mixed modeling Forecasting using statistical
  models created from expert rules
• The development of statistical techniques is
  guided by experts rules
• Rules Criteria, inputs, and other variables
  that experts use in producing forecasts using
  judgment methods
• Mixed model types
• Quantitative analogies
• Expert systems
• Rule-based forecasting
• Bootstrapping

18
A guide to choosing forecasting models

• Mixed model types
• Quantitative analogies Data from analogous
  situations are used as input to derive the target
  forecast
• Sample rule When data are not available, use
  data from a similar situation
• Conditions of use
• Not a good knowledge of the relationships in the
  data
• Cross sectional data (Data across multiple units
  for the same time period)
• Not used for policy analysis
• Advantages
• Statistical forecasting methodologies can be used
  on input data
• Allows for increased observations for a data-poor
  series such as new products

19
A guide to choosing forecasting models

• Mixed model types
• Expert systems Statistical models are designed
  to represent the rules used by experts in the
  forecasting process
• Rules are based on knowledge of target area
• Models are based directly on the rules
• Information on rules can be found in research
  papers, surveys, and interviews
• Expert systems should be easy to use, incorporate
  the best available knowledge, and reveal the
  reasoning behind the recommendations they make

20
A guide to choosing forecasting models

• Mixed model types
• Rule-based A system to develop and apply
  weights for combining extrapolations
• Time series extrapolation Univariate time series
  forecasting methods
• Rules result in each extrapolation method being
  assigned a weight based on trends, seasonality,
  and historical data
• The compilation forecast is the sum of the
  weighted extrapolation methods
• Knowledge for rules can be obtained through
  expert judgment, empirical research, and theory
• Guidelines for rules
• Give separate consideration to level and trend
• Use different models for short- and long-run
  forecasts
• Damp the trend as the forecast horizon increases

21
A guide to choosing forecasting models

• Mixed model types
• Bootstrapping Translating experts rules into a
  quantitative model
• Difference from expert system
• Based on inference about experts rules
• Requires repeated sampling
• Specific model design used in the translating
  (regression)
• Model is produced by regressing the forecasts
  produced upon the information that the expert
  used
• Guidelines for bootstrapping
• Use experts who differ
• Use simple analysis to represent behavior
• Note Quantifying the variables used by the
  experts can greatly affect the validity of the
  model

22
A guide to choosing forecasting models
23
A guide to choosing forecasting models

• Quantitative modeling Using statistical
  techniques to provide forecasts
• Forecasts are produced by statistical models
  modeling the behavior generating the data series
  (historical data or external variables)
• Experts suggest appropriate variables
• Stepwise regression, state-space models, and
  Bayesian techniques are statistical tools that
  can be applied either to univariate or
  multivariate models
• Quantitative model types
• Extrapolation/neural networks
• Statistical regression
• Segmentation
• Index

24
A guide to choosing forecasting models

• Quantitative model types
• Extrapolation Statistical models using only
  historical information to produce forecasts
• Univariate times series, Holt-Winters
  (exponential smoothing), Box-Jenkins and ARIMA
  (ARMA), autoregressive, linear trend (trend using
  only time), simple (single) regression
• From session 4, Naïve model and moving average
  models (without and with confidence intervals)
• Assumes that all the necessary information for
  forecasting is contained in the historical data
• Can also be used for cross-sectional data
• To estimate the probability of a new hire lasting
  more than a year, analyze the percent of the
  previous 50 applicants lasting more than a year

25
A guide to choosing forecasting models

• Quantitative model types
• Neural net Models using complex,
  interdependent, variable relationships to produce
  forecasts
• Inspired by the behavior of biological neurons
• Often a black box for understanding the
  relationships
• Conditions of use
• Best for quarterly or monthly data
• Discontinuous series
• Several-period lag between forecasting and
  forecasted periods
• Advantages
• Does not need to fully understand the
  relationship of explanatory variables
• Estimates nonlinear functions well

26
A guide to choosing forecasting models

• Quantitative model types
• Statistical regression (Econometrics) Using
  statistical methods to estimate the relationships
  of variables based on theory, prior studies, and
  expert knowledge
• (Augmented) Dickey-Fuller, vector autoregressive
  (VAR), error correction models (ECM), multiple
  regression
• Parameter estimates (elasticities, measures of
  influence on dependent variable by independent
  variable) can be obtained through using least
  squares or maximum likelihood
• Models use theory and expert knowledge to select
  the explanatory variables
• Dependent variable is the forecasted item and
  independent (explanatory, causal) variables are
  those which explain the behavior of the
  dependent variable

27
A guide to choosing forecasting models

• Quantitative model types
• Segmentation Forecasting a heterogeneous whole
  through forecasting of parts of the whole
  separately
• When the dependent variable responds in different
  ways to the independent variables
• Forecasts for the parts will be created from
  separate econometric models because of the
  difference in causal effects
• Airline tickets Business class and recreational
  coach class customers respond differently to
  price changes
• Better to forecast each type of passenger and
  then aggregate
• Similar, but not the same, as bottom-up
  forecasting
• Bottom-up down not necessarily need multiple
  model types

28
A guide to choosing forecasting models

• Quantitative model types
• Index Forecasting the value of the dependent
  variable by adding values of the independent
  variables
• Improper linear models. Unit-weight is a special
  case when variables are weighted evenly
• Explanatory variables can be subjective and
  assigned a 1 or 0 depending on if they are
  present or absent, respectively
• Explanatory variables can be quantitative data
  that have been normalized (units mathematically
  removed)
• Weights can be chosen by experts
• Values of the dependent variable can be used to
  forecast the probability of an event
• Example Factors contributing to drug use in
  adolescents
• Grades, parent relationship, self esteem, etc.
  (Bry et al. 1982)

29
A guide to choosing forecasting models
30
A guide to choosing forecasting models

• Criteria for combining or adjusting forecasts
• Two main questions on combining
• Are several methods producing useful forecasts?
• If so, how can they be combined?
• Two main questions on adjusting
• Is there a need to adjust the forecast because of
  omitted data?
• If so, how should the forecasts be adjusted?
• At what level in the hierarchy, over what time
  horizon, and by how much (percents or quantities)?

31
A guide to choosing forecasting models

• Criteria for combining or adjusting forecasts
• Best practices for forecasting include a business
  process for answering the questions on combining
  and adjusting.
• Tool is the demand management of the SOP process
• Tips for combining forecasts
• Use different data for different models
• Use equal weight unless there is strong evidence
• Use expert knowledge to vary the weights
• Collect historical data on weight accuracy

32
A guide to choosing forecasting models

• Criteria for combining or adjusting forecasts
• Tips for adjusting forecasts
• Adjusting forecasts can be necessary when
• Recent events are not fully reflected in the data
• Experts possess reliable knowledge about future
  events
• Key variables were omitted from the models
• To gain consensus for adjustment level, time
  horizon, and degree
• Construct scenarios with representative events
• Ask experts to provide explanations of outcomes
• Be careful to avoid boomerang effect

33
A guide to choosing forecasting models
34
A guide to choosing forecasting models
35
A guide to choosing forecasting models

• References
• Armstrong, J. Scott and Kesten Green. 2009.
  Selection Tree for Forecasting Methods.
  Forecasting Principles (April) http//www.forecast
  ingprinciples.com (accessed May 2009).
• Bry, Brenna H., P. McKeon, and R.J. Pandina.
  1982. Extent of drug use as a function of a
  number of risk factors. Journal of Abnormal
  Psychology 9 273-279., in Armstrong J. Scott,
  ed. 2001. Principles of Forecasting A handbook
  for researchers and practitioners. Norwell,
  Mass. Kluwer Academic Publishers.
• Green, K.C. and J.S. Armstrong. 2007. Structured
  analogies for forecasting. International Journal
  of Forecasting 23 365-376., in Armstrong J.
  Scott, ed. 2001. Principles of Forecasting A
  handbook for researchers and practitioners.
  Norwell, Mass. Kluwer Academic Publishers.