Forecasters

1
Forecasters view ofclimate-changeMethodology
also counts

• J. Scott Armstrong
• The Wharton School
• University of Pennsylvania
• Kesten C. Green
• Business and Economic Forecasting Unit
• Monash University
• March 9, 2009
• File Heartland 09-JSA v36

2
Role of methodology

• Inputs
• Data
• Problems with the data
• Lack of disclosure
• Changes in measurement
• Relationships
• Which variables
• Direction and effects of variables
• Forecasting methodology
• How to integrate inputs to produce forecasts and
  prediction intervals for policy analysis

2
3
Climate change forecasting problem

• Structuring the forecasting problem (JSA)
• Auditing the forecasting process (JSA)
• Selecting a proper forecasting method (JSA KCG)
• Testing forecast validity (KCG)

3
4
The forecasting problems

• Rational public policy requires scientific
  forecasts of the direction and magnitudeand
  confidence intervalsfor
• 1. Climate change
• 2. Effects of climate changes (costs benefits)
• 3. Effects of alternative policies (ditto)
• We have been unable to find any scientific
  forecasts for 1, climate change warming or
  cooling. (Green, Armstrong Soon 2009)
  
• Our search for 2 3 continues, also without
  success to date.

5
Handouts for Armstrong Green talks

• References for papers cited in our talks /
• WWF solicitation based on false advertising?
• Boxer vs.Armstrong Wager
• Gore vs. Armstrong Prediction Market.
• Suggestions on what to do?
• Analogies study
• WWF action steps
• CO2 policy forecasting
• Climate Change Forecasts are Useless for Policy
  Making

6

• Forecasting methodology
• Structuring the forecasting problem
• Auditing the forecasting process
• Selecting a forecasting method
• Validation tests

6
7
Auditing IPCC forecasts g

• IPCC projections of global temperature change
  used improper procedures. Green Armstrong
  audit showed
• IPCC authors violated 72 forecasting principles.
• 2. Forecasts by scientists, not scientific
  forecasts.
• 3. No proper evidence on predictive validity
• Green, K. C. J. S. Armstrong (2007)

7
8
Audit of Polar bear (PB)population forecasting

• Polar bear population has grown in recent decades
  due to restrictions on hunting.
• Government forecast global warming assumed to
  reduce sea ice.
• Analysis of 5 years of data (with outlier
  removed) related PB population to ice.
• PB experts used a and b to forecast rapid
  decline in PB population over 50 years.
• Government assumed that an endangered
  classification would boost the PB population. No
  forecasts for this, or for alternative policies.

9
Polar bear audit results
Only 12 of the principles were properly
applied. Government researchers declined to
provide peer review or to answer questions we had
about their paper. Offered to act as reviewers
for the journal. Government researchers declined
invitation to review or to provide commentary.
Armstrong, Green, Soon (2008)
10
Example of a principle Full disclosure

• Government researchers used complex mathematics
  to forecast PB populations.
• We requested the data they used on the PB
  population.

11
Here are the data provided to us in our requests
for our audit of the U.S. Fish and Wildlife
Services forecasts

12
Current polar bear situation

• Lacking scientific forecasts, the U.S. Fish and
  Wildlife Services classified the PB as
  threatened.
• Armstrong, Green Soon (2008)

13
Announcing the Boxer-Armstrong charity wager

• Senator Barbara Boxer supports government
  forecasts that the polar bear population will
  decrease by about 23 over the next ten years.
• With an upward trend over recent decades and high
  uncertainty, Armstrong, Green Soon forecast no
  decline in the PB population.
• Professor Armstrong challenges Senator Boxer to a
  5,000 charity wager on the polar bear population
  for the next ten years.
• An independent two-person panel would determine
  which forecast was closest to the actual
  population at the end of 2018.

14
Senator Boxers response as of March 8

15
Policy based on GW forecastsWorld Wildlife Fund

• WWF TV commercials In an appeal for members and
  donations, the ads claim that due to global
  warming, the number of polar bears is decreasing
  rapidly, thereby putting the species at risk of
  extinction.
• WWF website says The general status of polar
  bears is currently stable, though there are
  differences between the populations. Some are
  stable, some seem to be increasing, and some are
  decreasing due to various pressures. The status
  of several populations is not well documented.

16
False advertising by the WWF?

• Since Dec. 12, 2008, I have made attempts to
  contact the CEO and various Trustees of the WWF
  to raise this issue of what appears to be false
  advertising
• I also offered to report on their position at
  this conference.

17
WWF response as of March 8
17
18
Prevalence of false advertising

• Are many organizations engaged in false
  advertising with respect to global warming?
• If so, what might be done to eliminate such false
  advertising?

19
WWF action steps?

• Please put your suggestions on the sheets that we
  have passed out.
• I will summarize these on theclimatebet.com

20

• Forecasting methodology
• Structuring the forecasting problem
• Auditing the forecasting process
• Selecting a forecasting method
• Validation tests

20
21
Selecting a proper forecasting method

• Evidence-based forecasting methods
• Expert judgment
• Extrapolation
• Causal models
• Forecasting methods that have been
  experimentally tested for efficacy in given
  situations

21
22
Expert judgment

• Unaided expert judgment not useful given high
  complexity and high uncertainty. (Tetlock 2005)
• Applies with respect to
• Accuracy
• Experts statements about confidence
• Agreement among experts

22
23
Evidence-based expert judgment methods (use
structure)

• Delphi
• Prediction markets
• Structured analogies
• Causal models

24
Delphi
Delphi A multi-round survey of unbiased
experts who make anonymous forecast with
feedback on predictions and reasons
given between rounds. Assumes that experts can
make useful forecasts. It also helps to assess
expert opinions in an objective manner and
provides decision makers with reasons.
24
25
Prediction markets

• We are currently examining the use of prediction
  markets
• 1) assess expert opinion
• 2) forecast climate change
• Potential problems
• a. If individuals cant forecast, does combining
  their forecasts help?
• b. Is a market for long-term bets feasible
• c. Would a series of short-term bets be useful?
• d. Is play money adequate?
• e. Might people manipulate the market?

25
26
Prediction markets can also summarize prevailing
opinions in an objective manner

• When will Manmade Global Climate Change next be
  disputed by a major media outlet? (HubDub.com)

Time span Market forecast ()
2008-2009 28
Year by year estimates . . . . .
2018 or later/never 27
26
27
Gore vs. Armstrong conducted via a prediction
market

• To stimulate work on validation of alternative
  methods to forecast climate change.
• I have informed Mr. Gore of this prediction
  market and asked for his comments and suggestions.

28
Gores response as of March 8
28
29
Progress on prediction market

• Assume Armstrong and Gore made a ten-year bet
  starting January 1, 2008.
• Armstrong forecast no change in global mean
  temperature.
• Gore implied that global mean temperatures would
  increase at least as rapidly as the IPCCs 1992
  projection of 0.03oC per year
• Criterion mean absolute errors of annual
  forecasts for the ten-year period using UAH
  global mean temperature record.

29
30
Current betting on a play-money market (hub-dub)

• Armstrong by 62 as of March 4, 2009 (100
  bets)
• http//tinyurl.com/gore-armstrong-bet

31
Plans on prediction markets

• Alternative criteria (e.g., Hadley temperatures)
• Real money sites
• Add information via comments
• Vary time horizon (e.g., one-year ahead?)
• Compare with Delphi expert panel
• Graefe, A. J., J. S. Armstrong, K.C. Green (2009)

31
32
Handouts for Armstrong Green talks
References for papers cited in our talks /
WWF solicitation based on false advertising?
Boxer vs.Armstrong Wager Gore vs. Armstrong
Prediction Market. Suggestions on what to
do? Analogies study WWF action steps
CO2 policy forecasting Climate Change
Forecasts are Useless for Policy Making
33
References for the Armstrong and Green
talks Papers available at http//publicpolicyfore
casting.com unless otherwise indicated
Armstrong, J. S., Green, K. C., Soon, W.
(2008). Polar Bear Population Forecasts A
Public-Policy Forecasting Audit. Interfaces, 38,
5, 382405. (includes commentary response).
Graefe, A., Armstrong, J. S., Green, K. C.
(2009). Using Prediction Markets to Solve Complex
Problems An Application to the Climate Bet.
Working paper available at http//kestencgreen.com
/cbpm.pdf Green, K.C., Armstrong, J. S.
(2007). Global Warming Forecasts by Scientists
versus Scientific Forecasts. Energy and
Environment, 18, No. 78, 995-1019. Green, K.C.,
Armstrong, J. S. (2007). Structured Analogies
in Forecasting. International Journal of
Forecasting, 23, 365-376. Green, K.C., Armstrong,
J. S., Soon, W. (2009). Validity of Climate
Change Forecasting for Public Policy Decision
Making. International Journal of Forecasting,
Forthcoming. Tetlock, P. E. (2005). Expert
Political Judgment How Good Is It? How Can We
Know? Princeton University Press, Princeton,
NJ. Gore-Armstrong prediction market at
http//tinyurl.com/gore-armstrong-bet Details
on Gore-Armstrong bet at http//theclimatebet.com