Elicitation in Decision Making

1
Elicitation in Decision Making

• Kenneth C. Lichtendahl, Jr.
• Darden School of Business
• University of Virginia
• Robert L. Winkler
• Fuqua School of Business
• Duke University
• SAMSI Risk Perception, Policy, Practice
  Workshop October 4, 2007

2
Overview

• Eliciting probabilities from multiple forecasters
• Scoring rules to encourage truthful reporting
• Reconciling the reports of truthful forecasters
• Competition among forecasters
• Equilibrium reporting strategies
• Reconciling the reports of competitive
  forecasters
• Implications for calibration
• Fostering cooperation through business-sharing
• Summary

3
Probabilities for Event E from Two Forecasters

• A Scoring Rule (quadratic)
• To maximize expected score, a forecaster should
  report truthfully ß p, where p is the
  forecasters probability for E
• A Common Prior for DM and forecasters

4
Reconciling Truthful Reports

• Under this model, a decision maker who consults
  only forecaster i should have P(E ßi) ßi
• If the decision maker consults both forecasters,

5
(No Transcript)
6
Competition Among Forecasters

• A forecasters Utility U(R,S) wR (1-w)S,
  where S is the score and the rank R 1(a)0 if the
  forecasters score is gt()lt the other
  forecasters score.
• An equilibrium reporting strategy

7
(No Transcript)
8
Responding to a Report from a Competitive
Forecaster

• A decision maker who consults only forecaster i
  should have
• The decision maker no longer takes the
  forecasters report at face value

9
(No Transcript)
10
Reconciling Reports from Competitive Forecasters

• The decision makers P(E ß1, ß2) changes also,
  as might be expected.

11
(No Transcript)
12
Implications for Calibration

• Well-calibrated competitive forecasters give
  reports that are miscalibrated and overconfident
• Properly reconciled reports from well-calibrated
  competitive forecasters are well calibrated
• Improperly reconciled reports from
  well-calibrated competitive forecasters (i.e.,
  reports that are assumed to be truthful) are
  miscalibrated and overconfident

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Consulting Competitive Forecasters

• The truthful-reporting incentive of proper
  scoring rules may no longer hold
• Forecasters reports may be extreme, exaggerating
  toward zero or one
• Observed overconfidence need not reflect a
  cognitive bias, but could be perfectly rational
  for a competitive forecaster
• Taking competitive forecasters reports at face
  value and reconciling them as if they were
  truthful can lead a decision maker to be
  overconfident

17
Inducing Cooperative Behavior

• Scoring rules may already be endogeneous,
  stemming from the forecasters realization that
  good outcomes for the decision maker can be good
  outcomes for the forecasters
• To extenuate competitive stakes, a decision maker
  can emphasize the business-sharing game the
  decision maker and forecasters are involved in
• Suppose a decision maker will choose between two
  actions a1 and a2, the forecasters will report
  probabilities for XE before this choice, and all
  will share in the consequences obtaining utility
  u(ai,X)

18
Inducing Cooperative Behavior Joint Scoring Rules

• In a business-sharing game, its an equilibrium
  for forecasters to truthfully report
  probabilities and for the decision maker to take
  an action based on reconciling these reported
  probabilities
• The equilibriums payoffs can be implemented as a
  joint scoring rule, a set of interdependent
  proper scoring rules, one for each forecaster
• Common prior beliefs about the forecasters
  probabilities dictate the form of the joint
  scoring rule.
• The quadratic score is in this class of scoring
  rules

19
Some Generalizations and Further Thoughts

• Similar results hold for different scoring rules,
  different common priors, asymmetric forecasters,
  more than two forecasters, and a non-diffuse
  decision maker.
• The common prior is an unrealistic device, but it
  gives us an idea of what can happen if
  forecasters have some idea of other forecasters
  abilities.
• Different utility functions can, of course, lead
  to different results (e.g., nonlinear utility for
  the score itself or trying to avoid last place
  instead of trying to get first place).
• It is important for a decision maker to be aware
  of such competitive aspects and perhaps to try to
  take steps to minimize them.
• This is another example of how other stakes can
  create difficulties when we try to interpret
  reports from experts.