AMBIGUITY

1
UNCERTAINTY
3 Bumper Stickers for Expert Judgment
AMBIGUITY
Roger Cooke Resources for the Future Dept. Math,
Delft Univ. of Technology April 15,16 2008
INDECISION
2
1. Expert Judgment is NOT Knowledge

• Scientific method NOT EJ methods - produces
  agreement among experts
• EJ is for quantifying ....not removing.....
  uncertainty.

3
2. Experts CAN quantify uncertainty as subjective
probability

4
Including 6700 calibration variables
5
Calibration questions for PM2.5RESS-PM25.pdf

• In London 2000, weekly average PM10 was 18.4
  µg/m3. What is the ratio
• non-accidental deaths in the week with the
  highest average PM10 concentration (33.4 µg/m3)
• ????????????????????????
• Weekly average non-accidental deaths.
• 5 _______ 25_______ 50 _______
  75________95________

6
3. Experts confidence does NOT predict performance

• Very informative assessors may be statistically
  least accurate
• PM25-Range-graphs.doc

7
4. Experts are sometimes well calibrated

• AMS-OPTION-TRADERS-RANGE-GRAPHS.doc
• realestate-range graphs.doc
• Sometimes not
• GL-invasive-species-range-graphs.doc

8
5. Experts sometimes agree

• Dispersion-USNRC-EU-RANGE-GRAPHS.doc
• And sometimes dont
• Campy-range-graphs.doc
• Earlyhealth-USNRC-EU-Range-graphs.doc

9
6. Classical model usually works, but not always

• All experts, and all combinations can be bad
• Soil-animal-USNRC-EU-range-graphs.doc

10
7. We CAN do better than equal weightingRESS-TUDd
atabase.pdf

11
(No Transcript)
12
Out-of-sample ValidationRESS_response2comments.pd
f

13 studies with ? 14 seed vbls, split, initialize
on one half, predict other half
13
8. Citation-based (social network) weights do NOT
perform betterRESS-SocNetLikelwgts.pdf
14
9. Experts like performance assessment

• Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al
  Geol Soc _.pdf , Aspinall Cooke PSAM4 3-9.pdf,
  SparksAspinall_VolcanicActivity.pdf

Separate scientific assessment of uncertainty
from decision making
15

• In the first few weeks of the Montserrat
  crisis there was perhaps, at times, some
  unwarranted scientific dogmatism about what might
  or might not happen at the volcano, especially in
  terms of it turning magmatic and explosive. The
  confounding effects of these diverging,
  categorical stances were then compounded for a
  short while by an overall diminution in
  communication between scientists and the various
  civil authorities. The result was a dip in the
  confidence of the authorities in the MVO team
  and, with it, some loss of public credibility
  this was not fully restored until later, when a
  consensual approach was achieved.
• Aspinall et al The Montserrat Volcano
  Observatory its evolution, organization, rôle
  and activities.

16
10. The choice is NOT whether to use EJbut
do it well or do it badly?NASA must always
strive for the highest level of accomplishment,
to exceed the expectations of the Nation, and to
do what is right...(Return to Flight task group
p.14)
17
Lets have another break