Trick or treatment? Evaluating the quality of structured risk management decisions

1
Trick or treatment?Evaluating the quality of
structured risk management decisions

• Joe Arvai
• The Ohio State University
• Decision Research

2
Outline

• Structured DM in brief
• A practical example
• Questions of quality (3 experiments)
• Parting thoughts

3
SDM In Brief

• Structured decision approaches have their roots
  in
• the observation that people tend to have
  tremendous difficulty with making decisions that
  involve multiple objectives and, therefore,
  tradeoffs.

4
SDM In Brief

• Structured decision approaches have their roots
  in
• studies of the constructive nature of
  preferences in response to available cues.

5
SDM In Brief

• Structured decision approaches are designed based
  on
• value focused thinking.

6
SDM In Brief

• Structured decision approaches are designed based
  on
• the literature dealing with normative decision
  making, specifically the steps required for a
  complete analysis of a given decision

7
SDM In Brief

• Structured decision approaches are designed based
  on
• methods for decreasing the cognitive burden
  associated with complex choices.

8
The Case of Water Use Planningin British Columbia

• Work with B.C. Hydro on a comprehensive,
  stakeholder-based development of revised
  operating plans at all major hydroelectric
  facilities.
• In response to increasing, competing demands on
  provincial water resources

9
Scale of Study
10
The Case of Water Use Planningin British Columbia

• Multiple Objectives
• Electricity generation/trade
• Environmental quality
• Water, land, air
• Recreation opportunities
• Cultural values
• Learn over time reduce uncertainty

http//www.bchydro.com/wup/
11
Water Use Planning in B.C.The Basic Steps

• The approach is based on work with stakeholders
  on
• Eliciting objectives from various stakeholders.
• Identifying a series of workable options for
  management.
• Establishing attributes of/measures for each
  objective.
• Generating a matrix across these objectives and
  options and addressing the tradeoffs that
  selecting one option over another entails.

12
Water Use Planning in B.C.Participants

• BC Hydro (Crown Corporation)
• Federal Government
• Provincial Government
• Local Government
• First Nations
• Community Stakeholders
• Home owners, business operators, etc.

13
Water Use Planning in B.C.Objectives,
Attributes, Measures
Objectives Attributes
Recreation e.g., Weighted User Days
Erosion e.g., Weighted Erosion Days
Flooding e.g., Weighted Flood Days (flow level)
Fish e.g., Available Habitat, IBI
Water Supply e.g., Water Quality Impact Rating
Cultural Heritage e.g., Consistency Rating
Financial e.g., Annual Revenues M / Year
14
Water Use Planning in B.C.e.g., Financial
Objectives Attributes
15
Water Use Planning in B.C.e.g., Recreation
Objectives Attributes
16
Water Use Planning in B.C.Assessing Value
Objectives Attributes Mimic Natural Hydrograph Enhanced Summer Releases Enhanced Winter Releases
Environment Conserve Salmon Viable Spawning Habitat 50 20 25
Economic Revenue Generation NPV () 60 Million 80 Million 65 Million
Social Recreation Opportunities Number of User-Days 1400 1200 1500
17
Water Use Planning in B.C.Risk Uncertainty
Objectives Attributes Mimic Natural Hydrograph Enhanced Summer Releases Enhanced Winter Releases
Environment Conserve Salmon Viable Spawning Habitat 50 20 30
Economic Revenue Generation NPV () 60 M 80 M 65 M
Social Recreation Opportunities of user days 1400 1200 1500
18
Water Use Planning in B.C.
19
Water Use Planning in B.C.
20
A High Quality Process?

• Government Support
• Approach originally conceived as a pilot project
  at a single site
• Now implemented at 23 sites province wide (18
  Water Use Plans completed to date)

• Cost-effective
• Original budget to complete all 23 plans 25
  Million
• Revised budget Approx. 14 Million

• Participant Satisfaction
• Historically adversarial groups work together
• BC Hydro has a clear mandate
• Consensus plans often achieved

21
Experiment 1Quantitative Measures of Quality

• Designed to compare two approaches for involving
  stakeholders in water use planning in B.C
• small groups (7-10)
• both conditions provided with the same
  information
• Structured vs. Unstructured process

22
Experiment 1Quantitative Measures of Quality

• UNSTRUCTURED
• 1. Self-rating questions
• 2. Technical Information
• newspaper article
• informative booklet
• audio-documentary
• 3. Group Discussion
• 4. Evaluation of policy alternatives
• referendum
• willingness to pay
• 5. Self-rating questions

• STRUCTURED
• 1. Self-rating questions
• 2. Technical Information
• newspaper article
• informative booklet
• audio-documentary
• 3. Decision structuring
• characterize values objectives
• group discussion about values and objectives.
• objectives ranking/tradeoffs
• 4. Evaluation of policy alternatives
• referendum
• willingness to pay
• 5. Self-rating questions

23
Experiment 1Results
24
Experiment 1Results
F
25
Experiment 1Conclusions

• Based on participants self-ratings and an
  analysis of deliberation periods, we conclude
  that the the structured approach leads to higher
  quality decisions.
• Arvai, J. L., R. Gregory, and T. McDaniels. 2001.
  Testing a structured decision approach
  Value-focused thinking for deliberative risk
  communication. Risk Analysis, 21 1065-1076.

26
Experiment 2Analyzing Choices

• Decision structuring to alleviate embedding in
  environmental valuation
• when a good is assigned a higher value on its
  own vs. when its part of a more inclusive set.
• e.g., 51 differences in WTP for a single vs. a
  set of disaster preparedness services

27
Experiment 2Analyzing Choices

• Providing much-needed structure may help to
  overcome the embedding problem by helping
  decision makers to think about the components of
  a valuation problem.

28
Experiment 2Analyzing Choices

• VERSION A
• 1. Technical Information
• newspaper article
• informative booklet
• audio-documentary
• 2. Decision structuring
• characterize values objectives
• group discussion about values and objectives.
• objectives ranking/tradeoffs
• 3. WTP 1 Rivers
• 4. WTP 10 River

• VERSION B
• 1. Technical Information
• newspaper article
• informative booklet
• audio-documentary
• 2. Decision structuring
• characterize values objectives
• group discussion about values and objectives.
• objectives ranking/tradeoffs
• 3. WTP 10 Rivers
• 4. WTP 1 River

29
Experiment 2Analyzing Choices
CHOICE 1 CHOICE 2
VERSION A WTP 1 River W WTP 10 Rivers X
VERSION B WTP 10 Rivers Z WTP 1 River Y
No Embedding W/X Y/Z
0.42 ? 0.25
W gt Y
ZltX
30
Experiment 2Conclusions

• Based on participants mean WTP judgments,
  embedding was not alleviated (according to the
  ratio standard).
• McDaniels, T., R. Gregory, J. L. Arvai, and R.
  Chuenpagdee. 2003. Decision structuring as a
  means of alleviating embedding in environmental
  valuation. Ecological Economics, 44 33-46.

31
Experiment 32-Part Evaluation

• An experiment that would measure both
  self-ratings of quality and subjects choices

32
Experiment 32-Part Evaluation

• UNSTRUCTURED
• 1. Self-rating questions
• 2. Technical Information
• 3. Choice Task
• Funding allocations across three risk problems
• 4. Self-rating questions

• STRUCTURED
• 1. Self-rating questions
• 2. Technical Information
• 3. Decision Structuring
• Objectives ranking, tradeoffs
• Linking objectives with management problems
• 4. Choice Task
• Funding allocations across three risk problems
• 5. Self-rating questions

33
Experiment 3Three Risk Problems
Problem Affect Rating Corresponding Objective
Wildlife Disease Affect Rich Human Health
Deer Overpopulation Affect Neutral Environmental Health
Damaged Trails Affect Neutral Recreation
34
Experiment 3Self-Rating Results
35
Experiment 3Ranked Objectives
OBJECTIVE RANK
Environmental Health 1
Human Health 2
Recreation 3
36
Experiment 3Funding Allocations
37
Experiment 3Conclusions

• Appears to be a disconnect between self-ratings
  of quality and actual funding choices
• Wilson, R.S. and J. L. Arvai. 2004. Evaluating
  the quality of structured risk management
  decisions. In Review.

38
Parting Thoughts

• Level of facilitation, time for deliberation,
  increased attention to tradeoffs, etc. all seem
  to be critical.

39
Parting Thoughts

• Anecdotal observations or evaluations based on
  self-reports alone are likely insufficient for
  evaluating the quality of structured decision
  approaches.

40
Parting Thoughts

• Affective responses to stimuli exert powerful
  influences on risk judgments

41
Parting Thoughts

• Not suggesting that the outcomes of all
  structured decision making approaches are suspect.

42
Thanks

• Tim McDaniels
• Robin Gregory
• Ying Chuenpagdee
• Robyn Wilson
• Louie Rivers
• Dan Ohlsen
• SSHRC
• NSF
• OARDC
• EPI