Riskbased Protocols

1
Integrated Environmental Modeling Model
Uncertainty and Management Decisions
IGOR LINKOV Cambridge Environmental Inc. 58
Charles Street Cambridge, MA 02141 Linkov_at_Cambri
dgeEnvironmental.com

US EPA CREM Meeting 30 June 2004
2
Current Decision-Making Processes
Decision-Maker(s)

• Include/Exclude?
• Detailed/Vague?
• Certain/Uncertain?
• Consensus/Fragmented?
• Iterative?
• Rigid/unstructured?

3
Evolving Decision-Making Processes
Decision-Maker(s)
Risk Assessment
Risk Analysis
Modeling / Monitoring
Stakeholders Opinion
Cost or Benefits
4
SUMMARY

• One of the greatest uncertainties in integrated
  modeling results from modelers interpretation of
  scenarios and approximations made by modelers.
  This source of uncertainty may be more
  significant than parameter and model
  uncertainties in practical applications.
• Linkov, I., Burmistrov, D (2003). Model
  Uncertainty and Choices Made by Modelers Lessons
  Learned from the International Atomic Energy
  Agency Model Intercomparisons. Risk Analysis 23
  1335-46.
• Current decision-making models typically offer
  little guidance on how to integrate or judge the
  relative importance of information resulting from
  modeling vs. other sources, such as
  socio-political and economic data. Multicriteria
  decision analysis (MCDA) not only provides
  better-supported techniques for the comparison of
  policy alternatives based on decision matrices
  but also has the added ability of being able to
  provide structured methods for the incorporation
  of stakeholders opinions into the ranking of
  alternative environmental policies.
• Linkov, I., Varghese, A., Jamil, S., Seager,
  T.P., Kiker, G., Bridges, T. (2004).
  Multi-Criteria Decision Analysis Framework for
  Applications in Remedial Planning For
  Contaminated Sites. in Linkov, I. And Ramadan,
  A. eds Comparative Risk Assessment and
  Environmental Decision Making Kluwer, 2004.

5
Overview

• Types of uncertainty
• Parameter
• Model
• Modeler
• Relative contribution of these sources of
  uncertainty IAEA BIOMASS Program
• Linking with Multi Criteria Decision Analysis
  Methods and Tools Army Corps of Engineers
  Projects

6
Model and Parameter Uncertainty and Variability

• Uncertainty Lack of knowledge about specific
  factors, parameters, or models
• parameter uncertainty (measurement error,
  sampling errors, systematic errors)
• model uncertainty (inaccurate model structure,
  model misuse)
• Variability Observed differences attributable to
  true heteorogeneity or diversity in a population
  or exposure parameter
• Examples body weight, biomass, root depth

7
Parameter Uncertainty
Lack of knowledge about specific factors,
parameters (measurement error, sampling errors,
systematic errors)
8
Model Uncertainty
Inaccurate model structure, model misuse
9
Modeler Uncertainty
subjective interpretation of the problem at hand
WHAT DO YOU SEE ?A HAT ORA BOA CONSTRICTOR
DIGESTING AN ELEPHANT
After Antoine Marie Roger de Saint-Exupéry
What is the relative influence of modeler
perception on model predictions?
10
Uncertainty Analysis Tools

• Parameter Uncertainty
• Monte-Carlo Simulation
• Analytical techniques
• Model Uncertainty
• Alternative Model Structures
• Weighting and combining models
• Expert Judgment Elicitation
• Modeler Uncertainty
• ????

11
International Atomic Energy Agency BIOMASS Model
Intercomparisons
ORGANIZATIONAL STRUCTURE

IAEA
IAEA
SCs MEMBERS
THEME 1

THEME 2

THEME 3

RADIOACTIVE
ENVIRONMENTAL RELEASES
BIOSPHERIC
PROCESSES
DISPOSAL
WASTE
STEERING
COMMITTEE
TECHNICAL
SECRETARY
TGROUPS
TGROUPS
TGROUPS
LEADERS
SECRETARY
PARTICIPANTS
12
Example Working Group ScenarioStrawberry
Contamination
Generic models (no Calibration)
Site-specific models (Calibrated)
13
Participating Models
14
Model Example FRUITPATH
Apple Tree
deposition wet, dry
tree removal
Berries
Organic
Layer
dissolution
root uptake
Labile Soil
Fixed Soil

adsorp-
tion /
desorption
leaching
Deep Soil
15
Modeling Approaches
16
Prediction Using Uncalibrated Generic Models
Based on Linkov and Burmistrov, 2003
Uncertainty--up to 7 orders of magnitude!
17
Prediction Using Partially-calibrated
Site-Specific Models
Based on Linkov and Burmistrov, 2001
Based on Linkov and Burmistrov, 2003
Uncertainty--1-2 orders of magnitude!
18
IAEA Forest and Fruit Working Group Scenarios
19
Modeler Uncertainty

• Differences in scenario interpretation results
  from heuristic procedures
• availability
• representativeness
• anchoring
• adjustment

Interpretation of the scenario by modelers can
result in model outcomes ranging over six orders
of magnitude.
20
Parameter Uncertainty

• Uncertainty and variability in model parameters
• data availability
• expert judgment
• empirical distributions

If models are properly calibrated and use similar
assumptions, parameter uncertainty can be much
less than modeler and model uncertainty.
21
Model Uncertainty

• Differences in model structure results from
• model objectives
• computational capabilities
• data availability
• knowledge and technical expertise of the group

If models are properly calibrated and use similar
assumptions, model uncertainty can be much less
than modeler uncertainty.
22
Conclusions

• Differences in model predictions may be quite
  high even for controlled experimental conditions
• Risk Characterization should be defined as
  analytic-deliberative process (NRC, 1996)
• Modeler Uncertainty should be addressed
• model calibration
• peer review and implementation of alternative
  models
• assigning uncertainty ranges

Solution Probabilistic modeling using Bayesian
calibration techniques.
23
Comparative Risk Assessment and Multi-Criteria
Decision Analysis A Framework For Managing
Contaminated Sediments

• Society for Risk Analysis Workshop
• 22-24 June 2004
• Co-Chairs Todd Bridges (US ACE) and Igor Linkov
  (Cambridge Environmental)

24
Why MCDA?

• Decision processes, while adequate in the past,
  are becoming more complicated/less effective.
• MCDA methods provide a means of integrating
  various inputs with stakeholder values
• MCDA methods provide a means of communicating
  model/monitoring outputs for scenario planning
  and stakeholder understanding

25
Review of Decision Analysis Applications

• Systematic review of decision analysis
  applications
• Academic articles - hundreds
• Real world examples a dozen
• Problem why are there so few examples?
• No systematic/adaptable framework found for
  structured and defendable decision making in
  EPA/DOE/USACE

26
MCDA Process (Yoe, 2002)
Problems
Alternatives
Criteria
Evaluation
Decision Matrix
Weights
Synthesis
Decision
27
Simple Decision Matrix
After Yoe (2002)
28
Realistic Decision Matrix
29
Essential Decision Ingredients
30
SUMMARY

• One of the greatest uncertainties in integrated
  modeling results from modelers interpretation of
  scenarios and approximations made by modelers.
  This source of uncertainty may be more
  significant than parameter and model
  uncertainties in practical applications.
• Linkov, I., Burmistrov, D (2003). Model
  Uncertainty and Choices Made by Modelers Lessons
  Learned from the International Atomic Energy
  Agency Model Intercomparisons. Risk Analysis 23
  1335-46.
• Current decision-making models typically offer
  little guidance on how to integrate or judge the
  relative importance of information resulting from
  modeling vs. other sources, such as
  socio-political and economic data. Multicriteria
  decision analysis (MCDA) not only provides
  better-supported techniques for the comparison of
  policy alternatives based on decision matrices
  but also has the added ability of being able to
  provide structured methods for the incorporation
  of stakeholders opinions into the ranking of
  alternative environmental policies.
• Linkov, I., Varghese, A., Jamil, S., Seager,
  T.P., Kiker, G., Bridges, T. (2004).
  Multi-Criteria Decision Analysis Framework for
  Applications in Remedial Planning For
  Contaminated Sites. in Linkov, I. And Ramadan,
  A. eds Comparative Risk Assessment and
  Environmental Decision Making Kluwer, 2004.