presentation to conference of the PorGrow Project,

1
Multi-Criteria Mapping performance,
uncertainty and framing of policy options

• presentation to conference of the PorGrow
  Project,
• Brussels, 14th September 2006

1 The Limits to Evidence-Based Policy
2 The Nature of Framing
3 The MCM Method
4 MCM Analysis and Results
2
Narrow Evidence-Based Approach

• Science-based assessment appears precise,
• but is sensitive to framing

risk externality
low risk high risk
3
Narrow Evidence-Based Approach

• Science-based assessment appears precise,
• but is sensitive to framing

0.001
0.01
0.1
1
10
100
1000
c/kWh
risk externality
low risk high risk
4
Narrow Evidence-Based Approach

• Science-based assessment appears precise,
• but is sensitive to framing

0.001
0.01
0.1
1
10
100
1000
c/kWh
ONSHORE WIND
PHOTOVOLTAICS
BIOMASS
LARGE SCALE HYDRO
NUCLEAR FISSION
NATURAL GAS
OIL
COAL
risk externality
low risk high risk
5
The Limits to Evidence Based Policy

• Science-based assessment appears precise,
• but is sensitive to framing

0.001
0.01
0.1
1
10
100
1000
c/kWh
ONSHORE WIND
PHOTOVOLTAICS
BIOMASS
LARGE SCALE HYDRO
NUCLEAR FISSION
NATURAL GAS
OIL
COAL
risk externality
low risk high risk
6
The Limits to Evidence Based Policy
knowledge about
knowledge about outcomes
probabilities
outcomes
not problematic
problematic
not problematic
RISK
AMBIGUITY
policy priorities
known demographics
UNCERTAINTY
problematic
IGNORANCE
future dynamics
surprises - unknown effects
Limits to sound science are practical,
methodological and theoretical Arrows Nobel
Prize shows unique evidence based policy is an
oxymoron!
7
The Nature of Framing
Some dimensions of framing in evidence based
policy
setting agendas defining problems
characterising options posing questions
prioritising issues formulating
criteria deciding context setting baselines
drawing boundaries discounting time
choosing methods including
disciplines handling uncertainties recruiting
expertise commissioning research constituti
ng proof exploring sensitivities
interpreting results
eg obesity defined in terms of public health
expenditure or consumer freedom priorities
attached to active adults, children,
disadvantaged groups

• All analysis requires framing all framing
  involves value judgements

sound science and expertise give different
answers to different questions we should be as
rigorous about validating the questions as the
answers
8
MCM what goes in
choose options
9
MCM what goes in
choose options
10
MCM what goes in
choose options
define criteria
11
MCM what goes in
choose options
define criteria
assess scores
12
MCM what goes in
choose options
define criteria
CRITERION A
option 1
option 2
option 3
option 4
performance
assess scores
explore uncertainty
13
MCM what goes in
choose options
define criteria
CRITERION A
option 1
option 2
option 3
option 4
performance
assess scores
explore uncertainty
14
MCM what goes in
choose options
define criteria
A
B
C
assess scores
explore uncertainty
15
MCM what goes in
choose options
define criteria
OVERALL RANKINGS
option 1
option 2
option 3
option 4
performance
assign weights
assess scores
explore uncertainty
16
MCM what goes in
choose options
define criteria
OVERALL RANKINGS
option 1
option 2
option 3
option 4
performance
assign weights
assess scores
explore uncertainty
17
MCM what goes in
choose options
define criteria
consider ranks
OVERALL RANKINGS
option 1
option 2
option 3
option 4
performance
assign weights
assess scores
explore uncertainty
18
MCM what goes in
choose options
define criteria
consider ranks
OVERALL RANKINGS
option 1
option 2
option 3
option 4
performance
assign weights
assess scores
explore uncertainty
19
MCM what comes out
maps of performance ranks of different
options under various perspectives Option
A Option B Option C Option D
academics
high performance low
performance as multi-criteria ranks
20
MCM what comes out
maps of performance ranks of different
options under various perspectives Option
A Option B Option C Option D
academics
industry
high performance low
performance as multi-criteria ranks
21
MCM what comes out
maps of performance ranks of different
options under various perspectives Option
A Option B Option C Option D Option A Option
B Option C Option D
academics
industry
government
high performance low
performance as multi-criteria ranks
22
MCM what comes out
maps of performance ranks of different
options under various perspectives Option
A Option B Option C Option D Option A Option
B Option C Option D
academics
industry
NGOs
government
high performance low
performance as multi-criteria ranks
23
MCM what comes out
detailed qualitative picture of conditions
associated with optimistic or pessimistic
performance
industry
Annotations and full transcripts entered and
processed in database
optimistic assumptions about technical operation
NGOs
pessimistic view of how option is likely to
perform in practice
24
MCM what comes out
rich body of background data concerning framing
of options by perspectives
25
MCM what comes out
rich body of background data concerning framing
of options by perspectives
uncertainties by option
A B C D
26
MCM what comes out
rich body of background data concerning framing
of options by perspectives
uncertainties by option
uncertainties by perspective
A B C D
academics industry government NGOs
27
MCM what comes out
rich body of background data concerning framing
of options by perspectives
uncertainties by option
uncertainties by perspective
A B C D
academics industry government NGOs
scores for particular issues
A B C D
28
MCM what comes out
rich body of background data concerning framing
of options by perspectives
uncertainties by option
uncertainties by perspective
A B C D
academics industry government NGOs
weights by issue for perspectives
scores for particular issues
economics health environment equity
academics industry government NGOs
A B C D
29
Key Features of MCM Analysis
A practical focus on performance of key policy
alternatives
eg ranks of different options for addressing
obesity
Combines quantitative clarity with qualitative
subtlety
eg different criteria, weights, pessimistic
and optimistic scores and comments
Shows transparently how results depend on
framing
eg graphical maps of ranks, weights,
uncertainties under different perspectives
Does not guarantee identification of single
best policy option
but produces socially robust picture of
relatively high and low performance
Outcome depends on evidence and perspectives that
are included
but this is true of any analysis, and MCM
provides high degree of auditability