Can We Avoid Biases in Environmental Decision Analysis ?

1
Can We Avoid Biases in Environmental Decision
Analysis ?

• Raimo P. Hämäläinen
• Helsinki University of Technology
• Systems Analysis Laboratory
• raimo_at_hut.fi
• www.paijanne.hut.fi

2
Structure of the presentation

• Background decision analysis interviews
• Goals of the study
• Case Regulation of Lake Päijänne
• Splitting bias swapping of levels
• Description of the experiment
• Results of the experiment
• Conclusions ?

3
Environmental decision analysis

• Parliamentary nuclear power decision
• (Hämäläinen et. al)
• Decision analysis interviews
• (Marttunen Hämäläinen)
• Spontaneous decision conferencing in nuclear
  emergency management
• (Hämäläinen Sinkko)

4
Cognitive biases

• Splitting bias
• attribute receives more weight if it is split
• origins subjects give rank information only
• (Pöyhönen Hämäläinen)
• Not observable in hierarchical weighting

5
Decision analysis interviews

• Opinions of large groups of people traditionally
  collected through questionnaires
• Decision analysis interviews may provide a more
  reliable way to collect these opinions
• Idea
• one value tree for all common terminology
• emphasis on finding the viewpoints of different
  stakeholder groups
• interactive, computer supported

6
Research interest

• Existence of biases in a real case
• Can biases can be avoided through training and
  proper instructing ?
• Identify what can go wrong in the Lake Päijänne
  case
• Compare the well trained university students and
  spontaneous stakeholders responses

7
The Lake Päijänne case

• Regulation started 1964
• Main aims were to improve hydroelectricity
  production and to reduce damages caused by
  flooding
• Environmental values increase in free time
• need for an improved regulation policy

8
Splitting bias

• When an attribute is split, the weight it
  receives increases

0.4
0.1
0.4
0.3
0.3
0.3
0.3
0.3
9
Swapping of levels

• Does the order of the levels affect the resulting
  weights?
• Important question in environmental decision
  analysis
• stakeholder groups may vary regionally
• Not studied before

10
Example of swapping of levels
Attribute 1
Lake Päijänne
Attribute 1
Lake Päijänne
Attribute 2
River Kymijoki
Attribute 3
Lake Päijänne
Attribute 2
River Kymijoki
Attribute 1
Lake Päijänne
River Kymijoki
Attribute 2
Attribute 3
Attribute 3
River Kymijoki
11
Earlier experiments on biases

• Structure of the decision model affects the
  results
• Previous experiments typically
• subjects university students
• problems artificial
• results taken from group averages
• Lake Päijänne-case a real problem with real
  stakeholders

12
Important new features

• Realistic case
• Decision analysis interviews instead of passive
  decision support or survey
• Interactive computer support (resulting weights
  shown immediately)
• Instructions and training before the weighting

13
Subjects

• University students attending a course on
  decision analysis (N 30)
• held during a tutorial session, not mandatory
• Habitants of Asikkala (N 40)
• 3 groups of students
• 1 group of adults (volunteers)
• 3 experts from the Finnish Environment Institute
  2 summer residence owners

14
Experimental setting

• Weighting done with the SWING method using a
  tailored Excel interface
• Subjects entered the numbers themselves, two
  assistants were present to help
• Resulting weights shown as bars
• Order of value trees partly randomized

15
Sessions

• A short introduction to
• Lake Päijänne case
• value trees weighting
• different structures of the value tree
• In HUT the avoidance of biases was emphasized
  more
• Duration 60 - 90 minutes

16
SWING method

• Easy to use
• Attribute ranges clearly presented
• Idea
• choose the attribute you would first like to move
  to its best level
• assign it 100 points
• assign other attributes points less than 100 in
  respect to the first attribute

17
Flat-weighting
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Lahtien
Luonto
umpeenkasvu
Rantakasvillisuus
Vesivoima
Vesivoima
Tulvat, maatalous ja
Muu talous ???
teollisuus
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
18
Upper level weights
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Lahtien
Luonto
umpeenkasvu
Rantakasvillisuus
Vesivoima
Vesivoima
Tulvat, maatalous ja
Muu talous ???
teollisuus
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
19
ENV5-tree
Rantojen käytettävyys
Virkistys
Virkistyskalastus
Kalojen lisääntyminen
Ympäristö
Lahtien
umpeenkasvu
Luonto
Rantakasvillisuus
Talous
20
ENV2-tree
Virkistys
Ympäristö
Luonto
Talous
21
EC5-tree
Ympäristö
Vesivoima
Vesivoima
Tulvat, maatalous ja
Muu talous ???
teollisuus
Talous
Tulvat, loma-asutus
Muu talous
Vesiliikenne
Ammattikalastus
22
EC2-tree
Ympäristö
Vesivoima
Vesivoima
Muu talous ???
Muu talous ???
Talous
Talous
Muu talous
Muu talous
23
Swapping of levels
Päijänne
Tulvavahingot
Tulvavahingot
Päijänne
Muu talous ???
Muu talous ???
Kymijoki ja muut
Rantakasvillisuus
Päijänne
Tulvavahingot
Rantakasvillisuus
Kymijoki ja muut
Kymijoki ja muut
Rantakasvillisuus
24
Flat weights vs. upper level weights

• Both in group averages and in results of
  individuals the total weights for the environment
  and economy were similar with both methods
• One explanation symmetric value tree

25
Splitting bias
26
A typical resident in Asikkala
ENVIRONMENT
ECONOMY
5 1 5 2 1 1
5 1 1 1 5 2
27
Example from HUT(one of the best ones)
ENVIRONMENT
ECONOMY
5 1 5 2 1 1
5 1 1 1 5 2
28
Why even weights ?

• Some students none of the attributes seemed to
  be important
• Asikkala all of the attributes were important

even weights for all attributes
29
What caused the bias ?

• Similar points for
• all attributes in one branch
• regardless of the structure
• of the value tree

30
Effect of instructions

• Students had good instructions
• only some had bias in their results
• In the spontaneous stakeholders sessions the
  information load was too high and thus the
  instructions were not adopted as well
• nearly all had systematically consistent bias

31
Adjusted / not adjusted weights
STUDENTS
STAKEHOLDERS
32
Examples
STUDENTS
STAKEHOLDERS
33
Observation

• The students and the experts from FEI could
  nearly avoid the splitting bias
• good background education instructions did
  reduce the bias
• What did the students think? - Arithmetics or
  real avoidance of biases

34
Avoiding the splitting bias ?

• Good instruction can eliminate it
• When the economical attributes were split, the
  magnitude of the bias was slightly larger
• Graphical feedback did not eliminate
• Hierarchical weighting

35
Swapping of attribute levels
If the order of the levels would not affect the
weigts, the pairs of weights should be equal
(as in the first picture)
36
Conclusions about swapping of levels ?

• Only a few had clearly differing weights with the
  two trees
• No systematic pattern was found
• Less differences residents of Asikkala and
  students than with the splitting bias
• A simple scale lead to similar weights with both
  trees (100, 70 for example)
• Neither tree gained clear support

37
Solutions to reduce biases ?

• Hierarchical weighting
• Models should be tested on real decision makers
• Interactiveness of weighting ( possibility to
  return to change the points given earlier )
• Well balanced trees

38
Other observations in Asikkala

• Concept of weight seemed to be difficult for most
  subjects in Asikkala
• Information load was high
• Facilitators role becomes important when the DMs
  are uncertain

39
Problems related to the Lake Päijänne case

• Current regulation policy cannot be improved very
  significantly
• no big differences between the alternatives
• unrealistic hopes and false information are
  probably larger problems than the regulation
  itself
• money is not money
• strong feelings against the power companies and
  regulation (shape of value function ?)

40
Suggestions for future research

• Hierarchical weighting
• Encouragement to reconsider and readjust the
  statements iterate
• Decision Analyst must supervise!

41
References
R.P. Hämäläinen, E. Kettunen, M. Marttunen and H.
Ehtamo Evaluating a framework for
multi-stakeholder decision support in water
resources management, Group Decision and
Negotiation, 2001. (to appear) M. Pöyhönen,
Hans C.J. Vrolijk and R.P. Hämäläinen Behavioral
and procedural consequences of structural
variation in value trees. European Journal of
Operational Research, 2001. (to appear) M.
Pöyhönen and R.P. Hämäläinen There is hope in
attribute weighting, Journal of Information
Systems and Operational Research (INFOR), vol.
38, no. 3, Aug. 2000, pp. 272-282. Abstract
R.P. Hämäläinen, M. Lindstedt and K. Sinkko
Multi-attribute risk analysis in nuclear
emergency management, Risk Analysis, Vol. 20, No
4, 2000, pp. 455-467. M. Pöyhönen and R.P.
Hämäläinen Notes on the weighting biases in
value trees, Journal of Behavioral Decision
Making, Vol. 11, 1998, pp. 139-150. Susanna
Alaja Structuring effects in environmental
decision models, Helsinki University of
Technology, Systems Analysis Laboratory, Theses,
1998.
42
M. Pöyhönen, R.P. Hämäläinen and A. A. Salo An
experiment on the numerical modeling of verbal
ratio statements, Journal of Multi-Criteria
Decision Analysis, Vol. 6, 1997, pp. 1-10. R.P.
Hämäläinen and M. Pöyhönen On-line group
decision support by preference programming
in traffic planning, Group Decision and
Negotiation, Vol. 5, 1996, pp.485-50. M.
Marttunen and R.P. Hämäläinen Decision analysis
interviews in environmental impact assessment,
European Journal of Operational Research, Vol.
87, No. 3, 1995, pp. 551-563. R.P. Hämäläinen,
A.A. Salo and K. Pöysti Observations about
consensus seeking in a multiple criteria
environment, in Proceedings of the Twenty-Fifth
Hawaii International Conference on System
Sciences, Vol. IV, 1991, IEEE Computer Society
Press, Hawaii, pp. 190-198. R.P. Hämäläinen
Computer assisted energy policy analysis in the
parliament of Finland, Interfaces, Vol. 18, No.
4, 1988, pp. 12-23. Also in Case and Readings
in Management Science, 2nd edition, M. Render,
R.M. Stair Jr. and I. Greenberg (eds.), Allyn
Bacon, Massachusetts 1990 pp. 278-288.