MCDA in Edemocracy: Why Weight Comparing Even Swaps and MAVT

1
MCDA in E-democracy Why Weight? Comparing Even
Swaps and MAVT

• Valerie Belton, University of Strathclyde
• George Wright, Durham University
• Gilberto Montibeller, Kingston University

TED Workshop, Helsinki, May 2005
2
Outline

• Rationale for the talk and motivation for our
  studies
• A brief introduction to the Even Swaps approach
• Overview of the studies
• Findings
• Lessons for E-Democracy?

3
Rationale

• Our extensive experience with the use of MAVT in
  practice and in teaching the approach to
  practicing managers continues to reveal issues in
  understanding criteria weights
• In face to face situations a facilitator can be
  alert to such difficulties and try to alleviate
  them, but this is not the case in the context of
  e-democracy
• The consequence may be that potential
  participants fail to engage, or that meaningless
  information is provided

4
The Holy Grail of MCDA?

• An interpretation of the notion of importance
  or weight which is both psychologically
  meaningful and operationally well defined?
• Or .
• A theoretically well founded and practically
  usable approach which does not depend on the
  specification of weights

Could Even Swaps be this approach?
5
The Even Swaps Approach

• Starting point A Consequences Table
• Conversion to a Ranking Table (not essential)
• Progressive simplification of the problem by
• Use of dominance or practical (near) dominance to
  eliminate alternatives
• Use of swaps to equalise performances on a
  selected criterion allowing the elimination of
  that criterion
• Until only one option remains

6
Even Swaps Example
7
Step 1 Practical Dominance
Chris practically dominates Angela
8
Step 1 Practical Dominance
David practically dominates Freda
Chris practically dominates Barry
9
Step 2 1st Swap

• Swap 1 Web Experience vs Marketing
  Experience
• Equalise scores on Web Experience
• Compensate on Marketing Experience

Eliminate Web Experience
10
Multi-Attribute Value Analysis using V.I.S.A
V(a) ?i 1 to N wi v i(a)
11
Even Swaps

• Defining an Even Swap in MAVT terms
• Despite the apparent complexity of this judgement
  it is one that intuitively seems to be
  psychologically meaningful.

• wi (v i(a)new - v i(a)old )
• wk (v k(a)old - v k(a)new)

12
Our initial thoughts about Even Swaps

• Conceptually attractive
• Doesnt require specification of weights
• Based on easy to understand principles
• Progressively simplifies the problem
• Concerns
• Focused on micro judgements loses holistic
  perspective
• Once swapping starts no longer comparing real
  options
• Curiosity
• Lack of attention in the decision making
  literature
• How do real managers / decision makers react?

13
Context for the studies

• Involved full-time and part-time MBA students
  taking the 6 credit (of 180) core class, Making
  Decisions
• Week 1 of a 5 week / 15 hour class
• How do people make decisions (1.5 hours)
• Introduce MAVT in class using worked example (1.5
  hours)
• Week 2
• MAVT in practice (process and cases)
• Start to use MAVT for group project
• Week 3
• Introduced to Even Swaps in class using worked
  example
• Students work through personnel selection
  exercise using paper-based pro-forma

14
Research methodology

• Study 1 quantitative orientation
• We looked at
• Performance on the Even Swaps exercise
• Responses to questionnaires comparing MAVT and
  ES, distributed immediately after the ES exercise
• Study 2 - more qualitative orientation
• Class exercise was seen as an opportunity to
  practice ES
• Individual assignment
• Students to re-analyse their group decision task
  using ES
• to Compare and contrast the MAVT / ES
  analyses, commenting on the relative strengths
  and weaknesses for individual and group decision
  making
• Comparative evaluations were analysed and
  categorised by two researchers to identify
  recurring themes
• Performance on the Even Swaps exercise was
  assessed recurring mistakes identified

15
Measure of performance

• 1st Study
• Students did not perform well in the initial Even
  Swaps exercise
• Average score 2.4 (on 1-5 scale), Average of
  error- free swaps 30
• 2nd and 3rd studies
• Better performance, but still approx 30 showing
  poor understanding

16
Nature of errors variations on the method

• Errors / shortcomings
• Misunderstanding / incorrect application of
  practical dominance
• Practical dominance not explained
• Use of Ranking Table for Swaps
• Swaps in the wrong direction
• Swaps not carried through to next step
• Didnt equalise scores on base criterion
• Swapped across alternatives or other incorrect
  swaps
• Assume order of elimination relates to overall
  rank (inverse)
• Variations
• Swapped against more than 1 criterion
• Focus swaps on 1 alternative (to create
  dominance)
• Defined weighted value table to equalise units

17
Study 1 Questionnaire

• Please tick the box that best represents your
  opinion of the strengths of each of the two
  techniques for
• Improving your decision making
• 17 questions

18
Results Study 1 - Questionnaire

• MAVT viewed as stronger than Even Swaps with
  respect to
• improving decision making
• quantifying decision making
•  providing insights into decision making
• justifying decisions to others
• challenging initial intuitive decisions
• documenting how a decision was made
• reconciling qualitative and quantitative aspects
  of a decision
• making decisions involving many attributes
• making decisions involving many alternatives
• trading-off costs against benefits
• enabling sensitivity analysis
• integrating objective measurement with value
  judgment.
• All results significant at 1 using the Wilcoxon
  Z-test

19
Results Study 1 Questionnaire - continued

• No significant difference was perceived between
  the two approaches with respect to
• making decisions involving few attributes
• making decisions involving few alternatives
• making tradeoffs.
• In addition there was no significant difference
  in the students rating of the perceived ease /
  difficulty of the two approaches, or of their own
  understanding

20
Studies 2 3 Process of Analysis

• Coding of 111 / 74 commentaries by two
  independent researchers
• The strength of the Even Swaps approach is that
  it is simple to use and can be carried out
  manually. However, it may not be suitable for
  complex decision problems. .. The MAVT approach
  provides structure for analysing complex problems
  in a systematic way that can be easily
  back-tracked, providing an audit trail,
  transparency and integrity
• Recorded in a spreadsheet
• For each cohort the independent analyses were
  compared and reconciled incorporating a process
  of cross-checking and merging concepts
• Views of competent students retained
• Finally, concepts matched across studies (56
  concepts cited by 10)

21
Qualitative Analysis Top Twelve Comments
22
Summary Remarks

• Although conceptually simple, the Even Swaps
  approach is not that easy to pick up only 33
  of students made a reasonable attempt in the 1st
  study, increasing to 70 and 79, with the
  opportunity to practice and reflect, in the 2nd
  and 3rd studies
• In the initial evaluation MAVT / V.I.S.A was
  rated equally or better with respect to all
  questions posed
• In the 2nd and 3rd studies the students perceived
  the Even Swaps approach as being easy to use,
  well suited to simple/small scale (in terms of
  number of options and criteria) individual
  decisions, but highly subjective
• They saw the MAVT/V.I.S.A approach as being
  better suited for complex decisions and for group
  working, being visual, permitting sensitivity
  analysis, facilitating shared understanding and
  generating ownership / consensus.
• Both approaches were seen as simple /
  straightforward to use
• MAVT seen to provide an audit trail views on
  Even Swaps divided

23
What can we learn for E-Democracy?

• ????
• Initial thoughts MAVT better suited for working
  with groups . therefore better suited to ED
• BUT . what is ED? (where does DGDSS stop and ED
  start?)
• Is ED about sharing, comparing or aggregating?

24
A student assignment

• Im tired and ready for bed
• All this swapping has messed up my head
• I should have been prudent
• And been a SMART student
• I now want that Hammond guy dead

25
Most Frequent Qualitative Comments (10)
26
Most Frequent Qualitative Comments (10)
27
Most Frequent Qualitative Comments (10)
28
Most Frequent Qualitative Comments (10)
29
Most Frequent Qualitative Comments (10)
30
Most Frequent Qualitative Comments (10)
31
Most Frequent Qualitative Comments (10)