Rank-Based%20Sensitivity%20Analysis%20of%20Multiattribute%20Value%20Models

1
Rank-Based Sensitivity Analysis of Multiattribute
Value Models

• Antti Punkka and Ahti Salo
• Systems Analysis Laboratory
• Helsinki University of Technology
• P.O. Box 1100, 02015 TKK, Finland
• http//www.sal.tkk.fi/
• forename.surname_at_tkk.fi

2
Additive Multiattribute Value Model

• Provides a complete rank-ordering for the
  alternatives
• Selection of the best alternative
• Rank-ordering of e.g. universities (Liu and Cheng
  2005) or graduate programs (Keeney et al. 2006)
• Prioritization of project proposals or innovation
  ideas (e.g. Könnölä et al. 2007)
• Methods for global sensitivity analysis on
  weights and scores
• Focus only on the selection of the best
  alternative
• Ex post Sensitivity of the decision
  recommendation to parameter variation
• Ex ante Computation of viable decision
  candidates subject to incomplete information
  about the parameter values
• (e.g., Rios Insua and French 1991, Butler et al.
  1997, Mustajoki et al. 2006)

3
Sensitivity Analysis of Rankings

• Consider the full rank-ordering instead of the
  most preferred alternative
• How sensitive is the rank-ordering
• How to compare two rank-orderings? How to
  communicate differences?
• We compute the attainable rankings for each
  alternative subject to global variation in
  weights and scores
• How sensitive is the ranking of an alternative
  subject to parameter variation?
• Is the ranking of university X sensitive to the
  attribute weights applied?
• What is the best / worst attainable ranking of
  project proposal Y?

4
Incomplete Information

• Model parameter uncertainty before computation
• Relax complete specification of parameters
• Error coefficients on the statements, e.g.
  weight ratios
• E.g. Mustajoki et al. (2006)
• Directly elicit and apply incomplete information
• Incompletely defined weight ratios 2 w3/ w2
  3
• Ordinal information about weights w1 w3
• Score intervals 0.4 v1(x12) 0.6
• E.g., Kirkwood and Sarin (1985),
• Salo and Hämäläinen (1992), Liesiö et al. (2007)
• Set of feasible weights and scores (S)

5
Attainable Rankings

• Existing output concepts of ex ante sensitivity
  analysis do not consider the full rank-ordering
  of alternative set X
• Value intervals focus on 1 alternative at a time
• Dominance relations are essentially pairwise
  comparisons
• Potential optimality focuses on the ranking 1
• Alternative xk can attain ranking r, if exists
  feasible parameters such that the number of
  alternatives with higher value is r-1

6
Attainable Rankings Example

• 2 attributes, 4 alternatives with fixed scores,
  w1?? 0.4, 0.7

V
x1
x2
x3
x4
0.4
0.7
0.6
0.3
7
Computation of Attainable Rankings

• Application of incomplete information ? set of
  feasible weights and scores (S)
• If S is convex, all rankings between the best and
  the worst attainable rankings are attainable
• Best ranking of xk
• Worst ranking of xk
• MILP model to obtain the best / worst ranking of
  each xk
• V(x) expressed in non-normalized form (linear in
  w and v)
• of binary variables X - 1

8
Example Shangai Rank-Ordering of Universities

• Shanghai Jiao Tong University ranks the world
  universities annually
• Example data from 2007
• http//ed.sjtu.edu.cn/ranking2007.htm
• 508 universities
• Additive model for rank-ordering of the
  universities

9
Attributes
Criterion Indicator Code Weight
Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Alumni 10
Quality of Faculty Staff of an institution winning Nobel Prizes and Fields Medals Award 20
Quality of Faculty Highly cited researchers in 21 broad subject categories HiCi 20
Research Output Articles published in Nature and Science NS 20
Research Output Articles in Science Citation Index-expanded, Social Science Citation Index SCI 20
Size of Institution Academic performance with respect to the size of an institution Size 10
Table adopted from http//ed.sjtu.edu.cn/ranking20
07.htm
10
Data
11
Sensitivity Analysis

• How sensitive are the rankings to weight
  variation?
• What if different weights were applied?
• Relax point estimate weighting
• 1. Relative intervals around the point estimates
• E.g. ?20 , wi0.20
• 2. Incomplete ordinal information
• Attributes with wi0.20 cannot be less important
  than those with wi0.10
• All weights lower-bounded by 0.02

12
Results Rank-Sensitivity of Top Universities
exact weights
20 interval
30 interval
University
incompl. ordinal
no information
10th
442nd
Ranking
13
Conclusion

• A model to compute attainable rankings
• Sufficiently efficient even with hundreds of
  alternatives and several attributes
• Attainable rankings communicate sensitivity of
  rank-orderings
• Conceptually easy to understand
• Holistic view of global sensitivity at a glance
  independently of the of attributes
• Applicable output in Preference Programming
  framework
• Additional information leads to fewer attainable
  rankings
• Connections to project prioritization
• Initial screening of project proposals for e.g.
  portfolio-level analysis
• Supports identification of clear decisions (cf.
  Liesiö et al. 2007)
• Select the ones surely in top 50
• Discard the ones surely outside top 50

14
References

• Butler, J., Jia, J., Dyer, J. (1997). Simulation
  Techniques for the Sensitivity Analysis of
  Multi-Criteria Decision Models. EJOR 103,
  531-546.
• Keeney, R.L., See, K.E., von Winterfeldt, D.
  (2006). Evaluating Academic Programs With
  Applications to U.S. Graduate Decision Science
  Programs. Oper. Res. 54, 813-828.
• Kirkwood, G., Sarin R. (1985). Ranking with
  Partial Information A Method and an Application.
  Oper. Res. 33, 38-48
• Könnölä, T., Brummer, V., Salo A. (2007).
  Diversity in Foresight Insights from the
  Fostering of Innovation Ideas. Technologial
  Forecasting Social Change 74, 608-626.
• Liesiö, J., Mild, P., Salo, A., (2007).
  Preference Programming for Robust Portfolio
  Modeling and Project Selection. EJOR 181,
  1488-1505.
• Liu, N.C., Cheng, Y. (2005). The Academic Ranking
  of World Universities. Higher Education in Europe
  30, 127-136
• Mustajoki, J., Hämäläinen, R.P., Lindstedt,
  M.R.K. (2006). Using intervals for Global
  Sensitivity and Worst Case Analyses in
  Multiattribute Value Trees. EJOR 174, 278-292.
• Rios Insua, D., French, S. (1991). A Framework
  for Sensitivity Analysis in Discrete
  Multi-Objective Decision-Making. EJOR 54,
  176-190.
• Salo, A., Hämäläinen R.P. (1992). Preference
  assessment by imprecise ratio statements. Oper.
  Res. 40, 1053-1061.