1. 1 Helsinki University of Technology Systems Analysis Laboratory Rank-Based Sensitivity Analysis of Multiattribute Value Models Antti Punkka and Ahti Salo Systems Analysis Laboratory Helsinki University of Technology P.O. Box 1100, 02015 TKK, Finland http://www.sal.tkk.fi/ forename.surname@tkk.fi

2. Helsinki University of Technology Systems Analysis Laboratory 2 INFORMS Annual Meeting, Washington DC 2008 Additive Multiattribute Value Model n Provides a complete rank-ordering for the alternatives –Selection of the best alternative –Rank-ordering of e.g. universities (Liu and Cheng 2005) or graduate programs (Keeney et al. 2006) –Prioritization of project proposals or innovation ideas (e.g. Könnölä et al. 2007) n Methods for global sensitivity analysis on weights and scores –Focus only on the selection of the best alternative 1.Ex post: Sensitivity of the decision recommendation to parameter variation 2.Ex ante: Computation of viable decision candidates subject to incomplete information about the parameter values (e.g., Rios Insua and French 1991, Butler et al. 1997, Mustajoki et al. 2006)

3. Helsinki University of Technology Systems Analysis Laboratory 3 INFORMS Annual Meeting, Washington DC 2008 Sensitivity Analysis of Rankings n Consider the full rank-ordering instead of the most preferred alternative –How ’sensitive’ is the rank-ordering –How to compare two rank-orderings? How to communicate differences? n We compute the attainable rankings for each alternative subject to global variation in weights and scores –How sensitive is the ranking of an alternative subject to parameter variation? –Is the ranking of university X sensitive to the attribute weights applied? –What is the best / worst attainable ranking of project proposal Y?

4. Helsinki University of Technology Systems Analysis Laboratory 4 INFORMS Annual Meeting, Washington DC 2008 Incomplete Information n Model parameter uncertainty before computation 1.Relax complete specification of parameters »”Error coefficients” on the statements, e.g. weight ratios »E.g. Mustajoki et al. (2006) 2.Directly elicit and apply incomplete information »Incompletely defined weight ratios: 2 ≤ w 3 / w 2 ≤ 3 »Ordinal information about weights: w 1 ≤ w 3 »Score intervals: 0.4 ≤ v 1 (x 1 2 ) ≤ 0.6 »E.g., Kirkwood and Sarin (1985), Salo and Hämäläinen (1992), Liesiö et al. (2007)  Set of feasible weights and scores (S)

5. Helsinki University of Technology Systems Analysis Laboratory 5 INFORMS Annual Meeting, Washington DC 2008 Attainable Rankings n Existing output concepts of ex ante sensitivity analysis do not consider the full rank-ordering of alternative set X –Value intervals focus on 1 alternative at a time –Dominance relations are essentially pairwise comparisons –Potential optimality focuses on the ranking 1 n Alternative x k can attain ranking r, if exists feasible parameters such that the number of alternatives with higher value is r-1

6. Helsinki University of Technology Systems Analysis Laboratory 6 INFORMS Annual Meeting, Washington DC 2008 Attainable Rankings: Example n 2 attributes, 4 alternatives with fixed scores, w 1  [0.4, 0.7] V w1w1 0.40.7 w2w2 0.60.3 ranking 1 is attainable for x 2 ranking 3 is attainable for x 3 ranking 1 is attainable for x 3 ranking 4 is attainable for x 1 x1x1 x2x2 x3x3 x4x4 Attainable rankings

7. Helsinki University of Technology Systems Analysis Laboratory 7 INFORMS Annual Meeting, Washington DC 2008 Computation of Attainable Rankings n Application of incomplete information  set of feasible weights and scores (S) n If S is convex, all rankings between the best and the worst attainable rankings are attainable –Best ranking of x k : –Worst ranking of x k : n MILP model to obtain the best / worst ranking of each x k –V(x) expressed in non-normalized form (linear in w and v) –# of binary variables = |X| - 1

8. Helsinki University of Technology Systems Analysis Laboratory 8 INFORMS Annual Meeting, Washington DC 2008 Example: Shangai Rank-Ordering of Universities n Shanghai Jiao Tong University ranks the world universities annually n Example data from 2007 –http://ed.sjtu.edu.cn/ranking2007.htm –508 universities n Additive model for rank-ordering of the universities

9. Helsinki University of Technology Systems Analysis Laboratory 9 INFORMS Annual Meeting, Washington DC 2008 Attributes CriterionIndicatorCodeWeight Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals Alumni10 % Quality of Faculty Staff of an institution winning Nobel Prizes and Fields Medals Award20 % Highly cited researchers in 21 broad subject categories HiCi20 % Research Output Articles published in Nature and ScienceN&S20 % Articles in Science Citation Index-expanded, Social Science Citation Index SCI20 % Size of Institution Academic performance with respect to the size of an institution Size10 % Table adopted from http://ed.sjtu.edu.cn/ranking2007.htm

10. Helsinki University of Technology Systems Analysis Laboratory 10 INFORMS Annual Meeting, Washington DC 2008 Data

11. Helsinki University of Technology Systems Analysis Laboratory 11 INFORMS Annual Meeting, Washington DC 2008 Sensitivity Analysis n How sensitive are the rankings to weight variation? –What if different weights were applied? –Relax point estimate weighting 1. Relative intervals around the point estimates –E.g.  =20 %, w i *=0.20: 2. Incomplete ordinal information –Attributes with w i *=0.20 cannot be less important than those with w i *=0.10 –All weights lower-bounded by 0.02

12. Helsinki University of Technology Systems Analysis Laboratory 12 INFORMS Annual Meeting, Washington DC 2008 Results: Rank-Sensitivity of Top Universities exact weights 20 % interval 30 % interval incompl. ordinal no information Unsensitive rankings ”Different weighting would likely yield a better ranking” Ranking University 10th442nd

13. Helsinki University of Technology Systems Analysis Laboratory 13 INFORMS Annual Meeting, Washington DC 2008 Conclusion n A model to compute attainable rankings –Sufficiently efficient even with hundreds of alternatives and several attributes n Attainable rankings communicate sensitivity of rank-orderings –Conceptually easy to understand –Holistic view of global sensitivity at a glance independently of the # of attributes n Applicable output in Preference Programming framework –Additional information leads to fewer attainable rankings n Connections to project prioritization –Initial screening of project proposals for e.g. portfolio-level analysis –Supports identification of ’clear decisions’ (cf. Liesiö et al. 2007) »”Select the ones ’surely’ in top 50” »”Discard the ones ’surely’ outside top 50”

14. Helsinki University of Technology Systems Analysis Laboratory 14 INFORMS Annual Meeting, Washington DC 2008 References »Butler, J., Jia, J., Dyer, J. (1997). Simulation Techniques for the Sensitivity Analysis of Multi-Criteria Decision Models. EJOR 103, 531-546. »Keeney, R.L., See, K.E., von Winterfeldt, D. (2006). Evaluating Academic Programs: With Applications to U.S. Graduate Decision Science Programs. Oper. Res. 54, 813-828. »Kirkwood, G., Sarin R. (1985). Ranking with Partial Information: A Method and an Application. Oper. Res. 33, 38-48 »Könnölä, T., Brummer, V., Salo A. (2007). Diversity in Foresight: Insights from the Fostering of Innovation Ideas. Technologial Forecasting & Social Change 74, 608-626. »Liesiö, J., Mild, P., Salo, A., (2007). Preference Programming for Robust Portfolio Modeling and Project Selection. EJOR 181, 1488-1505. »Liu, N.C., Cheng, Y. (2005). The Academic Ranking of World Universities. Higher Education in Europe 30, 127-136 »Mustajoki, J., Hämäläinen, R.P., Lindstedt, M.R.K. (2006). Using intervals for Global Sensitivity and Worst Case Analyses in Multiattribute Value Trees. EJOR 174, 278-292. »Rios Insua, D., French, S. (1991). A Framework for Sensitivity Analysis in Discrete Multi- Objective Decision-Making. EJOR 54, 176-190. »Salo, A., Hämäläinen R.P. (1992). Preference assessment by imprecise ratio statements. Oper. Res. 40, 1053-1061.