1 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Ranking and rating: Woodo or Science? Andrea Saltelli, European Commission, Joint Research Centre, Unit of Applied Statistics and Econometrics X Conferenza Nazionale di Statistica Roma 15-16/12/2010

2 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli October June 20061,440 May 20071,900 October 20083,030 September 20094,420 August 20105,240 (5,280 today) Searching “composite indicators” on Scholar Google:

3 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli David Hand, president of the UK Royal statistical Society “League tables […] are an easy target for criticism. […] surgeon can refuse to operate on the difficult cases, schools can refuse to enter those pupils likely to do poor in examinations, health authorities can defer making appointments for some patients, so that the waiting lists look smaller, and so on.”

4 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli

5 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators, i.e. the arbitrary character of the procedures used to weight their various components. Adding: […] The problem is not that these weighting procedures are hidden, non-transparent or non- replicable – they are often very explicitly presented by the authors of the indices, and this is one of the strengths of this literature. The problem is rather that their normative implications are seldom made explicit or justified.

6 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli […] The problem is not that these weighting procedures are hidden, non-transparent or non- replicable – they are often very explicitly presented by the authors of the indices, Disagree: Weighting problems are often not so evident. E.g. Most composite indicators are built by linear aggregation which are almost by definition wrong. Can something be done to alleviate the problem?

7 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Are ubiquitous Can deceive Can inform How can one tell the good from the bad? A first case study: University ranking

8 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Rickety Numbers: Volatility of university rankings and policy implications, Michaela Saisana , Béatrice d'Hombres, Andrea Saltelli, To appear on Research Policy ~2010/2011 Sources (I):

9 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Sources (II):

10 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Joint OECD-JRC handbook. 5 years of preparation, 2 rounds of consultation with OECD high level statistical committee, finally endorsed March 2008 with one abstention Sources III

11 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli 2010 Rule of Law Index (World Justice Project) 2010 Global Competitiveness Index (WEF) 2010 Regional Competitiveness Index (DG REGIO, JRC) 2010 Multidimensional Poverty Assessment Tool (UN IFAD) 2010/2008/2006 Environmental Performance Index (Yale & Columbia Uni) 2009 Index of African Governance (Harvard Kennedy School) 2008 Product Market Regulation Index (OECD) 2008 European Lifelong Learning Index (Bertelsmann Foundation, CCL) 2007 Alcohol Policy Index (New York Medical College) 2007 Composite Learning Index (Canadian Council on Learning) 2002/2005 Environmental Sustainability Index (Yale & Columbia University) Methodology applied to:

12 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Sensitivity Analysis

13 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli <<I have proposed a form of organised sensitivity analysis that I call “global sensitivity analysis” in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.>> Edward E. Leamer, 1990, Let's Take the Con Out of Econometrics, American Economics Review, 73 (March 1983),

14 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli “ One reason these methods are rarely used is their honesty seems destructive; ” Ibidem Tantalus on the Road to Asymptopia Edward E. Leamer, 2010 Journal of Economic Perspectives, 24, (2), 31–46.

15 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Peter Kennedy, A Guide to Econometrics. Anticipating criticism by applying sensitivity analysis. This is one of the ten commandments of applied econometrics according to Peter Kennedy: <<Thou shall confess in the presence of sensitivity. Corollary: Thou shall anticipate criticism >> The critique of models Uncertainty

16 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli

17 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Two international university rankings  SJTU ranking  THES ranking  Robustness (uncertainty & sensitivity analysis)

18 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli University rankings are used to judge about the performance of university systems

19 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Two international university rankings yearly published + Very appealing for capturing a university’s multiple missions in a single number + Allow one to situate a given university in the worldwide context - Can lead to misleading and/or simplistic policy conclusions

20 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Questions: can we have confidence in university rankings? How much do the university ranks depend on the methodology (weighting scheme, aggregation, indicators)?

21 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli SJTU ranking

22 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli METHODOLOGY 6 indicators Best performing institution =100; score of other institutions calculated as a percentage Weighting scheme chosen by rankers Linear aggregation of the 6 indicators SJTU ranking

23 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli PROS and CONS 6 « objective » indicators Focus on research performance, overlooks other U. missions. Biased towards hard sciences intensive institutions Favours large institutions SJTU ranking

24 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli THES ranking

25 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli METHODOLOGY 6 indicators z-score calculated for each indicator; best performing institution =100; other institutions are calculated as a percentage Weighting scheme: chosen by rankers Linear aggregation of the 6 indicators THES ranking

26 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli PROS and CONS Attempt to take into account teaching quality Two expert-based indicators: 50% of total Subjective indicators, lack of transparency Substantial yearly changes in methodology Measures research quantity THES ranking

27 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli RED=UK (all under the SJTU=THES line…)

28 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Robustness analysis of SJTU and THES

29 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli  Harvard, Stanford, Berkley, Cambridge, MIT: top 5 in more than 75% of our simulations.  Univ California SF: original rank 18 th but could be ranked anywhere between the 6 th and 100 th position  Impact of assumptions: much stronger for the middle ranked universities SJTU ranking

30 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli  Impact of uncertainties on the university ranks is even more apparent.  M.I.T.: ranked 9th, but confirmed only in 13% of simulations (plausible range [4, 35])  Very high volatility also for universities ranked 10 th -20th position, e.g., Duke Univ, John Hopkins Univ, Cornell Univ. THES ranking

31 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Thus far: Apart from the top 10 universities, neither the SJTU nor the THES should be used to compare the performance of individual universities. According to SJTU Universities in the US outperform those in Europe – less so for THES but there is a bias toward UK universities

32 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Can we say more about the relative quality of THES and SJTU? Are these indices coherent? Do the weights given by developers reflect the importance of the variables? Let us try a global sensitivity measure

33 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Nobel Alumni SJTU

34 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Using these points we can compute a statistics that tells us: How much (on average) would the variance of SJTU score be reduced if I could fix the variable ‘Alumni with Nobel ’?

35 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli This measure S i shall be our ruler for ‘importance’; example: S i =0.79  I could reduce the variation of the SJTU score by 79% by fixing ‘Nobel Alumni’.

36 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli At this point I can compare the importance of a sub-indicator as given by the nominal weight (assigned by developers) with the importance as measured by S i to test the index for coherence.

37 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli SJTU ranking weightSi

38 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli THES ranking weightSi

39 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli How about pillars In developing composite indicators pillars often represents normative dimensions which are given by design equal weights.

40 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli How about pillars We can now test when this is the case on the real data.

41 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Index of African governance weight Si

42 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Sustainable society index (NL) (1/7)*100 (2/7)*100

43 Constructing Composite Indicators: From Theory to Practice ECFIN, November 11-12, 2010 Andrea Saltelli Conclusions -Two methods to test the quality of composite indicators -Uncertainty modeling and propagations (invasive) - Check of coherence (non invasive)