Composite Indicators - The Controversy and the way forward Andrea Saltelli, Michela Nardo, Michaela Saisana and Stefano Tarantola European Commission Joint Research Centre of Ispra Statistics, Knowledge and Policy: OECD World Forum on Key Indicators Palermo November 2004

Prepared with Michela Nardo, Michaela Saisana & Stefano Tarantola Based on: [3] Saisana M., Saltelli A., Tarantola S., 2005, Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, J. R. Stat. Soc. A, 168(2), [11] Joint OECD JRC handbook on good practices in composite indictors building.

Outline CI controversy Composite Indicators as models Wackernagels critique of ESI … Putting the critique into practice: the TAI example Conclusions

CI controversy EU structural indicators – scoreboards versus indices

Report from the Commission to the Spring European Council 2004, Annex 1 Relative Performance Relative Improvement in Performance (av. since 1999)

Relative Performance

Assessing policies: Green – Country policy on a good path; Yellow – Country policy on a bad path (expert judgment) LevelsyATBE Labour productivity (EU 15=100) Employment rate (%) Employment rate of older workers (%)

Source: Financial Times Thursday January Enter the FT analysts … Source: Spring Report, European Commission 2004

Categorisation (star rating[*]) in three groups LEADERS UK, NL SE, DK, AT,LU MIDDLE OF THE ROAD DE, FI, IE, BE, FR LAGGARDS IT, GR, ES, PT done by FT and based likely on same synoptic performance and improvement tables in the Spring Report, 2004, Annex 1 (yellow-green boxes) [*] Like in the UK NHS hospital rating

Can league tables be avoided? Or are they an ingredient of an overall analysis and presentational strategy: Long list of 107 Short List of 14 Synoptic tables League tables

> Literature Review of Frameworks for Macro- indicators, Andrew Sharpe, 2004, Centre for the Study of Living Standards, Ottawa, CAN.

Reviews on methodologies and practices on composite indicators : State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development (2002) Michaela Saisana & Stefano Tarantola, European Commission, Joint Research Centre Composite indicators of country performance: a critical assessment (2003) Michael Freudenberg, OECD. Literature Review of Frameworks for Macro-indicators (2004), Andrew Sharpe, Centre for the Study of Living Standards, Ottawa, CAN. Measuring performance: An examination of composite performance indicators (2004) Rowena Jacobs, Peter Smith, Maria Goddard, Centre for Health Economics, University of York, UK. Methodological Issues Encountered in the Construction of Indices of Economic and Social Well-being (2003) Andrew Sharpe Julia Salzman Methodological Choices Encountered in the Construction of Composite Indices of Economic and Social Well-Being, Julia Salzman, (2004) Center for the Study of Living Standards, Ottawa, CAN.

Pros & Cons (Saisana and Tarantola, 2002) Pros Composite indicators can be used to summarise complex or multi-dimensional issues, in view of supporting decision-makers. Composite indicators provide the big picture […]. They facilitate the task of ranking countries on complex issues. Composite indicators can help attracting public interest […] Composite indicators could help to reduce the size of a list of indicators […].

Cons Composite indicators may send misleading, non-robust policy messages if they are poorly constructed or misinterpreted [… or ] may invite politicians to draw simplistic policy conclusions […] The construction of composite indicators involves stages where judgement has to be made: the selection of sub- indicators, choice of model, weighting indicators and treatment of missing values etc. […] There could be more scope for disagreement among Member States about composite indicators than on individual indicators […].

Pros & Cons (JRSS paper) […] it is hard to imagine that debate on the use of composite indicators will ever be settled […] official statisticians may tend to resent composite indicators, whereby a lot of work in data collection and editing is wasted or hidden behind a single number of dubious significance. On the other hand, the temptation of stakeholders and practitioners to summarise complex and sometime elusive processes (e.g. sustainability, single market policy, etc.) into a single figure to benchmark country performance for policy consumption seems likewise irresistible.

Composite indicators as models … and the critique of models

Indicators as models … and the critique of models The nature of models, after Rosen

The critique of models After Rosen, 1991, World (the natural system) and Model (the formal system) are internally entailed - driven by a causal structure. [Efficient, material, final for world – formal for model] Nothing entails with one another World and Model; the association is hence the result of a craftsmanship. N Natural system F Formal system Decoding Entailment Encoding

Wackernagels critique of ESI …

Environmental sustainability Index, figure from The Economist, Green and growing, The Economist, Jan 25th 2001, Produced on behalf of the World Economic Forum (WEF), and presented to the annual Davos summit this year. The critique of indicators

Mathis Wackernagel, mental father of the Ecological Footprint and thus an authoritative source in the Sustainable Development expert community, concludes an argumented critique of the study done presented at Davos by noting: The critique of indicators: Robustness …

"Overall, the report would gain from a more extensive peer review and a sensitivity analysis. The lacking sensitivity analysis undermines the confidence in the results since small changes in the index architecture or the weighting could dramatically alter the ranking of the nations. The critique of indicators: Robustness …

The quality of a composite indicator is in its fitness or function to purpose. The economist A. K. Sen, Nobel prize winner in 1998, was initially opposed to composite indicators but was eventually seduced by their ability to put into practice his concept of Capabilities, the range of things that a person could do and be in her life. Sen A., 1989, Development as Capabilities Expansion, Journal of Development Planning 19, The critique of indicators: Fitness

The example of the capabilities is relevant to the issue: CI are supposedly good at capture complex (someone would say poorly defined) concepts such as sustainability, welfare, achievement of an EU internal market, competitiveness, etc. Said otherwise, complex processes call for scoreboards, and scoreboards cry for an index. The critique of indicators: Fitness

In discussing pedigrees matrices for statistical information Funtowicz and Ravetz note (in Uncertainty and Quality in Science for Policy, 1990 [6]) […] any competent statistician knows that "just collecting numbers" leads to nonsense […] so in "Definition and Standards" we put "negotiation" as superior to "science", since those on the job will know of special features and problems of which an expert with only a general training might miss. We would add that, however good the scientific basis for a given composite indicator, its acceptance relies on negotiation and peer acceptance. The critique of indicators: Fitness

(1) A composite constructed on the basis of underlying indicators with high internal correlation will give a very robust CI, whose values and ranking are moderately affected by changes in the selection of weights, the normalisation method and other steps involved in the analysis (see paper, this conference). Open issues in CI Building 1 – Variables correlation

(2) When building composite indicators using automated tools such as factor analysis, one seeks to obtain a set of totally uncorrelated new variables. While this can be a powerful tool to benchmark countries performance, or to produce e.g. leading or lagging synthetic indicators, the interpretation in terms of original variables becomes more difficult. Variables correlation

(2) At the same time, it would be very difficult to imagine a composite indicator made of truly orthogonal variables. (3) In a multicriteria context, one would consider the existence of correlation among the attributes of an issue as a feature of the issue, not to be compensated for. A cars speed and beauty are likely correlated with one another, but this does not imply that we are willing to trade speed for design. Variables correlation

(1) Munda, and Nardo, 2003 [12], noticed how weights, customarily conceived as importance measures, act in practice as substitution rates, e.g. wi/wj is the ratio of substitution (or compensation) of indicator i with indicator j. Open issues in CI Building 2– Compensability

(2) This may be perceived as an important limitation of a CI (e.g. literacy should not be traded with GDP per capita). When one is not willing to accept this kind of trade offs, e.g. when the variable cannot be compensated with another, a multi criteria approach can be applied. Compensability

(2) This may be perceived as an important limitation of a CI (e.g. literacy should not be traded with GDP per capita). When one is not willing to accept this kind of trade offs, e.g. when the variable cannot be compensated with another, a multi criteria approach can be applied. See paper for a simplified description of a Condorcet-type of ranking procedure based on Munda, 1995, [13]. This approach produces rankings (ordered sequence of countries) instead of an index. Compensability

(3) The ordering thus obtained is only based on the weights, and on the sign of the difference between countries values for a given indicator, the magnitude of the difference being ignored. (4) With this approach no compensation occurs. To exemplify, a country that does marginally better on many indicators comes out better than a country that does a lot better on a few ones because it cannot compensate deficiencies in some dimensions with outstanding performances in others. Compensability

Points touched upon in this brief discussion of open issues in CI building are tackled in a forthcoming joint paper from OECD and JRC on composite indicators building. It aims to be a guide to the construction and use of CI. Ongoing work: the OECD JRC handbook

Theoretical framework - What is badly defined is likely to be badly measured. Data selection – The quality of composite indicators depends largely on the quality of the underlying indicators. Multivariate analysis – Multivariate statistic is a powerful tool for investigating the inherent structure in the indicators set. Imputation of missing data– The idea of imputation is both seductive and dangerous. Ongoing work: the OECD JRC handbook

Normalisation – Avoid adding up apples and pears. Weighting and aggregation – Relative importance of the indicators and compensability issues. Robustness and sensitivity – The iterative use of uncertainty and sensitivity analysis during the development of a composite indicator can contribute to its well-structuring. Ongoing work: the OECD JRC handbook

Link to other variables – Correlation with other simple indicators or composite indicators. Visualisation – If arguments are not put into figures, the voice of science will never be heard by practical men. Back to the real data – Deconstructing composite indicators for analytical purposes. Ongoing work: the OECD JRC handbook