1. May 2007 Ecoinformatics Indicators Workgroup: 21-22 June 2007 European Environment Agency Sustainability indicators – composites and aggregates Giuseppe Munda and Andrea saltelli European Commission, Joint Research Centre, G09

2. May 2007 Ecoinformatics Indicators Workgroup: 21-22 June 2007 European Environment Agency Uncertainty Management Silvio Funtowicz, Giuseppe Munda and Andrea saltelli European Commission, Joint Research Centre, G09

3. May 2007 Structure of the presentation Complexity, Quality and Uncertainty Social Choice and Aggregation Rules: Non Compensatory Multi-Criteria Evaluation Sensitivity Analysis Non-Compensatory Multi-Criteria and Sensitivity Analysis

4. May 2007 Complexity, Quality and Uncertainty

5. May 2007 Monetary versus Physical Sustainability Indicators? The subcomponents needed for the building the aggregate index are ad hoc. All the indexes are based on the assumptions that a common measurement rod needs to be established for aggregation purposes (money, energy, space, and so on). This creates the need of making very strong assumptions on conversion coefficients to be used and on compensability allowed. Aggregate indexes are somewhat confusing, if one wishes to derive policy suggestions. For example, ISEW, ecological footprint, weak sustainability index All these approaches belong to the more general family of composite indicators and as a consequence, the assumptions used for their construction are common to them all.

6. May 2007 The EC develops or uses several composite indices - Of Internal Market (†) - Of Innovation - Of knowledge based economy - Of firm readiness to take up e-business (e- readiness) … Not mentioning the historic ones as GDP, CPI, …

7. May 2007 See (http://www.oecd.org/publications/)

8. May 2007 On the OECD-JRC handbook on CI the ‘pros’ Composite indicators: Can summarise complex or multi-dimensional issues in view of supporting decision-makers. Easier to interpret than trying to find a trend in many separate indicators. Facilitate the task of ranking countries on complex issues in a benchmarking exercise.

9. May 2007 On the OECD-JRC handbook on CI the ‘pros’ Composite indicators: Can assess progress of countries over time on complex issues. Reduce the size of a set of indicators or include more information within the existing size limit. Place issues of country performance and progress at the centre of the policy arena. (Advocacy) Facilitate communication with general public (i.e. citizens, media, etc.) and promote accountability. (Advocacy)

10. May 2007 … while composite indicators’ ‘cons’ are: May send misleading policy messages if they are poorly constructed or misinterpreted. May invite simplistic policy conclusions. May be misused, e.g., to support a desired policy, if the construction process is not transparent and lacks sound statistical or conceptual principles. (if one dislikes the policy under discussion, he/she will most likely invoke one or more of the these)

11. May 2007 … and (cons): The selection of indicators and weights could be the target of political challenge. (a CI could exacerbate disagreement rather than focus minds) May lead to inappropriate policies if dimensions of performance that are difficult to measure are ignored. May disguise serious failings in some dimensions and increase the difficulty of identifying proper remedial action. (the problem of compensability)

12. May 2007 The practices described in the handbook are already being applied to existing composite indicators, e.g. the 2006 EPI from Yale, Columbia, WEF, JRC.

13. May 2007 For sure modelling is subject toady to an unprecedented critique, which is no longer limited to post-modern philosophers but involves intellectuals and scientists of different political hues. Have models fallen out of grace and is modelling -- just useless arithmetic as claimed by Pilkey and Pilkey-Jarvis 2007?

14. May 2007 Useless Arithmetic: Why Environmental Scientists Can't Predict the Future by Orrin H. Pilkey and Linda Pilkey-Jarvis Quantitative mathematical models used by policy makes and government administrators to form environmental policies are seriously flawed

15. May 2007 One of the examples discussed concerns the Yucca Mountain repository for radioactive waste disposal, where a very large model called TSPA (for total system performance assessment) is used to guarantee the safe containment of the waste. TSPA is Composed of 286 sub-models.

16. May 2007 TSPA (like any other model) relies on assumptions -- a crucial one being the low permeability of the geological formation and hence the long time needed for the water to percolate from the desert surface to the level of the underground disposal. The confidence of the stakeholders in TSPA was not helped when evidence was produced which could lead to an upward revision of 4 orders of magnitude of this parameter.

17. May 2007 We just can’t predict, concludes N. N. Taleb, and we are victims of the ludic fallacy, of delusion of uncertainty, and so on. Modelling is just another attempt to ‘Platonify’ reality… Nassim Nichola Taleb, The Black Swan, Penguin, London 2007

18. May 2007 The critique of models Since Galileo's times scientists have had to deal with the limited capacity of the human mind to create useful maps of ‘world’ into ‘model’. The emergence of ‘laws’ can be seen in this context as the painful process of simplification, separation and identification which leads to a model of uncharacteristic simplicity and beauty. Andrea Saltelli

19. May 2007 Complexity is an inherent property of natural and social systems addressed ignored

20. May 2007 What is a city?

21. May 2007 MAN-MADE NATURAL CULTURAL HUMAN SOCIAL A co-evolutionary interpretation of a city

22. May 2007

23. GOVERNANCE in a COMPLEX world Who has the power to impose a language of valuation?Who has the power to impose a language of valuation? Who has the power to privilege one analytical level or time-space scale?Who has the power to privilege one analytical level or time-space scale? Who has the power to simplify the complexity?Who has the power to simplify the complexity? Contradictory scientific findings and lay opinions must be integrated into the policy.

24. May 2007 Science for the post normal age is discussed in Funtowicz and Ravetz (1990, 1993, 1999) mostly in relation to Science for policy use. Jerry Ravetz Silvio Funtowicz facts uncertain values in dispute stakes high decisions urgent facts uncertain values in dispute stakes high decisions urgent

25. May 2007 To what extent a god technical preparation for a CI can make it more robust (to uncertainties in data, weights,…) resilient (remain relevant over time), defensible (in dialogue with stakeholders…) and facilitate negotiation rather than stand off? (Snippets from the JRC-OECD handbook)

26. May 2007 From the handbook. Step 1. Developing a theoretical framework What is badly defined is likely to be badly measured … Excerpt: For example, the Growth Competitiveness Index (GCI) developed by the World Economic Forum is founded on the idea “that the process of economic growth can be analysed within three important broad categories: the macroeconomic environment, the quality of public institutions, and technology."

27. May 2007 Step 2. Selecting variables A composite indicator is above all the sum of its parts… Excerpt: The strengths and weaknesses of composite indicators largely derive from the quality of the underlying variables. […] While the choice of indicators must be guided by the theoretical framework for the composite, the data selection process can be quite subjective as there may be no single definitive set of indicators. As in mathematical models

28. May 2007 Step 3. Multivariate analysis Analysing the underlying structure of the data is still an art … After Step 3, the constructor should have… Identified sub-groups of indicators or groups of countries that are statistically “similar”. …

29. May 2007 Step 4. Imputation of missing data. The idea of imputation could be both seductive and dangerous … Step 5. Normalisation of data Avoid adding up apples and oranges … Step 6. Weighting and aggregation The relative importance of the indicators can be become the substance of a negotiation …

30. May 2007 Step 7. Robustness and sensitivity Sensitivity analysis can be used to assess the robustness of composite indicators … Step 8. Links to other variables Composite indicators can be linked to other variables and measures …

31. May 2007 The four-quadrant model of the Sustainable Project Appraisal Routine (SPeAR®). Step 9. Back to the details De-constructing composite indicators can help extend the analysis … Step 10. Presentation and dissemination A well-designed graph can speak louder than words …

32. May 2007 It is clear that from a mathematical point of view a composite indicator entails a weighted linear aggregation rule applied to a set of variables.

33. May 2007 The meaning of weights in linear aggregation rules “Greater weight should be given to components which are considered to be more significant in the context of the particular composite indicator”. (OECD, 2003, p. 10). Weights as symmetrical importance, that is "… if we have two non-equal numbers to construct a vector in R2, then it is preferable to place the greatest number in the position corresponding to the most important criterion." (Podinovskii, 1994, p. 241).

34. May 2007 Weights in linear aggregation rules have always the meaning of trade-off ratio. In all constructions of a composite indicator, weights are used as importance coefficients, as a consequence, a theoretical inconsistency exists. The assumption of preference independence is essential for the existence of a linear aggregation rule. Unfortunately, this assumption has very strong consequences which often are not desirable in a composite indicator. In standard composite indicators, compensability among the different individual indicators is always assumed; this implies complete substitutability among the various components considered. For example, in a sustainability index, economic growth can always substitute any environmental destruction or inside e.g., the environmental dimension, clean air can compensate for a loss of potable water. From a descriptive point of view, such a complete compensability is often not desirable.

35. May 2007 Example of a Linear Aggregation Rule A hypothetical composite: inequality, environmental degradation, GDP per capita and unemployment Country A: 21, 1, 1, 1  6 Country B: 6, 6, 6, 6  6 Obviously the two countries would represent very different social conditions that would not be reflected in the composite.

36. May 2007 Social Choice and Aggregation Rules: Non-Compensatory Multi-Criteria Evaluation

37. May 2007 The Plurality Rule

38. May 2007 The Plurality Rule!

39. May 2007 Indic. GDPUnemp. Rate Solid wastes Inc. disp. Crime rate Country A 25,0000.150.49.240 B 45,0000.100.713.252 C 20,0000.080.355.380 weights0.165 0.3330.165 Sustainability Indicator ABCABC A B C 0 0.666 0.333 0.333 0 0.333 0.666 0.666 0 AB = 0.333+0.165+0.165=0.666 BA = 0.165+0.165=0.333 AC = 0.165+0.165=0.333 CA = 0.165+0.333+0.165=0.666 BC = 0.165+0.165=0.333 CB = 0.165+0.333+0.165=0.333 ABC = 0.666 + 0.333 + 0.333 = 1.333 BCA = 0.333 + 0.666 + 0.333 = 1.333 CAB = 0.666 + 0.666 + 0.666 = 2 ACB = 0.333 + 0.666 + 0.666 = 1.666 BAC = 0.333 + 0.333 + 0.333 = 1 CBA = 0.666 + 0.333 + 0.666 = 1.666

40. May 2007 A Real-World Application for 146 Countries

41. May 2007

42. Sensitivity Analysis

43. May 2007 In sensitivity analysis: Type I error: assessing as important a non important factor Type II: assessing as non important an important factor Type III: analysing the wrong problem

44. May 2007 The optimality of a model must be weighted with respect to the task. According to Beck et al. 1997, a model is relevant when its input factors cause variation in the ‘answer’. Requirement 1 - Focus Another implication Models must change as the question put to them changes.

45. May 2007 Requirement 2. Multidimensional averaging. In a sensitivity analysis all known sources of uncertainty should be explored simultaneously, to ensure that the space of the input uncertainties is thoroughly explored and that possible interactions are captured by the analysis. Requirements

46. May 2007 Requirement 3. Important how?. Define unambiguously what you mean by ‘importance’ in relation to input factors / assumptions. Requirements

47. May 2007 Requirement 4. Pareto. Be quantitative. Quantify relative importance by exploiting factors’ unequal influence on the output. … Requirement N. Look at uncertainties before going public with findings. Requirements

48. May 2007 Non-Compensatory Multi-Criteria and Sensitivity Analysis

49. May 2007 The three dimensions of sustainability chosen - environment, society, economy- are described by 29 indicators or variables. The data used are the most recent (2004, or nearest year) data from the EUROSTAT regional database.

50. May 2007

57. QUALITY OF PRODUCT PROCEDURAL RATIONALITY LEARNING HOLARCHIES MCDA QUALITY OF “SOCIAL” PROCESS PARTICIPATION TRANSPARENCY MULTI/INTER-DISCIPLINARITY ETHICS RESPONSIBILITY CONSISTENCY

58. May 2007 THANKS And what does the handbook say? ‘In God we trust, all others bring data’

59. May 2007 REFERENCES Funtowicz S.O., Ravetz J.R. (1990) - Uncertainty and quality in science for policy, Kluwer Academic Publishers, Dordrecht. Funtowicz S., Martinez-Alier J., Munda G. and Ravetz J. – (1999) Information tools for environmental policy under conditions of complexity, European Environmental Agency, Experts’ Corner, Environmental Issues Series, No. 9. Munda G. (1995) - Multicriteria evaluation in a fuzzy environment, Physica-Verlag, Contributions to Economics Series, Heidelberg. Munda G. (2007) – Social Multi-Criteria Evaluation, Economics Series, Springer-Verlag, Heidelberg, New York, Forthcoming. Munda G. and Saisana M. - Is Regional Sustainability Compatible with Economic Success? A Computation for Spain Based on Multi-Criteria and Sensitivity Analysis, submitted to Journal of Economic Geography. Nardo M., Saisana M., Saltelli A., Tarantola S., Hoffman A., Giovannini E. (2005) – Handbook on constructing composite indicators: methodology and user guide, OECD Statistics Working Paper, Paris. Saltelli A. Tarantola S., Campolongo, F. and Ratto, M. (2004) - Sensitivity Analysis in Practice. A Guide to Assessing Scientific Models, John Wiley & Sons publishers, New York. Saltelli, A., Andres, T., Campolongo, F., Cariboni J., Gatelli D., Ratto, M., Saisana, M., Tarantola, S. -Global sensitivity analysis. Gauging the worth of scientific models. A textbook of methods to evidence how model-based inference depends upon model specifications and assumptions, John Wiley, 2007

60. May 2007 REFERENCES Munda G. (2005) – “Measuring sustainability”: a multi-criterion framework, Environment, Development and Sustainability Vol 7, No. 1, pp. 117-134. Munda G., Nardo M. (2005) – Constructing consistent composite indicators: the issue of weights, EUR 21834 EN, Joint Research Centre, Ispra. Munda G., Nardo M. (2007) – Non-compensatory/non-linear composite indicators for ranking countries: a defensible setting, forthcoming in Applied Economics. Nardo M., Saisana M., Saltelli A., Tarantola S., Hoffman A., Giovannini E. (2005) – Handbook on constructing composite indicators: methodology and user guide, OECD Statistics Working Paper, Paris. Saisana M., Tarantola S., Saltelli A. (2005) - Uncertainty and sensitivity techniques as tools for the analysis and validation of composite indicators. Journal of the Royal Statistical Society A, 168(2), 307-323.