CONCEPTUAL ISSUES IN CONSTRUCTING COMPOSITE INDICES Nadia Farrugia Department of Economics, University of Malta Paper prepared for the INTERNATIONAL CONFERENCE ON SMALL STATES AND ECONOMIC RESILIENCE Organised by The Islands and Small States Institute of the Foundation for International Studies at the University of Malta and the Commonwealth Secretariat, London Valletta, Malta April 2007

Presentation Outline Introduction Introduction Desirable Attributes for Developing Statistics and Composite Indices Desirable Attributes for Developing Statistics and Composite Indices Main Conceptual Issues Main Conceptual Issues Indicator Selection Indicator Selection Dealing with Missing Data Dealing with Missing Data Normalisation Normalisation Weighting and Aggregation Weighting and Aggregation Testing and Reviewing the Results Obtained Testing and Reviewing the Results Obtained Conclusion Conclusion

Introduction

Definition A composite index, A composite index, is a weighted (linear) aggregation of a number of variables is a weighted (linear) aggregation of a number of variables w j is a weight, with 0≤w j ≤1 and ∑w j =1 w j is a weight, with 0≤w j ≤1 and ∑w j =1 X cj is the variable of country c in dimension j X cj is the variable of country c in dimension j for any country c the number of policy variables are equal to j=1,…,m. for any country c the number of policy variables are equal to j=1,…,m.

Uses Describe complex phenomena in a single indicator Describe complex phenomena in a single indicator Cross-national comparisons of country performance Cross-national comparisons of country performance Benchmarking exercises Benchmarking exercises General trends General trends Policy priorities and performance targets Policy priorities and performance targets Several examples of renowned composite indices, stock market indices, RPI, GDP Several examples of renowned composite indices, stock market indices, RPI, GDP

Strengths Summarises complex and multi-dimensional issues Summarises complex and multi-dimensional issues Helps set the direction for policymakers and to focus the discussion Helps set the direction for policymakers and to focus the discussion Supports decision making Supports decision making Helps disseminate information Helps disseminate information Make stakeholders and the public more aware of certain problems Make stakeholders and the public more aware of certain problems Generates academic discussion Generates academic discussion

Weaknesses Subjectivity in computation Subjectivity in computation May send misleading policy messages and can easily be misused May send misleading policy messages and can easily be misused May conceal divergences between different components May conceal divergences between different components Increase difficulty in identifying proper remedial action Increase difficulty in identifying proper remedial action Measurement problems Measurement problems

Desirable Attributes

Quality Frameworks IMF – Data Quality Assurance Framework IMF – Data Quality Assurance Framework Eurostat Framework Eurostat Framework OECD – Quality Framework and Guidelines for OECD Statistics OECD – Quality Framework and Guidelines for OECD Statistics Booysen – Dimensions for Classifying and Evaluating Development Indicators Booysen – Dimensions for Classifying and Evaluating Development Indicators Briguglio – Desirable Characteristics for Developing Vulnerability Indices Briguglio – Desirable Characteristics for Developing Vulnerability Indices JRC-OECD – Handbook on Constructing Composite Indicators JRC-OECD – Handbook on Constructing Composite Indicators

Desirable Attributes of Composite Indices 1. Accuracy 2. Simplicity and Ease of Comprehension 3. Methodological Soundness 4. Suitability for International and Temporal Comparisons 5. Transparency 6. Accessibility 7. Timeliness and Frequency 8. Flexibility

Main Conceptual Issues

Indicator Selection Indicator Selection Dealing with Missing Data Dealing with Missing Data Normalisation Normalisation Weighting and Aggregation Weighting and Aggregation Testing and Reviewing the Results Obtained Testing and Reviewing the Results Obtained

Indicator Selection Define the concept Define the concept Select indicators which satisfy desirable attributes Select indicators which satisfy desirable attributes Do not select variables which beg the question Do not select variables which beg the question Draft an initial indicator set and review the available data Draft an initial indicator set and review the available data Keep the number of variables as small as possible but not fewer than necessary (PCA, FA) Keep the number of variables as small as possible but not fewer than necessary (PCA, FA)

Indicator Selection (Cont.) Check for correlation between the variables or sub-indices (rank correlation test, Cronbach coefficient alpha, cluster and discriminant analysis) Check for correlation between the variables or sub-indices (rank correlation test, Cronbach coefficient alpha, cluster and discriminant analysis) Review the indicators selected and seek external advice and opinion Review the indicators selected and seek external advice and opinion

Dealing with Missing Data Exclude the country from the analysis Exclude the country from the analysis Imputation methods: Single or Multiple Imputation methods: Single or Multiple

Single Imputation Methods Case deletion Case deletion Mean/median/mode estimation Mean/median/mode estimation Hot deck imputation Hot deck imputation Regression imputation Regression imputation

Multiple Imputation Methods Regression Method Regression Method Propensity Score Method Propensity Score Method Markov Chain Monte Carlo Algorithm Markov Chain Monte Carlo Algorithm

Quantifying Qualitative Data Using a mapping (Likert) scale Using a mapping (Likert) scale Optimal spread of the scale Optimal spread of the scale Permits non-linearity Permits non-linearity Defect relates to subjectivity Defect relates to subjectivity

Normalisation Rescaling Rescaling Standardisation (or z-scores) Standardisation (or z-scores) Percentage differences over previous years Percentage differences over previous years Ratios Ratios Rankings Rankings Measuring the relative position vis-à-vis a specified point Measuring the relative position vis-à-vis a specified point

Weighting and Aggregation Equal Weighting Equal Weighting Differential Weighting Differential Weighting Country-Specific or Indicator-Specific Weights Country-Specific or Indicator-Specific Weights Weights Over Time: Constant or Changing Weights Over Time: Constant or Changing

Differential Weighting Weights Reflecting the Statistical Quality of the Data Weights Reflecting the Statistical Quality of the Data Stochastic Weights Stochastic Weights Participatory Methods Participatory Methods Precautionary Principle Precautionary Principle Regression Method Regression Method Benefit-of-the-Doubt Weighting System Benefit-of-the-Doubt Weighting System

Aggregation Linear or geometric aggregation Linear or geometric aggregation Aggregation methods and weighting systems Aggregation methods and weighting systems Non-compensatory multi-criteria aggregation Non-compensatory multi-criteria aggregation

Testing and Reviewing the Results Obtained Uncertainty and Sensitivity Analysis Uncertainty and Sensitivity Analysis Outliers Outliers Expert Opinion Expert Opinion Analysing the Results Obtained Analysing the Results Obtained

Conclusion

Conclusion Composite indices have their pros and cons. Composite indices have their pros and cons. Hard to imagine that the debate on the use of composite indices will be ever settled. Hard to imagine that the debate on the use of composite indices will be ever settled. Composite indices should be identified for what they are. Composite indices should be identified for what they are. However, their importance should not be undermined. However, their importance should not be undermined. Provided they are built on sound methodological considerations they are very useful to portray complex phenomena in a simple manner. Provided they are built on sound methodological considerations they are very useful to portray complex phenomena in a simple manner.

Thank you!