Comparative Policy Evaluation: Constructing Indices Alison Smith Social Policy University of Edinburgh

Seminar Structure Using indices in comparative policy analysis: the UNICEF example Building indices Harmonisation of variables Where to find comparative survey data for your own work Design your own index!

“UK is accused of failing children” Headline BBC news 14/2/07 – UK comes bottom of league table for child well-being across 21 industrialised countries Find the report online: “Child Poverty in Perspective: An Overview of Child Well-Being in Rich Countries”, UNICEF Innocenti Research Centre, Report Card 7

Child Well-Being Table 1.Netherlands 2.Sweden 3.Denmark 4.Finland 5.Spain 6.Switzerland 7.Norway 8.Italy 9.Republic of Ireland 10.Belgium 11.Germany 12.Canada 13.Greece 14.Poland 15.Czech Republic 16.France 17.Portugal 18.Austria 19.Hungary 20.United States 21.United Kingdom

Measurement and Policy (adapted from Innocenti report card 7:3) ‘to improve something, first measure it’ ‘measurement serves as the handrail of policy’ ‘measurement and comparison shows what is achievable in practice’ ‘comparisons demonstrate that given levels of child well-being are not inevitable but policy-susceptible’

How to Use the Index As a stand alone piece of research As a macro level indicator in cross- national multivariate analysis

Child Well-Being in Rich Countries Rank Score Netherlands Sweden Denmark Hungary United States United Kingdom

Dimensions of Variation 1.Material well-being 2.Health and safety 3.Educational well-being 4.Family and peer relationships 5.Behaviours and risks 6.Subjective well-being

Graphical Interface (not provided by UNICEF)

Components of the Dimensions The dimension scores are derived from information on the components Each component draws on one or more indicators The component scores are averaged to arrive at each country’s overall rating

Choice of Indicators For each indicator, countries are given a score which reveals how far that country stands above or below the OECD average Where more than one indicator has been used, scores are averaged giving an equal weighting to each indicator Evaluating indicators: reliability and validity

Building Indices Indices are composite measures of a concept What to do with the different elements of the index –Add them –Average ranking –Product scores –Factor scores from factor analysis

Standardising Variables Creating a common scale to facilitate comparison A z-score is a unit’s score expressed as the number of standard deviation units that it lies from the mean of the variable the z-score tells us two things for each unit/country –how far that individual unit’s score is from the mean –the individual unit’s position in the group

Complicating Issues Interpretation of index score Equivalence of items Missing data

Measuring Concepts Nominal definition Operational definitions s/authors.htmhttp://qb.soc.surrey.ac.uk/resources/author s/authors.htm

The Ladder of Abstraction I.Index of the concept of child well-being II.Dimensions of variation within this concept III.Components of each dimension IV.Indicator measures taken from survey data for each component

Survey Data Systematic measurement across cases to permit comparison Cases are collected as units of analysis: individual, firm, country The logic of survey analysis is that variation in one variable is matched with variations in other variables (following J. S. Mill A System of Logic)

Comparative Data Sets MTUS - CNEF - h/Centers-Programs/German-Panel/cnef.cfm h/Centers-Programs/German-Panel/cnef.cfm ECHP ISSP - Eurobarometer European Time Use LIS -

Cross-National Harmonisation of Variables No international standard for measuring complex concepts like income, education, or employment. “Researchers interested in doing cross-national work must investigate the institutions, laws and cultural patterns of a country in order to ensure that the variables they create for their analyses are equivalently defined across countries.” CNEF 2000

Question Bank Secondary analysis of existing data: finding questions that enable you to produce a satisfactory scale Does not cover all cross-national data sources but good for UK data

ZA Online Study Catalogue (ZACAT) mhttp:// m Access to data and documentation of around 200 studies with a focus on international comparative studies and election studies

Additional Reading Bradshaw, J., Ditch, J., Holmes, H. and Whiteford, P., (1993): “A comparative study of child support in fifteen countries”, Journal of European Social Policy, 3 (4), Bruning, G. and Plantenga, J. (1999): “Parental leave and equal opportunities experiences in eight European countries”, Journal of European Social Policy, 9 (3), Gornick, J. C., Meyers M. K. and Ross, K. E. (1997): "Supporting the employment of mothers: Policy variation across fourteen welfare states", Journal of European Social Policy, 7, Gornick, J. C. and Meyers, M. K. (2003): Families That Work. Policies for Reconciling Parenthood and Employment, Russel Sage Foundation: New York. Smith, A. J. and Williams, D. (2007): "Father friendly legislation and paternal time across Western Europe", Journal of Comparative Policy Analysis, 9 (3),

Technical Reading De Vaus, D. 2002: Surveys in Social Research (5th Edition), Chapter 1: The Nature of Surveys; Chapter 4: Developing Indicators for Concepts; Chapter 12: Overview of analysis; Chapter 11: Building Scales (needs to be read after chapter 12).

Design An Index! Put together a short presentation about an index you might like to develop. Consider the points below. (Groups of 3: 15 minute presentation) 1.Identify the concept you want to measure in an index (e.g. child well-being, father-friendliness of policy, support for women’s employment, any other topic of interest to you) 2.Develop a nominal definition of this concept 3.Identify relevant dimensions of the concept (not more than 4 for today’s exercise) 4.Specify appropriate indicators for each dimension (operational definitions). Reflect upon the validity and reliability of the indicators. 5.Which data sets might you use? 6.How would you construct your index?