1. 1 Surveys: Collecting Policy Relevant Data Rachel Govoni-Smith Kinnon Scott, DECRG January 17, 2006

2. 2 Sources- The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, eds. Francois Bourguignon and Luiz Al Pereira da Silva, World Bank, Washington, D.C., 2003. –Scott, Kinnon (2003) “Generating Relevant Household Level Data: Multi-topic Household Surveys” Muñoz, Juan and Kinnon Scott (2005) “Household Surveys and the Millennium Development Goals”, report for Paris21 Task Force on Improved Statistical Support for Monitoring Development Goals

3. 3 Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution Estimating Incidence of Indirect Taxes Analyzing the Incidence of Public Spending Behavioral Incidence Analysis of Public Spending Estimating Geographically Disaggregated Welfare Levels and Changes Assessing the Poverty Impact of an Assigned Program Ex Ante Evaluation of Policy Reforms Micro Level

4. 4 Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution The Effect of Aggregate Growth on Poverty Linking Macro-consistency Models to Household Surveys Partial Equilibrium; Multi-market Analysis The 123PRSP Model Social Accounting Matrices Poverty and Inequality Analysis and CGE models Macro Level

5. 5 Goals and Needs Goals: Measure the poverty impact of economic policy Measure the distributional impact of economic policy Needs: Rely heavily on household survey data

6. 6 Household Surveys Single Topic In-between Multi-topic

7. 7 Household Surveys Single Topic Labor Force Surveys( LFS) (ILO) Housing Surveys Census – national, UNFPA, 10 years In-between Multi-topic

8. 8 Household Surveys Single Topic In-between Agricultural Surveys (FAO) Demographic and Health (DHS) Household Budget Surveys (HBS) Multi-topic

9. 9 Household Surveys Single Topic In-between Multi-topic Multiple Indicator Cluster Surveys (MICS, UNICEF) Survey on Income and Living Conditions (SILC, EU) Core Welfare Indicator Surveys (CWIQ, WB) Living Standards Measurement Study Surveys (LSMS) and Integrated Surveys (IS) (WB) Family Life Surveys (FLS, RAND)

10. 10 What type of household data? Poverty measure: per capita or per adult equivalent consumption Government programs  receipt, format, costs (formal and informal), use level Consumption of taxed goods Labor market participation (sector, hours, earnings) Income by sources

11. 11 Census Accurate measure of the population of a country Geographic distribution of the population Basic demographic information Purpose

12. 12 Census Not a sample Universal coverage No sampling errors in estimates Some corrections for non-response may be needed Sample

13. 13 Census Short Trade-off between coverage and content Two types of errors: sampling and non- sampling Content

14. Sample size Sampling error Non-sampling error Sampling vs. non-sampling errors Total error

15. 15 Census Short Trade-off between coverage and content Two types of errors: sampling and non- sampling Content Cost Time Non-response Training

16. 16 Census Demographic information: age, sex, race/ethnicity, family and household composition Housing information Others: basic education, labor, disability Content

17. 17 Census Basic needs –Subjective –Limited monitoring use Poverty Measurement Income: Panama example Albania: 2001 (1989) BiH 1991 (1981) Montenegro 2003 (1991) – Limited use if looking at impact of policies affecting taxes, tariffs or pricing

18. 18 Census Sample frame Link with household surveys for small area estimation Uses

19. 19 Poverty Indicator by Commun, Albania, 200

20. 20 Labor Force Survey Direct measurement of unemployment General characteristics of the labor force Purpose

21. 21 Labor Force Surveys Relatively large samples  Need for precise estimates (change)  Desire to disaggregate to different geographic areas Individuals of working age Sample

22. 22 Labor Force Survey Characteristics of the labor force –Demographics –Education Sectoral distribution of employment Degree of formality Seasonal Income Content

23. 23 Labor Force Survey Three problems: LFS typically capture partial, not total, income –Under-estimate welfare (vs. NA) –Mis-ranking of households by welfare level Poverty Measurement

24. 24 Venezuela: Income and Expend Survey

25. 25 Venezuela: Social Survey

26. 26 Labor Force Surveys, cont. Three problems: LFS typically capture partial, not total, income Measurement Error –Labor income measurement error –At both ends of the distribution Poverty Measurement

27. 27 LFS in Latin America Item non-response SalariedSelf- employed EmployerAll Indep- endent Mean non- response rate 3.9%10.2%12.010.6% Source: Feres, 1998

28. 28 Labor Force Surveys, cont. Three problems: Partial vs total, income Measurement error Income vs consumption measure –Potential vs actual welfare –Smoothing –Measurement Error Poverty Measurement

29. 29 Household Budget Surveys Inputs to national accounts on consumer expenditures Track changes in expenditures over time Track changes in the relative share of different expenditures Weights for the consumer price index Purpose

30. 30 Household Budget Surveys Medium size sample Sampling errors high at disaggregated level High non-response rates In some parts: only urban (capital city or group of large cities) Sample Non response rates ( Eurostat, 2003) Bulgaria: 39.7% Estonia, 44% Hungary, 58.8% before replacement Romania, 21.6 %

31. 31 Household Budget Surveys Total Income Total Consumption Short Demographics In FSU and Central Europe: agriculture Content

32. 32 Household Budget Surveys Possible to construct both total consumption and total income Income may suffer from same measurement errors as LFS Poverty Measurement

33. 33 Household Budget Surveys Consumption based welfare measure Purpose of an HBS survey is NOT to measure welfare but to precisely measure mean expenditures on specific goods and services These are conflicting goals Poverty Measurement

34. 34 Household Budget Surveys Shortest possible reference periods Minimize number of omitted expenditures Good for precise measurement of regional or national means Because of lumpy nature of purchases, not good for comparisons among households Need to adjust (lengthen) the reference periods used in HBS Poverty Measurement

35. 35 Household Budget Surveys Focus on expenditures –Not all expenditures are consumption –Only purchases of durable goods and housing Durable goods: list of items owned by household, age of items, current value Housing: housing characteristics affecting value Poverty Measurement

36. 36 Household Budget Surveys Good for taxation issues Good for public (and private) transfers Sometimes has basic labor FSU and Central European countries: agriculture No health, education data Limited for other areas Uses

37. 37 Multi-topic Household Surveys Those with a focus on measuring poverty National Socio-Economic Survey of Indonesia, SUSENAS Survey on Income and Living Conditions (SILC) Rand Family Life Surveys (FLS) Living Standards Measurement Study Surveys (LSMS)

38. 38 Multi-topic Household Surveys Analysis of welfare levels and distribution Study links between welfare levels and individual and household characteristics, economic, human and social capital Social exclusion Causes of observed social outcomes Levels of access to, and use of, social services, government programs and spending Purpose

39. 39 Multi-topic Household Surveys Small sample sizes Trade-off issue: Quality and cost considerations Limits ability to assess programs or policies that affect small groups or small areas (over- sample) Infrequent in many countries (exceptions, inter alia, Indonesia, Panama, Jamaica, Peru, Ghana) Sample

40. 40 Multi-topic Household Surveys Content

41. 41 Multi-topic Household Surveys Total consumption –Longer reference periods –Able to calculate use value of durables and housing Total income –Suffers from standard measurement errors Poverty Measurement

42. 42 Multi-topic Household Surveys Poverty levels and distribution Social exclusion Public and private transfers Incidence analysis Tax policy Labor markets Education, health, social protections Changes in relative prices Monitoring (PRSP, MDGs), impact evaluation Uses

43. 43 Cross Section or Panel Surveys? Substantive applications Methodological issues

44. 44 Panels 1.Why do we need longitudinal data? 2.Designs for surveys across time 3.Advantages and uses of panels 4.Methodological issues

45. 45 Understanding change  Longitudinal data are needed to understand the process of change, transitions between states, and the factors or events that are associated with those transitions  ‘Longitudinal’ data is a catch-all phrase for a wide range of different types of studies

46. 46 Designs for surveys across time Repeated cross sectional surveys (e.g. Household Budget Survey, Labour Force Survey) Common design for large government surveys New sample drawn for each survey Carry similar questions each year Used for trend analysis at aggregate level

47. 47 Designs for surveys across time Cohort Studies Sample often based on an age group Follow up same sample members at fairly long intervals Developmental data as well as social and economic data Data from parents, teachers associated with cohort member

48. 48 Designs for surveys across time Rotating Panel Survey Survey of Income and Programme Participation, USA (SIPP) Respondents stay in the panel for a set period of time and are rotated out systematically and replaced by new sample members. Used where the interviews are fairly close together (every 3 to 6 months) and respondent burden is high. Used where the collection of short spells e.g. a few weeks unemployed or in receipt of a particular benefit, is critical.

49. 49 Designs for surveys across time Indefinite Life Panel Surveys e.g. Panel Study of Income Dynamics, USA – since 1968! Living in BiH, LSMS Albania, LSMS Serbia Draw a sample at one point in time and follow those sample members indefinitely (or as long as the funding continues) Collect individual level data in household context Repeated measures at fixed intervals ( annual data collection)

50. 50 Panels from conference attendee countries Albania – 4 waves 2002 - 2005 BiH – 4 waves 2001 - 2004 Serbia – 2 waves 2002 - 2003

51. 51 Advantages of Panel Data Comparison of same individual over time - outcomes Track of aspects of social change Facilitates study of change and causal inference Minimise the problem of inaccurate recall Compare a person’s expectations with real change Look at how changes in individuals’ behaviour affects their households  Identifies the co-variates of change and the relative risks of particular events for different types of people

52. 52 Changes in Employment Status A: CROSS-SECTIONAL INFORMATION Unemployed Employed 20012007 Net change - 0.1% unemployed

53. 53 Changes in Employment Status B: PANEL INFORMATION Still Unemployed Still Employed Unemployed Employed 20012007 Net change - 0.1% unemployedActual change is 10.1 continuously employed 86.7% employed 2001 but unemployed 2007 5% continuously unemployed 3.2% unemployed 2001 but employed 2007 5.1%

54. 54 Balkan Examples Albania - 15% of the unemployed in 2002 had made the transition to formal sector employment by 2004 BiH - About half who were poor in 2001 remained poor in 2004. Many individuals moved out of poverty. (Cross section headcount 18% for both years)

55. 55  Employment and the labour market  Unemployment duration and exit rates  Do the unemployed find stable employment?  The effect of non-standard employment on mental health  Temporary jobs: who gets them, what are they worth, and do they lead anywhere?  Family and Household  Patterns of household formation and dissolution  Breaking up - finances and well-being following divorce or split  The effect of parents’ employment on children's educational attainment

56. 56 Panel analysis Mobility, poverty and well-being among the informally employed – Peter Sanfey European Bank for Reconstruction and Development The origins of self employment, Leora Klapper et al, WB (soon to use Albania Panel also) The impact of health shocks on employment, earnings and household consumption, Kinnon Scott et al

57. 57 A Sample Concept of ‘longitudinal household’ problematic for a panel - households change in composition over time or disappear altogether Individual level sample

58. 58 Following rules All members of households interviewed at Wave One Children born to these original sample members Original members are followed as they move house, and any new individuals who join with them are eligible to be interviewed New sample members are followed if they split from the original member

59. 59 Questionnaire design Core content carried every wave Rotating core questions One-off variable components –lifetime job history –marital and fertility history Variable questions to respond to new research and policy agendas

60. 60 Attrition in panel surveys Inevitable to some extent but can be minimised Multiple sources of attrition in a panel –refusal to take part –respondents move and cannot be traced –non-contacts Worry is potential bias if people who drop out differ significantly from those who stay in

61. 61 UK Panel Wave 1 Respondents Wave-on wave re-interview rates

62. 62 Fieldwork respondent incentives as a ‘thank-you’ extended fieldwork period for ‘tail-enders’ refusal conversion programme tracking procedures during fieldwork panel maintenance between waves –Change of Address cards to update addresses –mailing of Respondent Report –details of contacts with respondents between waves

63. 63 Post-field checking and cleaning Within wave consistency Cross wave consistency and longitudinal integrity Sample management –individuals within households correctly identified across time –issuing of sample for each wave

64. 64 The user database Longitudinal data is complex Provide users with database structure which enhances usability Consistent record structure over time Key variables for matching and linking data cross wave Consistent variable naming conventions

65. 65 Added value ‘Added value’ to data set Extensive set of derived variables Production of weights –household and individual levels –cross sectional and longitudinal Imputation of missing data Flags to indicate imputed values

66. 66 Conclusions Longitudinal panel data allows us to answer research questions that cannot be answered with with cross-sectional data Provides a different view of the world - see process through the life-course not just a static picture Is complex (but so is the real world) - so needs to be well designed and conducted with sufficient resources to be successful

67. 67 System of Household Surveys GOAL: System able to respond to evolving needs: not produce data X or survey Y –Determine data needs before they are URGENT –Identify appropriate instruments, –Implement them properly, timely fashion, –Analyze the resulting data

68. 68 Improving the SHS Linking Users and Producers Providing adequate resources Continuous Survey Program –Not necessarily permanent survey –Benefits Avoid loss of capacity Create greater levels of capacity (building on existing) Economies of scale Policy makers know when data will be available Protects NSO from pressures for ad hoc surveys Ongoing system actually allows more flexibility and responsiveness

69. 69 Final points Welfare: household surveys- always missing the homeless, street children, institutionalized population No one survey can meet all needs, review its purpose, coverage, content and quality before using Need a system of surveys that meets the needs of data users