Measuring socio-economic impact of post-disaster shelter Developing a standardized methodology Shelter Meeting 09a, Geneva May 2009 Simone van Dijk Alexander van Leersum

Intro Measuring the impact of post-disaster shelter: short- or long-term? Status Quo: ◦Long term (i.e. impact) vs short-term evaluation Experience of case studies ◦Vietnam ◦Indonesia  Development of standardized methodology

Objectives of the presentation 1) Long term effects of housing 2) Aims of a standardized methodology 3) A proposed methodology ◦Before-after ◦With-without 4) Measuring socio-economic dimensions ◦Direct measurement ◦Indirect measurement (e.g. Wealth-index; PCA) ◦Self-assessment 5) Cross-programme comparison

1) Long term effects of housing What kind of a relation can be distinguished? Intertwined elements, problems with causalities and counterfactual analysis

2) Aims of a standardized methodology (1/2) What is the added value? Examples of expected gained insights/added values ◦Proportion of income spent on housing adaptations ◦How long does a HH uses a house, how do they use it? ◦Is the house cultural appropriate?

2) Aims of a standardized methodology (2/2) Developing a methodology for a long-term impact study that allows for comprehensive analysis of a household’s socio-economic situation. Developing a general approach and methodology for long- term socio-economic impact studies irrespective of location. Developing a general methodology that is suitable for the evaluation of different types of post-disaster housing programs. What do we want to achieve?

3) A proposed methodology (1/4) How to tackle causality questions? With-without (intervention vs control) ◦Selection of groups Before-after (impact of intervention) ◦Timing; project cycle

3) A proposed methodology (2/4) Before and after: The importance of timing Living standards Control group (i.e. non- beneficiaries) Impact of intervention Beneficiaries Time With-without & Before-after effect research Comparability of groups

3) A proposed methodology (3/4) With-without: selection of intervention- & control group(s) Control group, intervention group(s) Socio-economic position should be comparable between groups with respect to:  Average age of the household  Labor force  Average size of the household  Predominant sources of income  Educational level

3) A proposed methodology (4/4) Different (housing) interventions, e.g. in Ache Living standards Dec No-support Time With-without & Before-after effect research Permanent Housing (P.H.) Transitional Shelter (TS) Only T.S. T.S. + P.H. Impact of intervention?!

4) Measuring socio-economic dimensions What kind of data is required and feasible? Type of measurement DisciplineData processing Type of data Direct measurement Income & consumption Access to social services -- Indirect measurement Housing characteristics Durable assets In-house services Wealth index (Factor analysis) Objective Self-assessmentHousing Economic Living Standards Social standards 5-point scaleSubjective

4) Measuring socio-economic dimensions What kind of data is required and feasible? Type of measurement DisciplineData processing Type of data Direct measurement Income & consumption Access to social services -- Indirect measurement Housing characteristics Durable assets In-house services Wealth index (Factor analysis) Objective Self-assessmentHousing Economic Living Standards Social standards 5-point scaleSubjective What type of latrine does your household uses? Flush toilet (setting) [ ] 1 Toilet (standing) [ ] 2 Other, unhygienic latrine [ ] 3 No latrine [ ] 4 Do you have days of food deficits per week? No [ ] 1 Sometimes [ ] 2 Yes [ ] 3 The quality of your present house is... Very bad [ ] 1 Bad [ ] 2 Ok [ ] 3 Very good [ ] 4 Good [ ] 5

5) Cross-programme comparison (1/3) (How) can shelter programmes be compared among each other? Examples of questions which can make comparison possible  Only relative comparison is possible; comparison of different socio-economic status of groups E.g. Proportion of income spent on housing adaption/extension/repair Standardization of methodology Partly standardization of questions

5) Cross-programme comparison (2/3) Why a standardized methodology? Strength of method lies in the combination of ◦Focus: socio-economic dimensions related to housing ◦Approach: intervention + control group(s) ◦Timing: different moments in time, to measure impact ◦Data collection: objective and subjective ◦Data collection: quantitative and qualitative

5) Cross-programme comparison (3/3) What are the underlying aims of a standardized methodology? What can we expect from insights of impact studies? 1) Support from donors 2) Improvement of the program design 3) Improvement of the (shelter)design

Thank you

6) Extra Measurement of impact further explained Answer scale’s Measuring mean-difference between groups Statistically significant different? (e.g. T-test, ANOVA, Cross-tabs) 5% overlap (1 st order error, Alpha)  95% Confidence Interval Intervention Group Control Group H 0 =µ intervention ˃ µ control

6) Extra PCA further explained Data typeAnalyzed variables Actual variables to construct 'Wealth Index', based on Principle Component analysis Housingrooms/person, type of wall, floor, roof floor type Consumer Durablesradio, refrigerator, bicycle, TV, motorbike, car, ploughing machine/tractor, other agricultural equipment, mobile phone, fixed telephone, sewing machine, electric fan, rice cooker radio, bicycle, TV, electric fan, rice cooker Serviceswater, sanitation, electricity, fuel for cooking electricity, sanitation Wealth indexvalues ranging between 0 to 5 (most poor towards least poor)