Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation Alison Buttenheim, Princeton University Howard White, 3ie Rizwana Siddiqui, PIDE Katie Hsih, Princeton University April 1, 2009 Cairo

2 3ie post-disaster impact evaluation (PDIE) study Motivation: Frequency and severity of natural disasters Quantity of assistance provided in post- disaster settings Recent interest from humanitarian and development sectors in more and better impact evaluation Opportunity to use Pakistan ERRA experience as case study

3 3ie post-disaster impact evaluation (PDIE) study Goals: Review existing approaches to PDIE Develop a framework for rigorous PDIE Apply framework to the 2005 Pakistan earthquake case Identify a set of principles to guide PDIE

4 Disasters Natural events: 414 reported in 2007 (CRED criteria)

5 Disasters Natural events: 414 reported in 2007 (CRED criteria) Human consequences: 211 million affected 16,847 lives lost USD 100+ billion damages

6 Disasters Natural events: 414 reported in 2007 (CRED criteria) Human consequences: 211 million affected 16,847 lives lost USD 100+ billion damages Institutional responses

7 Post-disaster relief and recovery efforts USD 5.9 billion (pledged) for 2005 Pakistan earthquake USD 13.5 billion (pledged) for 2004 Indian Ocean tsunami Actors: Diverse mix of governments, funders, IFIs, aid agencies, humanitarian agencies, int’l/local NGOs.

8 How does PD assistance get evaluated? Extensive process evaluations Multiple levels of analysis (project, agency, sector, disaster) Some joint evaluations (e.g. TEC) Review of ALNAP database, etc. suggests few examples of “rigorous” impact evaluation

9 Why so little focus on IE in PD settings?

10 Why so little focus on IE in PD settings? “Disasters are different”

11 Are disasters any different? 1.Unpredictable, rapid-onset event 2.Proven life-saving measures cannot be randomized or withheld 3.Mismatch between resources and need (sometimes) 4.Absence of baseline data (usually) 5.Which counterfactual is the right one?

12 Are disasters any different? Maybe not… 1.Nonrandom exposure to disaster event and consequences 2.Nonrandom assignment of interventions 3.Fragile states/vulnerable populations 4.Multiple concurrent interventions 5.Which counterfactual is the right one?

13 Bangladesh floods, 1998 Hurricane Mitch, 1998 Indian Ocean tsunami, 2004 Hurricane Katrina, 2005 Lessons learned from other PDIE experiences

14 Disaster-related time periods 14 Pre-disasterImmediate post- disaster Post- intervention (1) Post- intervention (2) Emergency Relief Recovery/Reconstruction t -1 t0t0 t1t1 t2t2 DISASTER

15 Disaster-related populations 15 A Disaster-affected households* that receive assistance or interventions B Disaster-affected households that do not receive assistance or interventions ‡ C Non-affected households † that were similar to A before the disaster * or communities (or other unit of analysis) ‡ or receive them later, or receive different ones † or less-affected households/communities

16 TimeDescriptionDisaster-affected households, treated t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline A 0 -A -1 A 1 -A -1 t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery A 1 -A 0 A 2 -A -1 A 2 -A 0 A 2 -A 1 Within treatment group, single-difference over time

17 TimeDescriptionDisaster-affected households, treated t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline A 0 -A -1 A 1 -A -1 t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery A 1 -A 0 A 2 -A -1 A 2 -A 0 A 2 -A 1 Within treatment group, single-difference over time ERRA: “Build Back Better”

18 TimeDescriptionDisaster-affected households, treated t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline A 0 -A -1 A 1 -A -1 t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery A 1 -A 0 A 2 -A -1 A 2 -A 0 A 2 -A 1 Within treatment group, single-difference over time Problems: Recall bias if no baseline; attribution?

19 – ––– TimeDescriptionAffected treated- Non-affected t -1 Baseline (pre-disaster)A -1 -C - 1 t0t0 Emergency (immediate post-disaster)A 0 -C 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -C 1 t2t2 Recovery (post-intervention #2)A 2 -C 2 Cross-sectional, single-difference over treatment groups (A vs. C)

20 – ––––– TimeDescriptionAffected treated- Non-affected t -1 Baseline (pre-disaster)A -1 -C - 1 t0t0 Emergency (immediate post-disaster)A 0 -C 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -C 1 t2t2 Recovery (post-intervention #2)A 2 -C 2 Cross-sectional, single-difference over treatment groups (A vs. C)

21 – ––––– TimeDescriptionAffected treated- Non-affected t -1 Baseline (pre-disaster)A -1 -C - 1 t0t0 Emergency (immediate post-disaster)A 0 -C 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -C 1 t2t2 Recovery (post-intervention #2)A 2 -C 2 Cross-sectional, single-difference over treatment groups (A vs. C) Implied counterfactual: What would “A” households look like if there had been no disaster?

22 – ––––– TimeDescriptionAffected treated- Non-affected t -1 Baseline (pre-disaster)A -1 -C - 1 t0t0 Emergency (immediate post-disaster)A 0 -C 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -C 1 t2t2 Recovery (post-intervention #2)A 2 -C 2 Cross-sectional, single-difference over treatment groups (A vs. C) Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?

23 TimeDescription Affected — Unaffected t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline (A 0 -A - 1 ) (A 1 -A - 1 ) — (C 0 -C -1 ) — (C 1 -C -1 ) t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery (A 1 -A 0 ) (A 2 -A - 1 ) (A 2 -A 0 ) (A 2 -A 1 ) — (C 1 -C 0 ) — (C 2 -C -1 ) — (C 2 -C 0 ) — (C 2 -C 1 ) Difference-in-difference (A vs. C)

24 TimeDescription Affected — Unaffected t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline (A 0 -A - 1 ) (A 1 -A - 1 ) — (C 0 -C -1 ) — (C 1 -C -1 ) t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery (A 1 -A 0 ) (A 2 -A - 1 ) (A 2 -A 0 ) (A 2 -A 1 ) — (C 1 -C 0 ) — (C 2 -C -1 ) — (C 2 -C 0 ) — (C 2 -C 1 ) Difference-in-difference (A vs. C) Controls time-variant factors that are the same between A & C

25 – ––––– TimeDescriptionAffected treated- Affected control t -1 Baseline (pre-disaster)A -1 -B - 1 t0t0 Emergency (immediate post-disaster)A 0 -B 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -B 1 t2t2 Recovery (post-intervention #2)A 2 -B 2 Cross-sectional, single-difference over treatment groups (A vs. B)

26 – ––––– TimeDescriptionAffected treated- Affected control t -1 Baseline (pre-disaster)A -1 -B - 1 t0t0 Emergency (immediate post-disaster)A 0 -B 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -B 1 t2t2 Recovery (post-intervention #2)A 2 -B 2 Cross-sectional, single-difference over treatment groups (A vs. B)

27 – ––––– TimeDescriptionAffected treated- Affected control t -1 Baseline (pre-disaster)A -1 -B - 1 t0t0 Emergency (immediate post-disaster)A 0 -B 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -B 1 t2t2 Recovery (post-intervention #2)A 2 -B 2 Cross-sectional, single-difference over treatment groups (A vs. B) Implied counterfactual: What would “A” households look like if there had been no intervention?

28 – ––––– TimeDescriptionAffected treated- Affected control t -1 Baseline (pre-disaster)A -1 -B - 1 t0t0 Emergency (immediate post-disaster)A 0 -B 0 t1t1 Relief/Reconstruction (post-intervention #1)A 1 -B 1 t2t2 Recovery (post-intervention #2)A 2 -B 2 Cross-sectional, single-difference over treatment groups (A vs. B) Problems: How were interventions assigned to A but not to B?

29 TimeDescriptionDisaster-affected households “Treated”— “Control” t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline (A 0 -A - 1 ) (A 1 -A - 1 ) — (B 0 -B -1 ) — (B 1 -B -1 ) t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery (A 1 -A 0 ) (A 2 -A - 1 ) (A 2 -A 0 ) (A 2 -A 1 ) — (B 1 -B 0 ) — (B 2 -B -1 ) — (B 2 -B 0 ) — (B 2 -B 1 ) Difference-in-difference (A vs. B)

30 TimeDescriptionDisaster-affected households “Treated”— “Control” t 0 -t -1 t 1 -t -1 Disaster-related losses Restoration to baseline (A 0 -A - 1 ) (A 1 -A - 1 ) — (B 0 -B -1 ) — (B 1 -B -1 ) t 1 -t 0 t 2 -t -1 t 2 -t 0 t 2 -t 1 Recovery from disaster Sustained restoration to baseline Sustained recovery from disaster Persistence of recovery (A 1 -A 0 ) (A 2 -A - 1 ) (A 2 -A 0 ) (A 2 -A 1 ) — (B 1 -B 0 ) — (B 2 -B -1 ) — (B 2 -B 0 ) — (B 2 -B 1 ) Difference-in-difference (A vs. B) Controls time-variant factors that are the same between A & B

31 World Bank impact evaluation of housing and livelihood grants

32 Instrumental variable approach to disaster impact: –Villages at same distance from epicenter, at same elevation and slope had comparable pre- disaster SES –Villages at different distance from fault line experienced different earthquake severity. World Bank impact evaluation of housing and livelihood grants

33 Instrumental variable approach to disaster impact: –Villages at same distance from epicenter, at same elevation and slope had comparable pre- disaster SES –Villages at different distance from fault line experienced different earthquake severity. World Bank impact evaluation of housing and livelihood grants A 1 -C 1

34 Instrumental variable approach to disaster impact: –Villages at same distance from epicenter, at same elevation and slope had comparable pre- disaster SES –Villages at different distance from fault line experienced different earthquake severity. Variation in receipt of relief and recovery funds: –Between-district variation in implementing agency for housing grant –Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design. World Bank impact evaluation of housing and livelihood grants A 1 -C 1

35 Instrumental variable approach to disaster impact: –Villages at same distance from epicenter, at same elevation and slope had comparable pre- disaster SES –Villages at different distance from fault line experienced different earthquake severity. Variation in receipt of relief and recovery funds: –Between-district variation in implementing agency for housing grant –Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design. World Bank impact evaluation of housing and livelihood grants A 1 -C 1 A 1 -B 1

36 ERRA impact evaluation case study 1.Evaluation opportunities using existing data & HH sample –Household data collection at t 2 –Retrospective household reports of t 0 –Use of ongoing government household surveys (e.g., HIES) as baseline –Randomization of some interventions from 2009 

37 ERRA impact evaluation case study 2.Evaluation opportunities in a future disaster –Maintain surveillance sample in disaster-prone regions –Household-level data collection at t 0 –Randomized interventions, e.g, Timing of interventions: –Group 1: Housing grant first, followed by livelihood cash grant –Group 2: Livelihood cash grant first, followed by housing grant Conditionality of grants Types of interventions, e.g, different formats or recipients of livelihoods cash grant

38 PDIE Guiding Principles 1.PDIE is necessary to ensure that relief and recovery funds are appropriately targeted, effective, and efficient. 2.Each phase of a disaster (emergency, relief, recovery/reconstruction) presents distinct evaluation challenges and therefore may require a different evaluation approach or methodology. 3.“Evaluation preparedness” is an important part of disaster preparedness.

39 PDIE Guiding Principles 4.PDIE should incorporate evaluation of (pre-disaster) investments in disaster mitigation, prevention, and resilience. 5.Rigorous PDIE requires the tools and perspectives of multiple disciplines and sectors. 6.Quantitative PDIE can benefit from the qualitative and mixed-methods approaches.

40 PDIE Guiding Principles 7.Proportionate changes in outcomes over time and over groups can be as instructive as changes in levels. 8.Change-over-time impact evaluations should recognize two distinct baselines: pre-disaster, and immediately post-disaster.

41 PDIE Guiding Principles (ct’d) 9.PDIE will be most successful when the goals of the intervention are clearly defined through a logical framework or similar model; when the interventions are appropriately targeted, and when the purpose/use of the evaluation is clear. 10.Experimental and quasi-experimental approaches are feasible in PDIE if ethical, logistical and “fit” issues are adequately addressed.