UKES Annual Conference Climate adaptation and resilience impact assessments: Identifying ways to improve usefulness and feasibility UKES Annual Conference 11th May 2017

Context Growing interest in knowing what impact climate adaptation and resilience building interventions have But challenging to conduct impact assessments that are useful, practical and ultimately good value for money Objective: Identify ways of doing this using experiences from the Malawian Enhancing Climate Resilience Programme

Overview Types of resilience and adaptation interventions Challenges of doing impact assessment Potential ways of addressing the challenges: missing baselines and making assessments useful

Types of interventions Resilience interventions includes anticipatory, absorptive and adaptive approaches Examples of interventions (focussed on Household and community levels): Agricultural practices and technologies Weather information, early warning systems Credit access: insurance, micro finance Irrigation supply

Feasibility and usefulness challenges Climate shock Intervention Welfare Impact Flooding - The disaster is wide-spread (a significant drought, very large flood or storm etc.). - Your project is only being rolled out in a portion of the area affected by the disaster, or has a phased geographic rollout. - The disaster is relatively predictable (based on annual weather patterns), so a counterfactual can be planned for. A suitable comparison region can be identified within the affected area, with similar environmental and/or demographic profile. - Disaster preparedness and response projects are often not suitable for a quasi-experimental approach to counterfactual analysis because: Any disaster preparedness or response project will naturally aim to include all vulnerable groups in its delivery, making it difficult (or possibly unethical) to exclude certain groups for the sake of comparison. - Disasters are often localised, making it hard to compare across regions. Even when a disaster strikes a large region, there will always be areas that are closer/further from the epicentre, making it difficult to find affected comparison groups outside of your treatment group. - The extent of a disaster’s impacts on people or infrastructure is specific to the unique characteristics of that location (for example their level of vulnerability) adding further characteristics that have to be accounted for when selecting a comparison region. Baseline End-line Time Drought exposure

Using existing data: missing baselines Sensitivity Exposure Adaptive capacity Interviews with project staff about their targeting process Malawi hazards and vulnerability mapping tool: http://tools.rcmrd.org/vulnerabilitytool/

Using recall: missing baselines All survey data is recall: matter of degree Using the baseline data to cross-check recall responses Being selective about which things you ask people to recall Ask them to recall well-known events that coincide with the baseline date of interest Bamberger 2010. Reconstructing Baseline Data for Impact Evaluation and Results Management. The World Bank Report 4.

Bound analysis: comparison group issues Potential impact: increased food security -20% 40% 80% 80% Households secure 40% Households Food secure Endline 40% Households Food secure 50% Households Food secure Baseline Intervention area Non-intervention area Manski (2013) Public policy in an uncertain world Manski (2007) Identification for Prediction and Decision

Combining counterfactual with theory-based approaches: improving usefulness Judea Pearl (2009) Causality: Models, reasoning and inference

Other ways to improve usefulness and feasibility Asking project stakeholders what intervention impacts they are interested in (used to develop a protocol) Power tests informed by pilot study data Ensuring that that cohorts of interest for the second phase are sufficiently powered Mitchell, S. et al. 2016. Causal Inference with Small Samples and Incomplete Baseline for the Millennium Villages Project. Working paper.

Questions and discussion points Tracking long-term impacts What to do when no non-beneficiaries How to use big data (mobile phones, social media) Multiple overlapping interventions Spillovers Small sample sizes (synthetic controls) Expert or modelling informed counterfactuals Other theory-based impact assessment methods that could be applied