How Shocking! The Challenges of Shocks, Variability, and Resilience in Evaluating Impact in Adaptation Projects Marc D Shapiro, Ph.D. Project Leader, Global Climate Change Monitoring and Evaluation Project Development and Training Services (dTS)

Definitions of resilience “The ability of countries, communities and households to anticipate, adapt to and /or recover from the effects of potentially hazardous occurrences … in a manner that protects livelihoods, accelerates and sustains recovery, and supports economic and social development” –Working Group on Resilience (Feb 2013) “…the ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth.” –United States Agency for International Development

Resilience in Adapting to Climate Change Stressors Common theme across definitions: Anticipate/mitigate, adapt to, and recover from stress and shocks Should definition differ in adapting to climate change (CC)? No, though different probabilities/frequencies or magnitude of stressors may imply differences in surveying Food security examples –No CC: typical annual/seasonal variability or cycles –W/ CC: shift in average, wider variability, different timing Disaster risk reduction examples –No CC: infrequent flooding/drought –W/ CC: more frequent, more severe/longer, different timing 3

Challenges in Measuring Project-affected Resilience in Adapting to Climate Change How determine BAU counterfactual accounting for change in climate means and deviations Lack of localized (down-scaled) climate modeling predictions for site-specific counterfactual predictions Longer timelines for CC-related outcomes to occur; shorter contract timelines Where change already evident; behavioral change may already have begun (attenuation & larger samples) Key outcomes may be infrequent events Difficulty in finding/assuring geographical-equivalence for controls or comparisons 4

Challenges in Measuring Resilience in Adapting to Climate Change CC adaptation interventions frequently integrated within suite of other good practices Interventions smaller than other exogenous factors Unexpected outcomes (positive and negative) may be even less expected than usual Many interventions focus on enabling environment –External validity, esp. of ministerial-level interventions Confusion about adaptation to climate variability (e.g., disaster response) vs climate change (prevention / changed behavior) 5

No Drought in Ideas of what to Measure Shocks/stresses Contextual indicators Absorptive capacity: ability to manage risk; ability to recover quickly, Adaptive capacity: ability to make informed proactive choices about alternative livelihood strategies Institutional capacity: presence of appropriate systems, enabling conditions that support ability to move beyond chronic vulnerability 6

What to Measure Well-being outcomes –End of project unlikely to be right timing –Modeling predicted outcomes probabilistically (as information increases on prior expectations) Monte carlo for range Expectation of decay rate over time as proxy for turnover 7

What to Measure when you Can’t Wait? (Intermediate Results/Leading Indicators) Absorptive capacity: Latent change in capacity, practice, or response as outcome –Knowledge and attitudes about changes –Awareness of CC-related information, systems etc. Institutional capacity: –Analysis of systems for potential for sustainability –Simulations (Pre- mid-, post-) e.g., simulated response to disasters (beware of testing effects) 8

What to Measure when you Can’t Wait? (Intermediate Results/Leading Indicators) Behavioral change in practice or response –Mid-term measurement easier when behavior frequently repeated (e.g., seasonal planting; seasonal droughts that lengthen) –Challenging when behavior highly infrequent, conditional on major event/disaster, decrease in probability over time Measure directional proxies for resilience: (E.g., change in social capital, household resources, network ties) 9

What to Measure Cross-indicator strategies –Longer time series data collections to use variability across years or locations (high/low stresses) as proxy for shocks –Organize longer data collection timeline and contracts Time endline for irregular event (using early warning / seasonal forecasts)? Heterogeneity –To understand heterogeneity in vulnerability (rather than average response) means need for large sample sizes / additional focus groups and nuanced examination of data to understand variable types of responses and adaptive strategies 10

More evidence that “unexpected” (possibly CC-related) disasters could touch anyone: Guess location 11

Answer 12 Speaker’s family!