Felix Naschold Cornell University & University of Wyoming Christopher B. Barrett Cornell University AAEA 27 July 2010 A stochastic dominance approach to program evaluation And an application to child nutritional status in arid and semi-arid Kenya

Motivation 1. Program Evaluation Methods By design they focus on mean. Ex: “average treatment effect” In practice often interested in distributional impact Limited possibility for doing this by splitting sample 2. Stochastic dominance By design look at entire distribution Now commonly used in snapshot welfare comparisons But not for program evaluation. Ex: “differences-in-differences” 3. This paper merges the two  Diff-in-Diff (DD) evaluation using stochastic dominance (SD) 2

Main Contributions of this paper 1. Proposes DD-based SD method for program evaluation 2. First application to evaluating welfare changes over time 3. Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya Unique, large dataset of 600,000+ observations collected by the Arid Lands Resource Management Project (ALRMP II) (one of) first to use Z-scores of Mid-upper arm circumference (MUAC) 3

Main Results 4 1. Methodology (relatively) straight-forward extension of SD to dynamic context: static SD results carry over Interpretation differs (as based on cdfs) Only up to second order SD 2. Empirical results Child malnutrition in Kenyan ASALs remains dire No average treatment effect of ALRMP expenditures Differential impact with fewer negative changes in treatment sublocations ALRMP a nutritional safety net?

Program evaluation (PE) methods 5 Fundamental problem of PE: want to but cannot observe a person’s outcomes in treatment and control state Solution 1: make treatment and control look the same (randomization) Gives average treatment effect Solution 2: compare changes across treatment and control (Difference-in-Difference) Gives average treatment effect:

New PE method based on SD 6 Objective: to look beyond the ‘average treatment effect’ Approach: SD compares entire distributions not just their summary statistics Two advantages 1. Circumvents (highly controversial) cut-off point. Examples: poverty line, MUAC Z-score cut-off 2. Unifies analysis for broad classes of welfare indicators

Definition of Stochastic Dominance 7 First order: A FOD B up to iff S th order: A s th order dominates B iff MUAC Z- score Cumulative % of population F A (x) F B (x) 0x max

SD and single differences 8 These SD dominance criteria Apply directly to single difference evaluation (across time OR across treatment and control groups) Do not directly apply to DD Literature to date: Single paper: Verme (2010) on single differences SD entirely absent from PE literature (e.g. Handbook of Development Economics)

Expanding SD to DD estimation - Method 9 Practical importance: evaluate beyond-mean effect in non- experimental data Let, G denote the set of probability density functions of Δ. and The respective cdfs of changes are G A ( Δ ) and G B ( Δ ) Then A FOD B iff A S th order dominates B iff

Expanding SD to DD estimation – 2 differences in interpretation Cut-off point in terms of changes not levels. Cdf orders changes from most negative to most positive  ‘poverty blind’ or ‘malnutrition blind’. (Partial) remedy: run on subset of ever-poor/always-poor 2. Interpretation of dominance orders FOD: differences in distributions of changes between intervention and control sublocations SOD: degree of concentration of these changes at lower end of distributions TOD: additional weight to lower end of distribution. Sense in doing this for welfare changes irrespective of absolute welfare?

Setting and data 11 Arid and Semi-arid district in Kenya Characterized by pastoralism Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds Data From Arid Lands Resource Management Project Phase II 28 districts, 128 sublocations, June 05- Aug 09, 600,000 obs. Welfare Indicator: MUAC Z-scores Severe amount of malnutrition: 10 percent of children have Z-scores below and percent of children have Z-scores below and -2.06

The pseudo panel used 12 Sublocation-specific pseudo panel 2005/ /09 Why pseudo-panel? 1. Inconsistent child identifiers 2. MUAC data not available for all children in all months 3. Graduation out of and birth into the sample How? 14 summary statistics – mean & percentiles and ‘poverty measures’ Focus on malnourished children Thus, present analysis median MUAC Z-score of children below 0 Control and intervention according to project investment

Results: DD Regression 13 Pseudo panel regression model No statistically significant average program impact

Results – DD regression panel 14 (1)(2)(3)(4)(5) VARIABLES median of MUAC Z <0 10th percentile 25th percentile median of MUAC Z <-1 median of MUAC Z <-2 intervention dummy based on ALRMP investment (0.248)(0.316)(0.371)(0.188)(0.155) change in NDVI 2005/06-08/ *2.611***2.058***0.927*0.768* (0.0545)( )( )(0.0997)(0.0767) squared change in NDVI 2005/06-08/ ** * (0.0293)(0.136)(0.0510)(0.802)(0.479) Constant0.501***0.892***0.839***0.203***0.120*** (2.99e-07)(1.40e-08)(8.70e-09)( )( ) Observations R-squared Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included.

Stochastic Dominance Results 15 Three steps: Steps 1 & 2: Simple differences SD within control and treatment over time: no difference in trends. Both improved slightly SD control vs. treatment at beginning and at end: control sublocations dominate in most cases, intervention never Step 3: SD on DD (results focus for today)

16

17

18

Conclusions 19 Existing program evaluation approaches  average treatment effect This paper: new SD-based method to evaluate impact across entire distribution for non-experimental data Results show practical importance of looking beyond averages Standard DD regressions: no impact at the mean SD DD: intervention sublocations had fewer negative observations ALRMP II may have functioned as nutritional safety net (though only correlation, no way to get at causality)

Thank you. 20

Expanding SD to DD estimation – controlling for covariates 21 In regression DD: simply add (linear) controls In SD-DD need a two step method 1. Regress outcome variable on covariates 2. Use residuals (the unexplained variation) in SD DD In application below first stage controls for drought (NDVI)

SD, poverty & social welfare orderings (1) SD and Poverty orderings Let SD s denote stochastic dominance of order s and P α stand for poverty ordering (‘has less poverty’) Let α=s-1 Then A P α B iff A SD s B SD and Poverty orderings are nested A SD 1 B  A SD 2 B  A SD 3 B A P 1 B  A P 2 B  A P 3 B

SD, poverty & social welfare orderings (2) Poverty and Welfare orderings (Foster and Shorrocks 1988) Let U(F) be the class of symmetric utilitarian welfare functions Then A P α B iff A U α B Examples: U 1 represents the monotonic utilitarian welfare functions such that u’>0. Less malnutrition is better, regardless for whom. U 2 represents equality preference welfare functions such that u’’<0. A mean preserving progressive transfer increases U 2. U 3 represents transfer sensitive social welfare functions such that u’’’>0. A transfer is valued more lower in the distribution Bottomline: For welfare levels tests up to third order make sense

The data (2) – extent of malnutrition 24

DD Regression 2 25 Individual MUAC Z-score regression To test program impact with much larger data set Still no statistically significant average program impact

Results – DD regression indiv data 26 Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included. Dependent variable: Individual MUAC Z-score VARIABLES time dummy (=1 for 2008/09) (0.290) control - intervention by investment (0.425) Diff in diff (0.782) Normalized Difference Vegetation Index1.029*** (6.25e-07) Constant-1.391*** (0) Observations R-squared0.033

Full table of SD results 27

28