1. A stochastic dominance approach to program evaluation
And an application to child nutritional status in arid and semi-arid Kenya
A theoretical paper: program evaluation VS. stochastic dominance
Empirical application: ALRMP evaluation project
AAEA paper  comments, please
Felix Naschold University of Wyoming
Christopher B. Barrett Cornell University
May 2012 seminar presentation
University of Sydney

2. Motivation Program Evaluation Methods Stochastic dominance
By design they focus on mean
Ex: “average treatment effect” (ATE)
In practice, often interested in broader distributional impact
Limited possibility for doing this by splitting sample
Stochastic dominance
By design, look at entire distribution
Now commonly used in snapshot welfare comparisons
But not for program evaluation. Ex: “differences-in-differences”
This paper merges the two  Diff-in-Diff (DD) evaluation using stochastic dominance (SD) to compare changes in distributions over time between intervention and control populations
As much as this is true for many parts of the worlds in terms of MDGs, also for villages in rural India
What do we do about this as researchers?
One aspect: who moves in and out of poverty? Characterstics and targeting?

3. Main Contributions Proposes DD-based SD method for program evaluation
First application to evaluating welfare changes over time
Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya
Unique, large dataset of 600,000+ observations collected by the Arid Lands Resource Management Project (ALRMP II) in Kenya
(One of) first to use Z-scores of Mid-upper arm circumference (MUAC)

4. Main Results Methodology Empirical results
(relatively) straight-forward extension of SD to dynamic context: static SD results carry over
Interpretation differs (as based on cdfs)
Only feasible up to second order SD
Empirical results
Child malnutrition in Kenyan ASALs remains dire
No average treatment effect of ALRMP expenditures
Differential impact with fewer negative changes in treatment sublocations
ALRMP a nutritional safety net?

5. Program evaluation (PE) methods
Fundamental problem of PE: want to but cannot observe a person’s outcomes in treatment and control state
Solution 1: make treatment and control look the same (randomization)
Gives average treatment effect as
Solution 2: compare changes across treatment and control (Difference-in-Difference)
Gives average treatment effect as:
To next slide: So what is the key drawback of standard PE methods?

6. New PE method based on SD
Objective: to look beyond the ‘average treatment effect’
Approach: SD compares entire distributions not just their summary statistics
Two advantages
Circumvents (highly controversial) cut-off point Examples: poverty line, MUAC Z-score cut-off
Unifies analysis for broad classes of welfare indicators

7. Stochastic Dominance First order: A FOD B up to iff
Sth order: A sth order dominates B iff
MUAC Z-score
Cumulative % of population
FA(x)
FB(x)
xmax

8. SD and single differences
These SD dominance criteria
Apply directly to single difference evaluation (across time OR across treatment and control groups)
Do not directly apply to DD
Literature to date:
Single paper: Verme (2010) on single differences
SD entirely absent from the program evaluation literature (e.g., Handbook of Development Economics)

9. Expanding SD to DD estimation - Method
Practical importance: evaluate beyond-mean effect in non- experimental data
Let , and G denote the set of probability density functions of Δ, with 𝑔 𝐴 ∆ , 𝑔 𝐵 ∆ ∈𝑮
The respective cdfs of changes are GA(Δ) and GB(Δ)
Then A FOD B iff
A Sth order dominates B iff

10. Expanding SD to DD: interpretation differences
1. Cut-off point in terms of changes not levels.
Cdf orders change from most negative to most positive  ‘initial poverty blind’ or ‘initial malnutrition blind’.
(Partial) remedy: run on subset of ever-poor/always-poor
2. Interpretation of dominance orders
FOD: differences in distributions of changes between intervention and control sublocations
SOD: degree of concentration of these changes at lower end of distributions
TOD: additional weight to lower end of distribution. Is there any value to doing this for welfare changes irrespective of absolute welfare? Probably not.

11. Setting and data Arid and Semi-arid districts in Kenya Data
Characterized by pastoralism
Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds
Data
From Arid Lands Resource Management Project (ALRMP) Phase II
28 districts, 128 sublocations, June 05- Aug 09, 602,000 child obs.
Welfare Indicator: MUAC Z-scores
Severe malnutrition in 2005/6:
Median child MUAC z-score -1.22/-1.12 (Intervention/Control)
10 percent of children had Z-scores below -2.31/ (I/C)
25 percent of children had Z-scores below -1.80/-1.67 (I/C)

12. The pseudo panel Sublocation-specific pseudo panel 2005/06-2008/09
Why pseudo-panel?
Inconsistent child identifiers
MUAC data not available for all children in all months
Graduation out of and birth into the sample
How?
14 summary statistics for annual mean monthly sublocation - specific stats: mean & percentiles and ‘poverty measures’
Focus on malnourished children
Thus, present analysis median MUAC Z-score of children z ≤ 0
Control and intervention according to project investment

13. Results: DD Regression
Pseudo panel regression model
where D is the intervention dummy variable of interest
NDVI is a control for agrometeorological conditions
L are District fixed effects to control for unobservables within major jurisdictions
No statistically significant average program impact

14. DD regression panel results
(1)
(2)
(3)
(4)
(5)
VARIABLES
median of MUAC Z <0
10th percentile
25th percentile
median of MUAC Z <-1
median of MUAC Z <-2
intervention dummy
0.0735
0.0832
0.0661
0.0793
0.0531
(0.248)
(0.316)
(0.371)
(0.188)
(0.155)
change in NDVI 2005/06-08/09
1.308*
2.611***
2.058***
0.927*
0.768*
(0.0545)
( )
( )
(0.0997)
(0.0767)
(change in NDVI)2 2005/06-08/09
-12.91**
-8.672
-12.70*
-0.954
1.924
(0.0293)
(0.136)
(0.0510)
(0.802)
(0.479)
Constant
0.501***
0.892***
0.839***
0.203***
0.120***
(2.99e-07)
(1.40e-08)
(8.70e-09)
( )
( )
Observations
114
106
R-squared
0.319
0.299
0.297
0.249
0.280
Not significant dummy
Few observations
What do you think of the fit? Not bad given just location dummies and NDVI?
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.

15. SD Results Three steps: Steps 1 & 2: Simple differences
SD within control and treatment over time: No difference in trends. Both improved slightly.
SD control vs. treatment at beginning and at end: Control sublocations dominate in most cases, intervention never dominates.
Step 3: SD on Diff-in-Diff (results focus for today)

16. Expanding SD to DD – controlling for covariates
In regression Diff-in-Diff: simply add (linear) controls
In SD-DD need a two step method
Regress outcome variable on covariates
Use residuals (the unexplained variation) in SD-DD
In application below, use first stage controls for agro- meteorological conditions (as reflected in remotely-sensed vegetation measure, NDVI).

17. For (drought-adjusted) median MUAC z-scores:
Below z=0.2, intervention sites FOD control sites, although not at 5% statistical significance level.
ALRMP interventions appear moderately effective in preventing worsening nutritional status among children.

18. Similar results at other quantile breaks

19. Similar results at other quantile breaks

20. Conclusions
Existing program evaluation approaches focus on estimating the average treatment effect. In some cases, that is not really the impact statistic of interest.
This paper introduces a new SD-based method to evaluate impact across entire distribution for non-experimental data
Results show the practical importance of looking beyond averages
Standard Diff-in-Diff regressions: no impact at the mean
SD DD: intervention locations had fewer negative observations and of smaller magnitude, especially median and below
ALRMP II may have functioned as nutritional safety net (though only correlation, there is no way to establish causality)

21. Thank you for your time, interest and comments

22. SD, poverty & social welfare orderings (1)
1. SD and Poverty orderings
Let SDs denote stochastic dominance of order s and Pα stand for poverty ordering (‘has less poverty’)
Let α=s-1
Then A Pα B iff A SDs B
SD and Poverty orderings are nested
A SD1 B  A SD2 B  A SD3B
A P1 B  A P2 B  A P3 B
Define U1 as the subset of U for which u’>0. U1 monotonic utilitarian welfare functions. Less malnutrition is better, regardless for whom.
Let U2 be a subset of U1 such that u’’<0. This subset of social welfare functions represents equality preference in that a mean preserving progressive transfer increases U2. Finally, define U3 as the subset of U2 for which u’’’>0. U3 contains the transfer sensitive social welfare functions which value a transfer more highly the lower in the distribution it occurs.
Again, with a nutritional indicator this is only defensible up to a certain point, but certainly up to xmax.

23. SD, poverty & social welfare orderings (2)
2. Poverty and Welfare orderings (Foster and Shorrocks 1988)
Let U(F) be the class of symmetric utilitarian welfare functions
Then A Pα B iff A Uα B
Examples:
U1 represents the monotonic utilitarian welfare functions such that u’>0. Less malnutrition is better, regardless for whom.
U2 represents equality preference welfare functions such that u’’<0. A mean preserving progressive transfer increases U2.
U3 represents transfer sensitive social welfare functions such that u’’’>0. A transfer is valued more lower in the distribution
Bottom line: For welfare levels tests up to third order make sense

24. The data (2) – extent of malnutrition

25. DD Regression 2 Individual MUAC Z-score regression
To test program impact with much larger data set
Still no statistically significant average program impact

26. Results – DD regression indiv data
Dependent variable: Individual MUAC Z-score
VARIABLES
time dummy (=1 for 2008/09)
0.0785
(0.290)
control - intervention by investment
(0.425)
Diff in diff
0.0245
(0.782)
Normalized Difference Vegetation Index
1.029***
(6.25e-07)
Constant
-1.391***
(0)
Observations
271061
R-squared
0.033
Not significant, even with very large sample size
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.

27. Full SD results 2 data sets; 2 indicators for panel data
Significant results for individual data
08 and control dominate
DD only for panel data (focus for the rest, as new method for this)
 Look at some of these results in more detail (across whole distribution)