1. Moving Along the Impact Pathway: The Case of IAA in Bangladesh John Antle & Roberto Valdivia Oregon State University Charles Crissman & Khondker Murshed-e-Jahan WorldFish Center Presented at Assessing the impacts of international agricultural research on poverty and under-nutrition: A mid-term workshop for studies commissioned by the CGIAR Standing Panel on Impact Assessment (2011 – 2013), London International Development Centre, May 8-9 2012.

2. Project Goals Estimate adoption, poverty reduction and household nutrition impacts from the promotion of integrated aquaculture-agriculture technologies in Bangladesh. H1: DSAP recommended practices are economically feasible for more than 50 percent of the target population H2: Incomplete adoption is explained by: (a) average productivity and/or cost of production; (b) high variability in productivity and/or cost of production. H3: Adopters have lower poverty and better nutrition than non-adopters. H4: Impacts are the same for small and large farms. H5: The TOA-MD model predicts adoption rates sufficiently well for use in ex ante and ex post impact assessment. H6: The TOA-MD model predicts aggregate impacts accurately without knowing adoption-outcome correlations. H7: Impacts within adopter and non-adopter sub-populations can be predicted accurately without knowing adoption-outcome correlations.

3. Development of Sustainable Aquaculture Project (DSAP) DSAP – objective to sustainably increase productivity through Integrated Aquaculture Agriculture (IAA) which focused on resource use efficiency through better utilization of resource flows between farm enterprises Project utilized a strategy of decentralized local-level long-term training, exposing farmers to a basket of 19 technologies and management practices. Large scale project implemented in 34 of the 64 districts in Bangladesh

4. DSAP Impact Assessment Data DSAP monitoring activity focused on farm survey and regular monitoring data from 260 participating farm households from four districts Baseline in 2003/2004 Training and extension during repeated visits from 2003/2004 through 2005/2006 during which regular follow up data was collected via a series of whole farm monitoring surveys Control: 123 non-project farmers from the same districts were surveyed in 2003/2004

5. DSAP Survey Data BaselineFollow up 2002/032003/042004/052005/062011/2012 Project225 Secondary Adopter 225 Control123 Baseline covered 260 farm households – 35 were large commercial rice-fish operations that are dropped for this analysis Secondary adopter – selected from matching project village Follow up survey will cover same households as in original project

6. Multi-Dimensional IA: Motivation “Moving along the impact pathway” how can the research community respond to stakeholders’ desire to understand tradeoffs and synergies between economic, environmental and social dimensions of sustainability? Given realities of data quality, costs of data and human resources, how good is good enough? But…complexity does not trump principle of parsimony

7. Simulation-based IA: Motivation There is a need for a feasible, generic, transparent approach to multi- dimensional IA that can be implemented at “low cost” in terms of data, time and human resources, that is based on received economic and statistical theory. The simulation-based approach to IA responds to this demand, by building on and integrating established concepts in the technology adoption, statistics and econometrics literatures. –It is a way to integrate various kinds of data to simulate the economically feasible adoption rate and various indicators based on quantifiable outcome variables –It can be linked to econometric behavioral models and market equilibrium models for aggregation and disaggregation

8. Some features of the simulation approach implemented in TOA-MD: Is a parsimonious, transparent model: results can be easily interpreted in relation to underlying data and parameters Can utilize all types of available data: –survey data, experimental data, modeled data, meta data, expert data Provides a framework to carry out sensitivity analysis –Can use preliminary or “minimum data,” provide guidance for efficient collection of additional data when needed Can be used for prospective and policy-relevant IA (extrapolation; ex ante; analysis of policy impacts) –Overcomes the critical “support” assumption in the econometric approach that both “treated” and “untreated” individuals are observed Estimates various kinds of impacts –Conventional average “treatment” effects –Mean and threshold impacts on adopters and non-adopters, at any degree of adoption –Policy-relevant impacts: taxes, subsidies, Payments for Ecosystem Services, etc.

9. kk  opportunity cost k 10000  k (2,0)  k (0) 2 1  k (2)  k (1,0)  k (1) r (2,0) 100 r (2) Conceptual Model of Technology Adoption and Impact Assessment (Antle 2011 AJAE) Contours of equal density of joint distributions of ω and outcome variable k, for systems 1 and 2 Adoption curve for system 2 Impact indicators for outcome k (in this case, mean outcomes) Technology adoption, environmental econ, & policy evaluation literatures

10. Step 1: System choice (ω)(ω) 0 Map of a heterogeneous region Opportunity cost, system choice and adoption Opportunity cost  = v 1 – v 2 follows distribution  (  ) Generalize  to any ordering, e.g. “willingness to adopt”? System 1:  > 0 (non-adopters) System 2:  < 0 (adopters)  opportunity cost Key point: a model of a population using a system, not a farm using a technology

11. Key point: a fundamental piece of information about a system is its economic feasibility: an upper bound on potential use or adoption  = v 1 – v 2 is distributed in the farm population –“Every farm has its  ” –   =  1 -  2 –   2 =  1 2 +  2 2 - 2  1  2  12 –Ex post: observe system 2, approximate system 1 (counterfactual) –Ex ante: observe system 1, approximate system 2 (counterfactual) –How to estimate  12 ? “Unobserved heterogeneity”: how to approximate? Populations can be stratified by various criteria: geographic, technological and socio-economic Opportunity cost distribution

12. ()()  100 ω > 0 Derivation of adoption rate from spatial distribution of opportunity cost with adoption threshold a = 0 ω < 0 r (2,0) r(2) This model shows the relationship between the mean and variance of ω and the economically feasible adoption rate. If the mean of ω is positive (negative), the adoption rate is less than (greater than) 50%. Similarly, changes in variance have predictable effects on the adoption rate. Link to models with “selection on unobservables”: ω is unobservable, but observed or otherwise approximated distribution of returns can be used to estimate the moments of the ω distribution 50

13. kk  k 10000 2 1  k (1) r (2,0) 100 r (2) Step 2: Impact Assessment Adopter, non-adopter, and population means of unconditional and conditional outcome distributions  k (2)  k (0)  k is mean of outcome k When k is expected returns, the population mean is maximized at the adoption rate r(2,0) Key concepts: unconditional outcome distributions (ellipsoids) and outcome distributions conditional on adoption (filled). The latter are truncated by adoption decisions.  k (h) = mean of outcome k when all farms use system h  k (0) = population mean at predicted adoption r(2,0) for adoption threshold a=0 Population mean Adopter mean Non-adopter mean

14. kk  k 10000  k (2,0)  k (0) 2 1  k (2)  k (1,0)  k (1) r(2,0) 100 r (2) Counterfactual mean of adopters and the Average Treatment effect on the Treated (ATT) Counterfactual mean of adopters ATT at r(2,0) Counterfactual mean of non- adopters ATU at r(2,0) Similarly, the distribution of system 2 for ω> 0 can be used to construct the counterfactual for non- adopters and the average treatment effect on the untreated (ATU).

15. kk 1000  k (2)  k (1) Means, counterfactuals, ATT, ATU, ATE, LATE, MTE (see Heckman, Urzua and Vytlacil, RE Stat 2006) r(2,a) 100 0 ATT = average treatment effect on the treated (adopters) ATU = average treatment effect on the untreated (non-adopters) ATE Given the joint distributions of adoption variable ω and outcome k, we can compute all treatment effects as well as mean and threshold indicators at all adoption rates simulated by varying the adoption threshold a. Also we can show that: ATE =  k (2) -  k (1) = r(2,a) * ATT/100 + {100 – r(2,a)}*ATU/100 MTE = d  k (a) /dr(2,a)  LATE/  r(2,a) Adopter mean r(2,0) Adopter CF Non-adopter CF Non-adopter mean Population mean  k (a) MTE

16. kk  k 10000  k (2,0)  k (0) 2 1  k (2)  k (1,0)  k (1) r(2,0) 100 r (2) Effects of selection on impact indicators: Sorting gain and counterfactual bias Degree of selection bias in counterfactual depends on correlation between outcome and adoption variables. If correlation is zero, means and counterfactuals have zero slopes, and population mean is linear with slope equal to ATE.

17. Threshold indicator for system 1 (areas e+f) Threshold indicator for system 2 (areas b+d) Threshold indicator for system 1 (areas c+d+e+f)*  e k 0   k (2,0)   k (1,0)  System 1 System 2 b cdf BEFORE ADOPTION OF SYSTEM 2  k 0   k (2,0)   k (1,0)  System 1 System 2 b cdef AFTER ADOPTION OF SYSTEM 2 Threshold Indicators The same concepts can be used to define and simulate threshold indicators (poverty rates, poverty gap, environmental risk, nutritional risk, etc). * Ignoring areas outside the contour as negligible

18. kk  k 10000 2 1 r(2,0) 100 r (2) Policy-Relevant Impacts and “Local” Impacts: E.g., a policy reducing constraints on adoption to increase adoption rate from r c to r(2,0) rCrC As in Heckman and Vytlacil (Econometrica 2005) we consider policies that affect adoption but do not change the unconditional distributions of k and ω.

19. Implications for IA There is a fundamental symmetry between “ex ante” and “ex post” IA: each involves parameterization of: –the unconditional joint distributions between  and the outcome variables –the mechanism or process determining choice between systems By characterizing the joint distributions of the adoption variable  and the outcome variables, we can simulate the economically feasible adoption rate of system 2, and all relevant average and threshold impact indicators and “treatment effects.” Data collection should focus on characterizing these joint distributions in the relevant populations. Characterization of counterfactuals (both ex post and ex ante) should utilize all relevant information: primary, secondary, experimental, modeled, expert, meta data.

20. Counterfactual system design In most cases, counterfactual system can be constructed as a transformation of the observed system –E.g., changing crop variety leaves most of the observed system intact, but may alter productivity and land allocation –E.g., introduction of IAA changes management, productivity and “bio-resource flows” but not the components of the system Random coefficient model –In general, v 2 = v 1 + , where  is a convolution of v 1 and v 2 –Then v 2 = (1+  /v 1 ) v 1 = r i v 1 giving r = (1+  /v 1 ) where r =  r +  r ,   (0,1) –Using this model, with observations on one system and plausible bounds on  r &  r we can approximate mean, variance and between-system correlations for the other system –data for  r &  r can come from various sources: observations, models, experiments, meta data –Draft paper available on this concept

21. See Jahan and co-authors, Aquaculture Research (2010), Agricultural Systems (2011) “The impact of long-term IAA training provided to small-scale farmers in Bangladesh is assessed using panel data from 260 project and 126 control farmers who were monitored from 2002/2003 to 2005/2006. We find that the training had a significant positive impact on farmers’ technical efficiency, total factor productivity and net incomes. These result in higher food consumption and better nutrition for trained households compared to control farmers.” We interpret these results as “ATE” Bangladesh DSAP Case Study

22. Here we replicate and extend Jahan et al. using TOA-MD System 1: Farms with no training support and low integration of aquaculture and agriculture. Low integration is 2 or fewer managed bio-resource flows among farm enterprises in 2002-2003 –control group data show no significant trend from 2002/03 to 2005/06 System 2: Farms with training support and highly integrated aquaculture agriculture (treated group in 2005/06) Farms stratified by small and large (small is less than 1 hectare) Bangladesh DSAP Case Study (2)

23. DSAP Data 260 participating farms in the baseline survey: –59 classified as System 1 –after the training and extension activities, 46 of these switched to System 2, implying an adoption rate of about 78% Farm size, pond size, family size, and non-farm income Production activities defined as rice, vegetables, other crops, poultry and livestock, and fish Net return calculations based on per-farm revenues and costs of each activity

24. Impact indicators Mean farm income and per/capita income - $1500/year and $356/year. Fish culture contributes about 16% of farm income and 11% of total annual income. Poverty rate – an estimated 45% to 50% are live below the $1.25/day poverty line – in the survey 54% are below the poverty line Food consumption – the national Household Income and Expenditure Survey shows that on average rural households consume 2253 kilo calories of which fish contribute 52 kilocalories.

25. The TOA-MD Model Software with documentation in SAS and Excel, available to registered users at tradeoffs.oregonstate.edu –Self-guided course and training workshops Represents heterogeneous populations with multiple strata (e.g., small, large farms; agro-ecozones; etc) Generic whole-farm structure –Crop, livestock and aquaculture sub-systems with multiple activities within each system –Farm household size, non-ag income –Income, poverty and generic indicators (mean, threshold) Technology adoption/impact; ecosystem service supply; environmental change & adaptation

26. The TOA-MD Model Some features –Parameter parsimony: fundamental parameters are: 2 means, 2 variances and 1 between-system correlation of system expected returns (5 parms) 2 means, 2 variances, 3 correlations for each outcome variable (7 parms x N outcomes) –Unconditional joint outcome distributions for each stratum of the population is normal (thus, aggregate distribution is non-normal) –Predicted “adoption rate” (choice between systems) is based on expected returns over a relevant decision period –When system 1 and 2 data are not matched, the between-system correlations cannot be estimated from observations, so plausible values are estimated and used in sensitivity analysis. Show the model…

27. Adoption curves for small and large farms Observed adoption rate was approximately 76% based on 59 observations, very close to the predicted adoption rate of 78% averaged over small and large farms. Mean opportunity cost occurs at 50% adoption rate in this model. If it is negative then the adoption rate must exceed 50%, as in this case. The predicted adoption rate depends on the mean and the variance. Sensitivity to model parameters can be performed easily.

28. Mean net returns/farm for small farms ATE = 144 TT = 205

29. Small farm net returns: ATT, ATU and MTE

30. Poverty rates for small and large farms ATE = -8.3 TT = -11.6 ATE = -4.6 TT = -8.1

31. Mean fish consumption in small farm households (kcal/person/day) ATE = 13.4 TT = 16.5

32. Percent of small farm households exceeding the population average fish consumption of 51.6 kcal/person/day Note that more than 50% of adopters exceed average consumption, whereas non- adopters are very low and have very low incomes. ATE = 30.3 TT = 38

33. Large farm households exceeding the population average fish consumption of 51.6 kcal/per/day (%) ATE = 33.2, TT = 34.3 (note they are close because correlation between outcome and opp cost is near zero)

34. Sensitivity analysis to between-system correlation RHO12, small farms Adoption curves Mean returns Poverty ratesMean fish consumption

35. Conclusions TOA-MD model predicts IAA adoption rate very close to observed rate IAA technology has substantial positive impacts on farm income, nutrition -- to be further verified with additional new data Analysis shows IAA technology has substantial income and nutritional benefits for both small and large farms. Selection MAY be (but is not necessarily) important in predicting income and nutrition effects sufficiently accurately to draw meaningful policy implications. Data quality is the greatest challenge to meaningful impact assessment –Especially for large surveys based on RECALL How “adoption” is defined is important – “adoption of technologies” or “choice between systems”?

36. Conclusions Work to be done: –complete new surveys –compare simulation and ex post statistical analysis Extensions –Further testing/validation: adoption, extrapolation –Further explore systematic methods to combine primary, experimental, modeled, expert, and meta data Using crop simulation models to construct counterfactuals for CC impact & adaptation analysis Use for technology adoption Improve survey/field experiment design –Develop methods to estimate standard errors Can use bootstrap to construct SEs but implementing in publicly available software a challenge (try to maintain simplicity, generic structure) –Link to market models (Valdivia et al; IMPACT; other)