How to evaluate ultimate impact of value chain interventions? Mixed methods design for attributing indirect interventions to farmers’ income. The case of maize in Bangladesh Gideon Kruseman and Shovan Chakraborty 25 March 2013

Overview  Case  Evaluation questions  Design (conference theme)  Impact logics  Mixed methods design  Results  Conclusions  Communication and results (conference theme)  Lessons learned

Case description  2004 Katalyst introduced maize through  Retailer training  Extension officers training  Contract farming  2011 impact evaluation indicated strong income impact but method lacked before-after and with-without  2012 new impact evaluation based on rigorous scientific methods

Questions 1.What ultimate impact can be expected 8 years later? 2.How can that be measured?  No baseline possible in VC (uncertain who will benefit ex-ante)  Difficult to determine comparison group with notorious spill-over effects 3.What is the impact?  Outreach  Impact  Attribution Q3 challenging -> Mixed methods Q1&2 uncommon -> Multi- disciplinary team with knowledge of value chains

Design of IE What expectations underpinned choices for design? 1.There is impact that is attributable to the intervention 2.Measurement of impact is limited by lack of baseline, no clean control groups and many external factors 3.Attribution is strongest low in impact chain (with retailers, officials and companies, not with farmers) 4.If intervention is unique, attribution is easier to establish

Impact logics and the IE (Q1)

Design (Q2)  Approach  For each question test assumptions of the impact logic at every step  Core methods  Large n farmer survey data  Small n in-depth interviews with farmers, contractors and retailers, treated and non-treated  Maize sector study  Analysis  Production cost-revenue analysis  Factor analysis  Qualitative analysis of in-depth interviews

Assumptions in impact logic (Q1&2)

Mixed methods (Q3)  What outreach?  In-depth contractors interviews  Validation with in-depth farmers interviews  What income effects?  Large n farmer survey for production cost analysis and quantitative cohort comparison  Validation with in-depth farmers interviews  To what extent can impact be attributed?  Qualitative analysis of all in-depth interviews  Factor analysis of large n survey

Results: outreach Farmers affected by contract farming char newchar old mainland new mainland old Total Direct outreach Indirect outreach

Results: income effect Relevant assumption to be tested: farmers have higher yields than the benchmark char newchar old mainland new mainland oldcopy yield (maund/dec) Total Rev. Impact not considering Land size Change (BDT)17,41042,9801,9383,8797,947 Total Rev. Impact including Land size Change (BDT)38,52380,7574,74210,39916,902 Total income increase (1 yr after intervention)33,399,450177,908,5154,580,29318,230,162395,988,876

Results: contribution of Katalyst  Contract farming has started as a result of Katalyst interventions. ● Only contractors are those involved in the intervention ● These contractors have growing number of contract farmers  Conclusion: True  Knowledge passed on through Katalyst training of contractors is crucial for contractors ● Knowledge comes from many sources including Katalyst  Conclusion: True / False

Results: contribution of Katalyst  Knowledge passed on from contractors to farmers is crucial for farmers ● Knowledge comes from many sources including Contractors ● Contractors are main source of information  Conclusion: partly true  Service provision by contractors is important to contract famers  Conclusion: true  There is a spill over effect of maize cultivation from contract farmers to neighbouring farmers  Conclusion: true

Results: contribution of Katalyst  Knowledge passed on through Katalyst training of Retailers is crucial for retailers ● Knowledge comes from many sources including Katalyst  Conclusion: True / False  Knowledge passed on from retailers to farmers is crucial for farmers ● Knowledge comes from many sources including retailers  Conclusion: True / False

Conclusions  There is impact of contract farming especially because of service provision specifically related to this production form  The impact of contract farming is 100% attributable to Katalyst  Knowledge on maize cultivation comes from many sources, knowledge is vitally important but relative importance of information sources cannot be attributed to any single source.  There is a contribution to the knowledge base through Katalyst interventions

Communication and use of IE Communication is focussed on:  Underpinning and justifying estimated impact  To which interventions could this be attributed  Some interventions (contract farming) have attributable impact  Some interventions (retailer training) do not  Results are used for:  Fine-tuning current interventions  Design of improved monitoring of interventions and immediate and intermediate results to ensure more robust future impact evaluations (counterfactual analysis)

Lessons learned  Importance of impact logic framework  Importance of defining hypothesis based on impact logic  Importance of design based on hypotheses and application of mixed methods to overcome threats to validity of conclusions  Separation of impact and attribution/contribution  Need to inform donors on costs benefits of IE

END Thank you for your attention.