Evaluating agricultural value chain programs: How we mix our methods 15/01/2019 Marieke de Ruyter de Wildt AEA Conference San Antonio 10-13 November 2010

Regulatory environment From Agricultural Value Chains to Systems 15/01/2019 Supporting services Inputs, land, energy, price information, R&D Inputs Farmers Brokers Processors Trader Retailers Consumer Regulatory environment Standards, regulation, sector policies International prices, trade agreements, tariffs policies, speculations

Change in Approach, change in impact patterns 15/01/2019 End program Now: developing market systems around chain actors Outreach Before: direct delivery to chain actors Time 3

Guideline 1: Critical Ingredients 15/01/2019 Any evaluation should at least have: Logic Model (beliefs, activities and results) Methods that can face scrutiny Insights that allow replication

Guideline 2: Test Against Validity Threats 15/01/2019 Explore robustness of ingredients from different angles: Construct validity: Are concepts properly defined and operationalized Internal validity: Resolve issues of causality/attribution Statistical conclusion When using statistics, do it validity: properly External validity: Under what conditions do conclusions apply Source: Shadish, W. R., T. D. Cook, et al. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference

Combining Ingredients and Validity Threats 15/01/2019 Combining Ingredients and Validity Threats Construct Internal Statistic External Logic model High Method Replication

1. Focus Define logic model On what basis do we expect success? 15/01/2019 Define logic model On what basis do we expect success? What ‘level’ of definition suits our evaluative question? What are critical assumptions, obviously about impact but also about assumed causalities (arrows)? How to test logic model and counterfactuals (critics)? Construct Internal. X

2. Method Mix methods to anticipate validity threats 15/01/2019 Mix methods to anticipate validity threats 1. Negotiate core methodology that fits main evaluative questions and ‘real-world constraints’: how can we anticipate implementation issues? 2. Add additional methods for assumptions in A. Program theory: are concepts precise enough to measure? B. Methodology: Right timing? Have enough indicators? Control group biased or sufficiently clean (spillover)? Construct Statistical X

3. Replication Explore conditions that make it work 15/01/2019 Explore conditions that make it work Reflect on common elements in pilots: do we have ‘generalisation domain’ defined? Focus on mechanisms in context: What works for whom under what conditions? We have methods that allow more general conclusions? Construct External X

Example: training coffee farmers in Vietnam

1. Logic Model Critical assumptions Methodological implications 15/01/2019 More knowledge on good practices More income, better quality and less damage (PPP) Better agricultural practices Training farmers Critical assumptions Methodological implications Training is fairly homogeneous: Realist case comparison of access criteria, modules and delivery More knowledge leads to better practices: and of application mechanisms of knowledge More training = better: compare intensive training to less intensive training (factor or cluster analysis) Reduce threats to construct and internal validity

Reduce threats to Construct and 2. Method 15/01/2019 Core method: Difference in Difference to scan for results in PPP Added mixed methods: For key assumption in program theory Realist case studies to scan unexpected outcomes (eg increase in self esteem) Considering mediating and moderating variables (thanks to Kathleen, Research Works) For methodological assumptions Nested survey (power analysis) Pilots (data availability) Reduce threats to Construct and Statistical validity

Reduce threats to Construct and 3. Replication 15/01/2019 Reduce threats to Construct and External validity

Conclusions 15/01/2019 One-method research might be good for publication in top journals, but rarely for generating convincing evidence for involved agents Evaluation design needs to be Theory-based (clarify evaluative questions) Using mix methods (minimize validity threats) Address policy relevance (make sense of diversity) Considering validity threats up front helps to find a more robust mix of methods

Marieke.ruyter@wur.nl