Empirical validity of the evaluation of public policies: models of evaluation and quality of evidence. Marielle BERRIET-SOLLIEC 1, Pierre LABARTHE 2*, Catherine LAURENT 2, and Jacques BAUDRY 3 1 AGROSUP Dijon, UMR CESAER (Dijon, France) 2 INRA, UMR SAD-APT (Paris, France), 3 INRA, UMR SAD-Paysages (Rennes, France) * Corresponding author: 122 nd European Association of Agricultural Economists Seminar Evidence-Based Agricultural and Rural Policy Making Methodological and Empirical Challenges of Policy Evaluation February 17 th – 18 th, 2011, Ancona (Italy) associazioneAlessandroBartola studi e ricerche di economia e di politica agraria Centro Studi Sulle Politiche Economiche, Rurali e Ambientali Università Politecnica delle Marche

Outlines of the presentation  Research question. How to cope with the “Babel tower” of the methods for public policies evaluation? what is the level of empirical validity of the results of evaluation / goals? is it possible to combine these results? How?  Methodology. a simplified typology of evaluation methods. a conceptual framework: types and levels of evidences.  Results. The issues of the empirical validity of evaluation methods depend on the goal of the evaluation: to learn to measure to understand  Discussion. Is it possible to combine various goals of evaluation?

Empirical validity of the evaluation of public policies: models of evaluation and quality of evidence. Analytical framework Types and levels of evidence and goal of evaluation Marielle Berriet-Solliec, Catherine Laurent & Jacques Baudry.

Diversity of evaluation approaches typology of evaluation models  3 evaluation models / goal of the evaluation approaches 1.[Goal 1: To learn] the evaluation is primarily designed as a collective learning process 2.[Goal 2: To measure] the evaluation is designed to assess the impact of a public programme 3.[Goal 3: To understand] the evaluation identifies and analyses the mechanisms by which the programme under evaluation can produce the expected outcomes or not

Types of evidences  Evidences of existence. Demonstration of the existence of a fact o for example: biological inventory lists for the biodiversity  Evidences of causality. Demonstration of the causal relation between two variables o for example: the relation between a landscape mosaic of agricultural fields and the level of population of an insect specie  Evidences of effectiveness. Demonstration of the specific impact of a public action on its goal o for example: the impact of an agri-environmental scheme (grassy strip) on a biodiversity indicator  Evidences of harmlessness. Demonstration of the absence of adverse effects of a public action o for example: the absence of negative effect of an agri-environmental schemes on the survival of certain categories of farms

Level of evidences 4. Evidence obtained from at least one properly randomized controlled trials. 3. Evidence from well-designed controlled trials without randomization. 2. Evidence from historical comparisons. Evidence from cohort or case- control analytic studies. 1. Opinion of respected authorities, based on clinical experience, descriptive studies or reports of expert committees. Hierarchy of evidences / empirical validity Level of evidence  The possibility to use the results of an evaluation depends on the empirical validity of these findings  There is a hierarchy of the level of empirical validity of the evidences produced by a method of evaluation

Empirical validity of the evaluation of public policies: models of evaluation and quality of evidence. Results Marielle Berriet-Solliec, Catherine Laurent & Jacques Baudry.

[To learn] (1/2) General principles and example  Goal. to promote learning through close collaboration between stakeholders to build awareness and consensus, and to encourage new practices  Method. A sociogram of the network of the stakeholders involved in the programme o  nature and intensity of ties between stakeholders In-depth individual interviews with stakeholders o  to gather each person’s point of view and suggestion for improving the programme Working groups and collective discussions about results o  reach consensus  Example. The Soft System methodology (SSM, Checkland and Scholes 1990) applied to public programmes of training for farm advisors (Navarro et al. 2008)

[To learn] (2/2) The empirical evidence issue  The issue of level of evidence is often neglected. secondary to learning objectives, but raise problems ( van der Sluijs et al. 2008, Salner 2000). no direct testing of the reliability of the evidences brought in by various stakeholders  The issue of the competition of evidence. very often pointed out arbitration often based on non-transparent criteria risk of misuse of evidence in cases of conflict or interests between stakeholders (Jackson 1991)   a risk to focus on consensual solutions rather than on the most effective ones?

[To measure] (1/2) General principles and example  Goal. to measure the effectiveness of a given public programme, to identify its specific impact on a proxy representing the goal of this programme. a major issue: tackling the counterfactual problem.  Method. Econometrics gold method: Randomized Control Trials (RCT) with two core hypothesis that raise technical and ethical problems. o  randomization o  SUTVA hypothesis Second bets methods: Double difference, Matching, etc.  Example. The measurement of the impact of farm advisory public programmes on the knowledge of farmers (Godtland et al. 2004) or on the performance of their farms (Davis et al. 2004)

[To measure] (2/2) The empirical evidence issue  Obtaining evidence of a high level of empirical validity is a challenging issue. costly practices. strong methodological specifications with technical and ethical problems.  Some limitations in the scope of the results. it measures only the specific impact in a given context. it does not indicate precisely the mechanisms which rendered a public action effective it does not prove causality   some limitation in the use of the results? cannot be used to extend a programme to other contexts or periods cannot help understanding why a programme fail

[To understand] (1/2) General principles and example  Goal. to understand the mechanisms operating in the programme evaluated. To reveal in a reliable way the causal relations that explain why a programme works or not in a given context.  Method: Realistic evaluation (Pawson and Tilley 1997) o focuses on (i) the object evaluated; (ii) the mechanisms of public action; (iii) the context Program theory (Chen 1990) o putting forth hypotheses on the causality patterns (diagram)  Example. An economic incentive A must cause a change of agricultural practice B which has an ecological impact C

[To understand] (2/2) The empirical evidence issue  Two ways of using evidences of causality. To produce such evidences: To use theory in order to support the construction of the diagram of causality of the public action:  which limitation and which empirical validity of evidences of causality? theoretical models are always partial representations of complex phenomena there are different possible theoretical models for modeling the mechanisms by which a public action operates the observation of the real effects (evidences of effectiveness) cannot be replaced by expected effects (estimated with the measurement of the means actually employed in the public programme)   is it possible to combine different theoretical frameworks ?

Empirical validity of the evaluation of public policies: models of evaluation and quality of evidence. Discussion Marielle Berriet-Solliec, Catherine Laurent & Jacques Baudry.

Discussion  Different goals for evaluation to learn / to measure / to understand.  Different requirements for the use of evidences to measure evidences of effectiveness. to understand evidences of causality.  To build a rigorous framework about evidences enables to avoid misuses of evidences and of the results of the evaluation. to formalize the trade-off between the goals of evaluation and to support the choice of the method that best fits to it. to open a discussion about the possibility to combine different goals or methods of evaluation.

Thank you for your attention Marielle BERRIET-SOLLIEC 1, Pierre LABARTHE 2*, Catherine LAURENT 2, and Jacques BAUDRY 3 1 AGROSUP Dijon, UMR CESAER (Dijon, France) 2 INRA, UMR SAD-APT (Paris, France), 3 INRA, UMR SAD-Paysages (Rennes, France) * Corresponding author: 122 nd European Association of Agricultural Economists Seminar Evidence-Based Agricultural and Rural Policy Making Methodological and Empirical Challenges of Policy Evaluation February 17 th – 18 th, 2011, Ancona (Italy) associazioneAlessandroBartola studi e ricerche di economia e di politica agraria Centro Studi Sulle Politiche Economiche, Rurali e Ambientali Università Politecnica delle Marche