Pluralité des connaissances scientifiques et intervention publique: agriculture, environnement, et développement durable. Policy evaluation and empirical evidence : lessons from the EBP debates. Paris Seminar, June, 5 th, Marielle BERRIET-SOLLIEC Pierre LABARTHE Jacques BAUDRY Catherine LAURENT

 “Boom”of policy evaluations: tool for their modernization  Emergence of typologies of public policies evaluations Hansen (2005)Six types: oEconomic models (1. cost/benefit; 2. cost/effectiveness; 3. cost/advantages) o4. Results models o5. Explanatory process models o6. Programme theory models Perret (2001):Five evaluation paradigms : o1. Accountability; o2. Social value; o3. Utilization focused evaluation; o4. Theory driven evaluation; o5. Experimental methods  These typologies are rather complex constructions that mix considerations about goals, theories and methods…  … but pay little attention to empirical validity Introduction (1/2)

 Aim of our paper: Starting point: A simple “typology” of evaluation procedures / their aim: (learning / measuring / understanding) Main idea: To use the outcomes of the EBD debates to discuss about the empirical validation of these procedures oTypes of evidences oLevels of evidences Two case studies: oAgricultural extension oBiodiversity policies Introduction (2/2)

Pluralité des connaissances scientifiques et intervention publique: agriculture, environnement, et développement durable. Section 1. Rethinking typologies of procedures for the evaluation of public policies according to their aim. Paris Seminar, June, 5 th, 2009.

Different aims for evaluating public policies  Learning Aim: Evaluation as an instrument to learn Methods:Large forum or participative approaches Ex (extension): Soft system methodology for extension pg. Ex (biodiversity): SMA.  Measuring Aim:Assessing the specific effect of a public policy Methods:Quantitative experimental settings / econometrics Ex (extension): Matching for farmers’ field school assessment (Kenya) Ex (biodiversity): Matching for “prime à l’herbe” assessment  Understanding Aim:Identifying the causality scheme of a policy Methods:Very diverse (from pure theory to monography) Ex (extension): Integrating extension in agricultural production fonction Ex (biodiversity): UK agri-env schemes on birds

Pluralité des connaissances scientifiques et intervention publique: agriculture, environnement, et développement durable. Section 2. EBP and the debates about the empirical evidences Paris Seminar, June, 5 th, 2009.

Different types of evidences  Evidences of existence Definition: verifying the facts at field level Ex (extension): accountability (investments and field operations) Ex (biodiversity): faunistic and floristic inventories  Evidences of effectiveness Definition: measuring the specific effects of a policy on its goal Ex (extension): yield growth of farms benefiting from extension Ex (biodiversity): positive correlation between a policy and observable level of biodiversity  Evidences of causality Definition: identifying the relations of cause and effects of a policy Ex (extension): services  new knowledge  new practices  effects Ex (biodiversity): policy  landscape mosaic  reproduction conditions  biodiversity  Evidences of harmlessness Definition: identifying the relations of cause and effects of a policy Ex (extension): effects of extension programmes on income inequalities between farmers Ex (biodiversity): effects of biodiversity on income inequalities between farmers

EBP method: hierarchy of evidence, a key step  Classification of evaluation methods / quality in terms of empirical validity of the evidence produced 1. Opinion of respected authorities, based on clinical experience, descriptive studies or reports of expert committees. 2. Evidence from historical comparisons. Evidence from cohort or case- control analytical studies. 3. Evidence from well-designed controlled trials without randomization. 4. Evidence obtained from at least one properly randomized controlled trials. Level of empirical validity

Pluralité des connaissances scientifiques et intervention publique: agriculture, environnement, et développement durable. Section 3. Revisiting evaluation methods For extension or biodiversity programmes Paris Seminar, June, 5 th, 2009.

Better understanding the trade-of between “understanding” and “measuring” (1/2)  Case study 1: Does extension commercialization work?  A policy based on standard economic assumptions: “efficiency and demand-driven extension” (Baxter, Carney)  Most of evaluation are based on standard modeling that compare public and private efficiency “ad-hoc” methodologies and validation that do not fit with the EBD recommendation about evidences of effectiveness Some strong assumptions that are not discussed in the line of their empirical validity (individual level without interactions between farmers, short-term analyses, no back-office consideration)… … even though some other approaches provide highlights about the failures that could derives from such assumptions (asymmetry of information, innovation networks, technological lock-in, etc.)

Better understanding the trade-of between “understanding” and “measuring” (2/2)  Case study 2: biodiversity  Within field management (undrilled patches) has a positive effect on birds (skylarks)  An analysis of chicks diet shows the importance of invertebrates (measuring by correlation)  These invertebrates thrive in undrilled patches covered by weeds (understanding)

Limits in the use of scientific knowledge in evaluations aimed at “learning”  Case study 1: extension & soft system methodology  Aim: providing communication techniques (soft system) to support collective and consensual decisions (Checkland 1981)  Scientific knowledge, and the production of evidences is not pictured as central to these methods, BUT Scientific knowledge is used by stakeholders The control of the empirical validity of the scientific content of the debates is a key problem of these evaluation methodologies (Salner 2000) These methods can be distorted and can lead to strong asymmetries between stakeholders when there are conflicts between them (Jackson 1991)

Pluralité des connaissances scientifiques et intervention publique: agriculture, environnement, et développement durable. Discussion Paris Seminar, June, 5 th, 2009.

A new perspective on methods for evaluating public policies  Discussing the trade-of between methods For effects evaluating, a need for evidences of effectiveness For cause/effects understanding, a need for evidences of causality A major trade-of: oFor evidences of effectiveness  randomization (RCT) oFor evidences of causality  a need for ad-hoc experimental settings Potential failure Transposition failure (even though evidences of effectiveness exist) Etc.  How to overcome this difficulty? Possible failures of participative approaches? A need for collective organizations and new theories of the contextualisation and synthesis of scientific knowledge fitting with the requirements of decision in practice (Cartwright 2007)