EBP methodology aims at specifying - Which scientific evidences are actually used for policy decision (surveys) (Cf WP1) - Which scientific evidences are available, which are the scientific controversies (systematic reviews, meta-analyses…). How to make available evidence accessible. (Cf WP2) We are not naïve. We know that it is difficult to implement such reviews and such meta-analyses - There are technical difficulties (the volume of publications, etc.) - There are conceptual obstacles when dealing with complex problems

Objectives of WP2 To better assess and formalize advantage and limits of systematic reviews methodology (and meta analyses) by applying the method to questions related to the overall concern “Is farm activity (especially small scale farming) compatible with biodiversity conservation goals?” To get 3 kinds of outputs - A critical analysis of EBP method that claims a new role in policy making - A set of tests that will result in recommendations on scientific literature assessment - A set of analyses and state of the art on sub-questions that will converge in a state of the art to address the question “Is farm activity (especially small scale farming) compatible with biodiversity conservation goals?” (but will not pretend to be done on strict EBD criteria)

A possible systematic review in EBP Biosoc ET1 : clarifying the need for a systematic review ET2 : Systematic analysis and meta analyses to be tested around the question « Is farm activity (and especially small scale farming) compatible with biodiversity conservation? »

First tests were made in economics They show the heuristic power of the method … and its difficulties.

Heuristic value of EB methodologies An analytical framework enabling a critical assessment of existing studies on policy issues : questioning the « evidence » –Studies often claim their policy recommendations are the only ones possible on a scientific basis –EB analysis enables to question the evidence presented by using explicit criteria (especially for the review protocol and the quality assessment)

Heuristic value of EB methodologies An illustration from Winters A., McCulloch N., McKay A. : « Trade Liberalisation and Poverty : The Evidence So Far », JEL XLII, March 2004, pp –Authors claim their scientific status (no reference to WB activities) –Authors refer explicitly to “evidence” Applying EB methodologies : an opportunity to seize –Stage 1 : Do not make explicit the «review protocol » (how are the studies selected, what criteria are retained) –Stage 2: Do not make explicit the quality criteria (scientific coherence, axiomatic clarity, or empirical methods considered as relevant) –Stage 3: Data synthesis and extraction is not done on the basis of « evidence » : « although trade liberalization may not be the most powerful or direct mechanism for addressing poverty in a country, it is one of the easiest to change » p. 108)

The difficulties of EBD literature reviews Standard application of EB methods may lead to certain biases and difficulties : 1.Co-existence of different research programs which are not equally represented in publications and citation indexes 2. Method difficult to apply to complex problems (involving various research fields and disciplines) 3. Selection of key words, database, etc. may be tricky

Learning from the difficulties of EBD literature reviews… and new solutions -heterogeneity of research programmes- Co-existence of different research programs (RP) which are not equally represented in publications and citation indexes  to map out the different research programs in each discipline and take care to try include in the analysis the more relevant publications of each RP

Learning from the difficulties of EBD literature reviews… and new solutions -heterogeneity of research programmes- Illustration from economics : mapping out the different research programmes (RP) –Identifying the different research programmes (fundamentental hypothesis, analytical methods, …) –Linking research programmes to publications (i.e. what RPs are published in Journal XX?)

Learning from the difficulties of EBD literature reviews… and new solutions -heterogeneity of research programmes- –Extending identification of research programmes and links with rewiews to other disciplines : Avoid biaises of standard EBP states of the art Construct bridges and comparisons between disciplines involved in EBP-Biosoc problematics

Learning from the difficulties of EBD literature reviews… and new solutions - to deal with complex issues- Method difficult to apply to complex problems (involving various research fields and disciplines).  Problems related to issues involving multifactorial causality and involving various research fields.. Necessity to deal with several sub-questions that will converge in a state of the art to address the question “Is farm activity (especially small scale farming) compatible with biodiversity conservation goals?” (but will not pretend to be done on strict EBD criteria)

Learning from the difficulties of EBD literature reviews… and new solutions -conventions to implement the EB methods- Some elements to be clarified : –Stage 1: Review protocol Key words and selection of studies What status for unpublished material? –Stage 2 : Clarifying the quality criteria Scientific coherence (main hypotheses, analytical framework, coherence) Status of empirical data (methodology, status in front of theoretical framework) Status of produced knowledge : does the study aim at producing ‘truth’ or does it aim at being ‘consensual’ or ‘easy to use’ (i.e. post-normal science approaches)

Learning from the difficulties of EBD literature reviews… and new solutions –expected results - By carrying out systematic analysis with these new perspectives in mind, we expect : –To construct the critique of standard EBP methods –To construct tests resulting in recommendations on scientific litterature assessment –To underline other possible policy recommendations by a more complete state of the art