Wrap up and consequences for TOR’s Athens June 2017

Research/ evaluation problems Prevent ‘bags of problems’; having to address 17 questions and subquestions with no logic between them is not doable; “An Issue Is Not A Research Question”…. The rapporteur of the Transport Group Don’t try to take on board everything Make sure that the background knowledge you assume is correct and Make that knowledge ‘shareable’ ( in the form of –eg– a working paper Decomplexify

Role of theories/ ‘assumptions’ Basically two routes: The program theory /intervention theory / the Policy are Theories Incarnate” –route and The scientific (‘explanatory) theory route which can assist in developing and implementing policies

Route 1: the Intervention th/ Program theory route Log Frames are not more than the beginning ; Boxes with colours, graphics, arrows and other ‘nice things for nice people’ are not `program/ interventions. Only a starter. Labelling the interventions/ measures as Mechanisms is wrong Exercise, when finding M”s, in working with policy makers and stakeholders and ‘end groups’ to find the ‘theory incarnate’. O. Swab: A Pr theory is the theory of the programme makers + something else

Types of Mechanisms / Coleman boat model Transformational mechanisms Situational mechanisms Action-formation mechanisms

Summarize what is known about mechanisms by searching knowledge repositories like CAMPBELL, 3IE, COCHRANE, WHAT WORKS – SITES and others Dont believe the statement that information on mechanisms in earlier studies is not relevant ‘because contexts always differ’. No, they do not , as there are demi-regs and path dependencies Fake handbags: be careful Methods: several. See EVALSED 2012 but also Leeuw (2003): an overview of 3 methods in “American Journal of Evaluation”. Criteria: logical consistency/ validity and problem relevance

Route 2: explanatory theories In helping developing and implementing policies / measures etc make use of insights from explanatory theories on causes of behavioral changes/ institutional changes Remember this:

EBM: Pawson’s medicine development ‘pipeline’ Basic Research Therapeutic Discovery Preclinical Development Animal Testing Phase I Safety and dose-finding Phase II Feasibility Studies Phase III Large-scale RCT Regulatory Approval Look closely at drug development and its evidence base in more detail. Many versions of this diagram. Many stage before you get to the large scale RCT. I’ve called then Basic research, Therapetic discovery etc, Note; Naming and numbering not consistent – some versions use labels like – ‘stage zero’, ‘proof of concept. Etc Not a linear progression - research moves back and forth The stages not only represent different methods they involve entire disciplines. The process eats up a great deal of time (illustrated!) Go though SOME of these stages in the following slides. typically 10-14 years

EBP: Pawsons’ Pathway to [social] policy programs RCT Policy Instigation Programme management Demonstration project Full-scale evaluation (…RCT…) EBP. Onto the contrast - onto the weakling. RCT in social programme. Here is the pipeline. Lets compare and contrast. Pressing problem, Some policy maker in the Whitehall or the Town Hall puts forward a big idea. Then [passes to programme management (managers, administrators practitioners) to devise all details of programme implementation and delivery). If you are lucky you get a demonstration project to troubleshoot a bit. But without further ado – Outcome evaluation often moves directly to the RCT. Emphasise with the timescale. 2 to 5 years

The six key principles of a theory-based impact evaluation are: 1. Map out the causal chain (programme theory) 2. Understand context 3. Anticipate heterogeneity 4. Rigorous evaluation of impact using a credible counterfactual 5. Use mixed methods and do rigorous analysis 6. Results visualization & transfer / use focused The six key principles of a theory-based impact evaluation are:

CHALLENGES FOR TDE THE COMPLEXITY PROBLEM PREVENT DOING SLOPPY TBE´S: TILLEY’S CASE OF PRODUCING ERROR COSTS ‘POLISHING UP’ A POLICY THEORY THAT DOES NOT EXIST PREVENT DESIGNED BLINDNESS IS IT A SLOPPY EVALUATION WHEN NO PROGRAM THEORY /TOC HAS BEEN FOUND/ TESTED?

Consequences for TORS:

Assumptions/ toc/ program th ~ impl th VISUALIAZATION, TRANFER AND USE WORLD BANK TEMPLATE When Assumptions/ toc/ program th ~ impl th BIG DATA Tde VISUALIAZATION, TRANFER AND USE

WHAT ELSE CAN BE ADDED (OR DELETED) WITH TDE AS STARTING POINT? ROLE OF THEORIES WHAT ELSE CAN BE ADDED (OR DELETED) WITH TDE AS STARTING POINT? ROLE OF MECHANISMS VICTORE

YOUR SUGGESTIONS

Good luck! Frans.leeuw@maastrichtuniversity.nl flleeuw@cuci.nl