1. Wrap up and consequences for TOR’s
Athens June 2017

2. Research/ evaluation problems
Prevent ‘bags of problems’; having to address 17 questions and subquestions with no logic between them is not doable;
“An Issue Is Not A Research Question”…. The rapporteur of the Transport Group
Don’t try to take on board everything
Make sure that the background knowledge you assume is correct and Make that knowledge ‘shareable’ ( in the form of –eg– a working paper
Decomplexify

3. Role of theories/ ‘assumptions’
Basically two routes:
The program theory /intervention theory / the Policy are Theories Incarnate” –route and
The scientific (‘explanatory) theory route which can assist in developing and implementing policies

4. Route 1: the Intervention th/ Program theory route
Log Frames are not more than the beginning ;
Boxes with colours, graphics, arrows and other ‘nice things for nice people’ are not `program/ interventions. Only a starter.
Labelling the interventions/ measures as Mechanisms is wrong
Exercise, when finding M”s, in working with policy makers and stakeholders and ‘end groups’ to find the ‘theory incarnate’. O. Swab: A Pr theory is the theory of the programme makers + something else

6. Types of Mechanisms / Coleman boat model
Transformational mechanisms
Situational
mechanisms
Action-formation mechanisms

7. Summarize what is known about mechanisms by searching knowledge repositories like CAMPBELL, 3IE, COCHRANE, WHAT WORKS – SITES and others
Dont believe the statement that information on mechanisms in earlier studies is not relevant ‘because contexts always differ’. No, they do not , as there are demi-regs and path dependencies
Fake handbags: be careful
Methods: several. See EVALSED 2012 but also Leeuw (2003): an overview of 3 methods in “American Journal of Evaluation”.
Criteria: logical consistency/ validity and problem relevance

8. Route 2: explanatory theories
In helping developing and implementing policies / measures etc make use of insights from explanatory theories on causes of behavioral changes/ institutional changes
Remember this:

9. EBM: Pawson’s medicine development ‘pipeline’
Basic Research
Therapeutic Discovery
Preclinical Development
Animal Testing
Phase I
Safety and dose-finding
Phase II
Feasibility Studies
Phase III
Large-scale RCT
Regulatory Approval
Look closely at drug development and its evidence base in more detail.
Many versions of this diagram.
Many stage before you get to the large scale RCT.
I’ve called then Basic research, Therapetic discovery etc,
Note;
Naming and numbering not consistent – some versions use labels like – ‘stage zero’, ‘proof of concept. Etc
Not a linear progression - research moves back and forth
The stages not only represent different methods they involve entire disciplines.
The process eats up a great deal of time (illustrated!)
Go though SOME of these stages in the following slides.
typically years

10. EBP: Pawsons’ Pathway to [social] policy programs RCT
Policy Instigation
Programme management
Demonstration project
Full-scale evaluation (…RCT…)
EBP. Onto the contrast - onto the weakling.
RCT in social programme. Here is the pipeline. Lets compare and contrast.
Pressing problem, Some policy maker in the Whitehall or the Town Hall puts forward a big idea. Then [passes to programme management (managers, administrators practitioners) to devise all details of programme implementation and delivery).
If you are lucky you get a demonstration project to troubleshoot a bit.
But without further ado – Outcome evaluation often moves directly to the RCT.
Emphasise with the timescale.
2 to 5 years

11. The six key principles of a theory-based impact evaluation are:
1. Map out the causal chain (programme theory)
2. Understand context
3. Anticipate heterogeneity
4. Rigorous evaluation of impact using a credible counterfactual
5. Use mixed methods and do rigorous analysis
6. Results visualization & transfer / use focused
The six key principles of a theory-based impact evaluation are:

12. CHALLENGES FOR TDE THE COMPLEXITY PROBLEM
PREVENT DOING SLOPPY TBE´S: TILLEY’S CASE OF PRODUCING ERROR COSTS
‘POLISHING UP’ A POLICY THEORY THAT DOES NOT EXIST
PREVENT DESIGNED BLINDNESS
IS IT A SLOPPY EVALUATION WHEN NO PROGRAM THEORY /TOC HAS BEEN FOUND/ TESTED?

13. Consequences for TORS:

14. Assumptions/ toc/ program th ~ impl th VISUALIAZATION, TRANFER AND USE
WORLD BANK TEMPLATE
When
Assumptions/ toc/ program th ~ impl th
BIG DATA
Tde
VISUALIAZATION, TRANFER AND USE

15. WHAT ELSE CAN BE ADDED (OR DELETED) WITH TDE AS STARTING POINT?
ROLE OF THEORIES
WHAT ELSE CAN BE ADDED (OR DELETED) WITH TDE AS STARTING POINT?
ROLE OF MECHANISMS
VICTORE

16. YOUR SUGGESTIONS

17. Good luck!