Conducting Efficacy Trials Dr Sarah Miller | EEF Evaluators’ Conference | June 2016 www.qub.ac.uk/cesi

Characteristics of an efficacy trial A trial in which the intervention is tested under ‘ideal’ conditions and often implemented with input from the programme developer. It answers the question ‘can it work’? Usually an earlier pilot trial has already indicated that that the intervention is feasible to implement in a classroom setting (there may also be some evidence of effects) In fact, a good pilot trial is key to a good efficacy trial www.qub.ac.uk/cesi

Theory of change In an efficacy trial: There is a greater focus on understanding the theory of change and underlying causal mechanism(s) The intervention should be underpinned by a conceptually feasible and evidence informed theory of change Consequently, relevant outcomes are selected on the basis of this logic model www.qub.ac.uk/cesi

Internal validity An efficacy trial will have high internal validity (at this stage external validity is less crucial), that is: Sufficient power to detect the estimated effect Robust (and tested) allocation procedures are employed Valid and reliable instruments are used to measure outcomes (as you would plan to in the effectiveness trial) Data collection methods are independent, tried and tested www.qub.ac.uk/cesi

A sufficiently in-depth process evaluation The process evaluation should be sufficiently in-depth to allow you to: Qualitatively test the proposed theory of change Ensure that all key agents are included in the process evaluation and use a range of methods where possible e.g. teacher survey, parent interviews, pupil focus groups Identify issues related to scaling up the intervention e.g. any challenges associated with implementation and delivery Understand and monitor inevitable adaptation in practice www.qub.ac.uk/cesi

A sufficiently in-depth process evaluation Reflect and learn lessons where possible e.g. Attrition (at school and pupil level) – avoid where possible but aim to fully understand it if and when it happens so that it can be avoided/minimised in an effectiveness trial Efficacy trials also provide an opportunity to better understand recruitment and retention - why schools are not keen to participate or drop out post randomisation Importance of regular and efficient communication between developers and evaluators www.qub.ac.uk/cesi