Criteria for Assessing The Feasibility of RCTs

RCTs in Social Science: York September 2006 Today’s Headlines: “Drugs education is not working” “ having reviewed research from across the world, a committee of doctors and scientists on the ACMD* concluded that the success of school-based schemes was "slight or non-existent" and could even be "counter-productive". * Advisory Council on the Misuse of Drugs (ACMD)

RCTs in Social Science: York September 2006 Why are we here? We share the goal of improving well being for society and individuals Interventions which we support as a society should benefit individuals and society and should not cause harm We value evidence

RCTs in Social Science: York September 2006 Who am I? Economist Managing Director of Matrix Research & Consultancy Ltd Member of the Campbell Crime and Justice Group Member of the Cochrane Campbell Economics Methods Group Advocate of improving the use of evidence to inform decisions

RCTs in Social Science: York September 2006 Who are you? Politicians? Policy advisers? Civil Servants? Practitioners? Researchers? Economists…..?

RCTs in Social Science: York September 2006 Important evaluation questions (adapted: Bain 1999) Should it work?  Theory Can it work?  Implementation Does it work?  Impact Is it worth it?  Value

RCTs in Social Science: York September 2006 Source of interventions Historical practice (“we have always done it this way”) Taught practice (“we teach people to do it this way”) Innovative practice (“I try new ideas”) Research (“I examine the theories”)  the popular vote………………

RCTs in Social Science: York September 2006 Role of trials for new medical interventions Idea Basic science Laboratory trials Clinical trials Licence Approval (e.g. NICE)  Average years?

RCTs in Social Science: York September 2006 Particular challenges for trials in social science (adapted: Farrington 1983) Pace of idea to practice Theory base of treatment Definition and scope of treatment Context complexity Community vs individual interventions Hawthorne effects Contamination Outcome measures & measurement Duration and decay

RCTs in Social Science: York September 2006 Arguments for Trials Interventions can do harm If undertaken well  trials minimize risk of bias Basic hypothesis: “can we identify an effect with confidence?” If results insignificant then: –There isn’t an effect; or –There is an effect but we haven’t detected it with confidence.

RCTs in Social Science: York September 2006 Arguments against trials Epistemology: “the world doesn’t work like this” Analytical: “there are too many analytical constraints” Ethical: “you can’t deny treatment” Legal: “you might be challenged if you deny treatments” Logistical: “there are too many practical constraints

RCTs in Social Science: York September 2006 Research on Feasibility Research partners Matrix Research & Consultancy Ltd The Jerry Lee Centre, University of Pennsylvania. Research base Collective research experience Home Office funded study for OBPs

RCTs in Social Science: York September 2006 Analytical challenges: Hierarchy 1.Internal validity: can we attribute effect to intervention? 2.Statistical power: can we measure the effect with confidence? 3.External validity: can we generalise the results?

RCTs in Social Science: York September 2006 Hurdles to ensure Internal Validity Participant selection targeting Completion and attrition rates Inconsistent treatments & treatment measurement Multiple outcomes and inconsistent outcome measurement Independent, contamination-free alternative, treatments or no treatment

RCTs in Social Science: York September 2006 Hurdles for Statistical Power Understanding expected/required effect size?  And….Given this… How to maximize statistical power given? –Heterogeneity of sample? –Completion rates? –Attrition rates?  And…given this… How many participants do you need?

RCTs in Social Science: York September 2006 Hurdles for External Validity Context / Characteristics: Local service provider Participants Community Tension between correcting for these and the increased challenges of achieving internal validity and statistical power.

RCTs in Social Science: York September 2006 Ethical Hurdles “denial of treatment is unethical” “validity of informed consent”

RCTs in Social Science: York September 2006 Legal Hurdles E.g. for offenders  tension between random assignment and requirements to sentence Scope of legal challenge: –from those denied the intervention –where programmes exist and are perceived to reduce risk to public/ stakeholders

RCTs in Social Science: York September 2006 Logistical Hurdles Quality of trial staff Cost of a trial relative to the cost of the intervention and the value of the expected effect Sample size required when completion rates are low and attrition post completion is high Avoidance of Hawthorne effect

RCTs in Social Science: York September 2006 Are there Solutions? Focus on particular well defined interventions Appoint independent trained research teams Randomise early Separate control groups Pick control treatments which enable effects to be measured Manage and monitor implementation Ensure programme stability Determine sample size from desired effect size – look at the theory and related research Measure outcomes consistently Identify cost effective study design Educate and inform participants and stakeholders

RCTs in Social Science: York September 2006 If these aren’t available? Only then ask yourself: “What is the next best alternative….and how can I minimize the risk of getting it wrong”

RCTs in Social Science: York September 2006 Final reflections Researchers should recognise: –decisions need to be made in real time –all information can provide evidence Users should recognise: –the risks of making the wrong decision increases as the quality of the evidence base declines In social science we are addressing the needs of people and communities who are vulnerable