How qualitative research contributes to evaluation Professor Alicia O’Cathain ScHARR University of Sheffield 22 June 2015

What is qualitative research? Normal in evaluation Understanding not measuring Set of methods – Focus groups – Semi-structured or in depth interviews – Non-participant observation – Diaries

What is evaluation? Researcher led evaluation Policy evaluation

Researcher-led MRC Framework developing an evaluating complex interventions ACTIF programme – 5 years – RCT – Qualitative at each phase

O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Hewison J. What can qualitative research do for randomised controlled trials? A systematic mapping review. BMJ Open 2013;3:e Intervention Trial design and conduct Outcomes Measures Health conditions

Intervention n=254 Develop n=48 Describe it n=10 Understand how it works n=23 Value and benefits n=42 Acceptability in principle n=32 Feasibility and acceptability n=83 Fidelity, reach and dose n=12 Implementation in real world n=4

Trial design and conduct n=54 RecruitmentDiversity Participation in trials Acceptability in principle Acceptability in practice Ethics/informed consent Adapting to local circumstances Impact on staff, researchers, patients

Potential value BiasAvoidance of measurement bias EfficiencyFaster recruitment Saves money EthicsTrials sensitive to human beings Improved informed consent ImplementationFacilitates replicability of intervention in the real world Facilitates transferability of findings in the real world InterpretationExplains trial findings Relevance Ensures interventions meet the needs of health professionals and patients SuccessMakes a trial successful, feasible, viable ValidityImproves internal validity Improves external validity

Maximising value… 1. Do it early – 28% pre-trial Intervention development 100% Acceptability of intervention in principle 25% Acceptability of intervention in practice 24% Recruitment 18% Breadth of outcomes 0% …otherwise its about future trials

2. Publish learning for specific trial or future trials 3. Think beyond interviews: non-participant observation 4. Try iterative or dynamic or participatory approaches at feasibility phase 5. Not just complex interventions 38% of 104 data extracted were drugs or devices 6. Think about the range of work

Problems with quantitative only Null RCTs….explain findings (context, mechanisms of action, implementation) Failed trials….prevent this at pilot stage It works but what is ‘it’? …..qualitative can fix

Policy evaluation Learning from early adopters (feasibility) Stakeholder reception (acceptability) Service delivered (implementation, workforce)

Useful but challenges remain – Fast evaluation – When to evaluate – Moving target – Replacement of difficult-to-measure outcomes with understanding of processes

Conclusions Useful contribution no matter what type of evaluation – essential due to complexity Can help to fix problems faced in researcher- led evaluation Challenges in policy evaluation need reflection