Challenges in evaluating social interventions: would an RCT design have been the answer to all our problems? Lyndal Bond, Kathryn Skivington, Gerry McCartney, Hilary Thomson York September 2009

Overview Importance of evaluating effects on health of social interventions Welfare to work – health benefits – what is known Pathways Advisory Service Evaluation challenges Design Comparison group Data sharing Ethics & recruitment Conclusions & where to from here?

Evaluating the health effects of social interventions programme Programme’s aim: To ensure policy decisions are based on the most reliable research evidence available By undertaking: systematic reviews, bringing together existing research on the effectiveness of social/policy interventions new studies evaluating the health impacts of social/policy interventions

What do we study? The effects on health and health inequalities of: Housing & community regeneration Transport policies Tobacco control legislation Work organisation & welfare to work policies

Welfare related policies & programmes Social/policy interventions affecting: employment & income welfare policies means of improving the health of the worst off & reducing health inequalities.

Welfare-related policy interventions: evidence for effectiveness Lack of evidence for effects on health House of Commons Health Committee (2009) inquiry into health inequalities concluded that the lack of evidence was: “in large part due to inadequate evaluation of the policies adopted ”

Welfare to Work programmes Aim to reduce number of people claiming benefits by moving them into employment May be regarded as important ‘healthy public policy’ interventions potential to impact on health and health inequalities A positive relationship between employment & health (Waddell, 2006) ‘work is good for health’

But… Potential for health benefits for welfare recipients moving into employment is likely to be dependent on type & suitability of the job individual-level factors – age, health etc Evidence for health benefits of returning to work for groups such as those in receipt of illness- related benefits is limited There is a need to evaluate such interventions for their effect (positive or negative) on health

The Pathways Advisory Service PAS places employment & benefit advisors into GP surgeries to engage with IB recipients Provides advice about benefits & return to work including guarantee of benefits in 12 months if return to work not successful GPs refer patients to the service Uptake is voluntary Patients can self-refer

DWP evaluation of pilot PAS Department for Work and Pensions (DWP) had commissioned an evaluation of the PAS pilot programme process, & employment & benefit outcomes no health outcomes (Sainsbury, 2008) PAS rolled out before pilot evaluation completed & rolled out service different to that which was piloted

Our planned evaluation of PAS Robust evaluation of the effects on health of PAS Presumed through a return to work? Experimental or quasi-experimental design Self-reported health Health services use

Challenges Evaluability of the intervention Who is eligible for and receives the intervention? Identifying and recruiting: issues of transfer of contact details to obtain informed consent to the study Defining a comparison group

Evaluability of the intervention Cluster RCT - design of choice appropriate & justified given uncertainty about harms & benefits with respect employment, welfare, income & health But before an evaluation of health impacts could be planned PAS was being rolled out across the UK so not possible to make the case for this design use of well conducted non-randomised study designs has been advocated as an appropriate alternative we proposed a quasi-experimental study; a non-randomised controlled trial.

Who is eligible & who receives the intervention? Service is available to all IB recipients GPs use their discretion to refer patients to PAS Uptake is voluntary and patients can self-refer About 20% of IB recipients in participating practices are estimated to engage with the PAS intervention It is likely that those who are referred have: relatively good health are closer to returning to the labour market

Identifying & recruiting 1.IB recipients from DWP records not possible due to implementation of a data transfer ban following high profile loss of sensitive personal information by a number of Government departments 2.Target population of IB recipients through GP electronic records indicating a GP has completed a medical reference for an IB claim Limited by GP practices who use particular software Estimated about 70% of IB patients would be so recorded

Defining a comparison group Need to be similar with respect to eligibility for intervention in receipt of IB age, health status, length of time on benefits, employment history etc Could identify IB recipients through GP records, registered with GP practices not offering PAS. BUT Referral and access to PAS was up to GPs’ discretion & patient self-selection proportion of IB recipients engaging with the service is small those engaging with PAS likely to have relatively good health & be closer to a return to the labour market than those not engaging with PAS. Making it difficult to identify an appropriate & available comparison group

Defining a comparison group Need to rely on capacity to collect appropriate data to adjust for baseline differences between Gp 1 & Gp3

Proposed recruitment Local Research Ethics Committee required: written opt-in consent for contact details to be given to researchers GPs could send out only 1 reminder letter to potential participants. Estimated <10% response GP writes to potential participants identified through medical records ` Potential participants given chance to stop contact details being given to researchers GPs pass contact details to the researchers Researchers write to potential participants with more details about the study Participants who give informed written consent are recruited to study

Interpretation of the Data Protection Act Data Protection Act – not clear ongoing discussion regarding what the Act means for data sharing for health research it does not necessitate individual consent for all medical research but its wording does not provide clear guidance on when consent by individuals is necessary for medical research Medical Research Council Guidance states: there must be an acceptable balance of risk & benefit for participants. justification for disclosure of a limited amount of personal information given in confidence without consent, if patients should be given the opportunity to raise objections, then limited disclosure is unlikely to give rise to any serious objections.

Summary of barriers No control over method of implementation compromising study design Ban on DWP data sharing limited identification transfer of contact details of potential participants the identification of a comparison group Data Protection Act – interpretation limited capacity to recruit RESULT: unethical to undertake study due to bias and lack of power

Ways forward Closer & timely working between policymakers & researchers Allowing input at early stage for evaluation design & develop policy-relevant evaluations “All too often Governments rush in with insufficient thought, do not collect adequate data at the beginning about the health of the population which will be affected by the policies, do not have clear objectives, make numerous changes to the policies and its objectives and do not maintain the policy long enough to know whether it has worked. As a result, in the words of one witness, ‘we have wasted huge opportunities to learn’. Simple changes to the design of policies and how they are introduced could make all the difference.” (2009) House of Commons Health Committee

Ways forward (2) Response to loss of data by Government departments by DWP has been to develop data sharing protocols: Data sharing protocols no data sharing protocols between the DWP and research organisations unless a ‘commissioned’ evaluator for DWP DWP not researcher driven research/evaluation Way forward???

Ways forward (3) Need clarity of legal and ethical interpretation of the Data Protection Act Is it guidance? Is it legislation? Who decides the balance between individual & collective good? Are the evaluations of ‘non-medical interventions’ perceived to be less important & therefore not worth even ‘low risk’ research? Is that acceptable given: Expensive Unexpected consequences Particularly unfair for disadvantaged groups

Back to control group This remains a fundamental issue Resolution? Establish a cohort study to enable nested evaluations of policies Expensive, large numbers, but could be worth it Other suggestions are welcome

Conclusion Without resolution robust experimental or quasi- experimental evaluations of these types of interventions are not possible “… it is nearly impossible to know what to do given the scarcity of good evidence and good evaluation of current policy. Policy cannot be evidence-based if there is no evidence and evidence cannot be obtained without proper evaluation. The most damning criticisms of Government policies we have heard in this inquiry have not been of the policies themselves, but rather of the Government’s approach to designing and introducing new policies which make meaningful evaluation impossible..” House of Commons Health Committee