© Nuffield Trust 22 June 2015 Matched Control Studies: Methods and case studies Cono Ariti

© Nuffield Trust Predictive risk modelling Resource allocation Descriptive studiesEvaluations Integrated care pilots nuffield trust Nuffield Trust Research team – data linkage projects Person based resource allocation nuffield trust Virtual Wards nuffield trust WSD nuffield trust British Red Cross nuffield trust

© Nuffield Trust Need for evaluation Need to know what works In a practical setting – “real world evaluation” Clarify the debate Likely impacts – unbiased results Link to qualitative work Refine programs Obtain feedback and learnings – the pain of implementation Explore sub-groups – where did it work? Where could it work?

© Nuffield Trust Issues with evaluations Randomised control trials “Gold standard” May not be feasible or ethical Inclusion and exclusion rules can limit generalisation Are still subject to poor implementation – can induce bias Potentially expensive! Observational studies Typically no “natural” experiment exists Often no comparable control group to provide a fair assessment

© Nuffield Trust Matched Control Studies - Methods

© Nuffield Trust Matched Control Studies The basic idea Match controls to those treated based on measured characteristics in existing datasets The control group and treated group should look similar “on balance” Mimics the idea of an RCT Based on propensity score theory (Rubin & Rosenbaum, 1983) and earlier work on matching (Cochran, 1965)

© Nuffield Trust Matched Control Studies Matching Prognostic risk score Demographics – age, gender, deprivation, ethnicity Prior acute care service use – admissions, OP and A&E attendances Prior diagnoses, targeted chronic conditions Balance In this case all matching variables Additional variables such as length of stay, additional diagnoses and longer service use history Assures comparability between the groups

© Nuffield Trust Matching Algorithm Algorithm Exact match not possible Computer intensive “genetic algorithm” Uses a weighted Mahalanobis “distance” to determine closest match Automatically assesses balance and moves to an improved solution Assessing Balance On overall group similarity Compares means and distribution of variables in the two groups

© Nuffield Trust Analysis of matched control studies Standard statistical methods to estimate the difference in the two groups Regression models, difference in difference analysis By including matching variables in the statistical adjustment remaining imbalances can be reduced – “doubly robust” Methods exist for sensitivity analysis – impact of unobserved variables Some controversy around accommodating the matching in analysis

© Nuffield Trust Case Study 1: Telehealth Programme

© Nuffield Trust Case Study 1: Telehealth program Intervention: Remote monitoring for patients with long term conditions Nuffield commissioned to evaluate impact: Primary: Reduction in emergency hospital admissions? Secondary: Reductions in Emergency attendances, outpatient attendances, mortality Methods: Retrospective matched control study – use of already existing administrative data

© Nuffield Trust Description: Telehealth program

© Nuffield Trust Matched control studies – broad aim >30,000 individuals – resident in local area June 2010 to March 2013, did not receive telehealth and were eligible for matching (local controls) Aim to find 716 individuals who match almost exactly on a broad range of characteristics Use this group as study control group 716 individuals – enrolled June 2010 to March 2013 & received Telehealth intervention & eligible for matching

© Nuffield Trust Datasets available TelehealthNuffield trust N = 716 person details dates of service type of service Identifiers: Names, DOB, Addresses, etc dates & place of death for all people in England, associated hospital (HES) records Identifiers: Nuffield Trust specific HESID Administrative dataONS deathsHospital inpatient, outpatient, AE

© Nuffield Trust TelehealthData Linkage ServiceNuffield Trust New Identifier (NHS no) Names Address DOB HESID Telehealth person identifiers (File A)

© Nuffield Trust Final datasets available for analysis Nuffield trust Identifiers: HESID on all ONS deathsHospital inpatient, outpatient, AETelehealth data - desensitised Use all this info to carry out matched control analysis

© Nuffield Trust Control group – how well matched?

© Nuffield Trust Control group – how well matched? Telehealth Controls

© Nuffield Trust Control group – how well matched? Telehealth Controls

© Nuffield Trust Key Result 1: Risk of admissions or death

© Nuffield Trust Key Result 2: Changes in admissions or attendances (six months pre and post intervention)

© Nuffield Trust Results Telehealth patients tended to be admitted for an emergency admission earlier than control patients There was no difference in mortality between the telehealth and control groups There were no statistically significant reductions in hospital admissions when comparing the period six months before and after the telehealth intervention In summary the Telehealth program did not have a significant impact on acute care outcomes Sensitivity analysis showed little evidence of an important unobserved variable

© Nuffield Trust Matched Control Studies: Summary

© Nuffield Trust Matched Controls: Summary Benefits Makes full use existing data, with relative ease Techniques applicable to many different types of services and datasets Decisions on what seems to work (and what may not) based on more robust analyses leading to better informed decisions Caveats If important unobserved variables exist results may be biased The routine data sources must contain the relevant data

© Nuffield Trust Implementing locally – key enablers Do you have …Can you … Access to data that contains the outcomes relevant to your evaluation? Access to data containing relevant matching characteristics? Do you have consent to access/link the data? Analysis tools to apply statistical methods to the data? Skilled analysts to analyse the data? Link multiple sources of data? Handle large amounts of data (millions of observations)? Identify recipients of the intervention? Transform and augment that data with bespoke variables? Apply sophisticated matching algorithms routinely to this data? Analyse the data with a variety of statistical methods and interpret the results appropriately?

© Nuffield Trust Sign-up for our newsletter Follow us on Twitter: Twitter.com/NuffieldTrust © Nuffield Trust Thank you