Analysing RWE for HTA: Challenges, methods and critique Rita Faria Research Fellow Centre for Health Economics, University of York

The decision Information required The ideal study The reality What is the cost-effective treatment? The decision Effect of treatment In the patients seen in clinical practice Over the short- and long-term Relative to other treatments available. Information required RCT or meta-analysis of RCTs In the population of interest Over the relevant time horizon Compared with the other treatment options The ideal study RCTs may be unfeasible. RCT may be in a selected subgroup, over a short time horizon and using a subset of comparators. The reality Decision modelling Real world evidence: non-randomised studies, observational data, administrative databases Potential solutions Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Challenges 2013 NICE Methods guide: 3.3.6: “The potential biases of observational studies should be identified, and ideally quantified and adjusted for. When possible, more than one independent source of such evidence should be examined to gain some insight into the validity of any conclusions.” Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Association ≠ Causation Is shaving good for your health? HR for stroke=1.90 (95%CI 1.37-2.70) Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Association, causation & confounders Stroke Infrequent shaving Confounder  Selection bias X Risk factor for stroke Infrequent shaving may be a confounding variable. Conditional on X, there may be no association between stroke and infrequent shaving. RWE is at high risk of selection bias. Selection bias arises from differences in the characteristics that have an independent influence on the outcome between the individuals in the treated and the control groups. TSD17 p13. Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

NICE DSU TSD 17 on non-RCT data What to do in practice? How to know it is right? Methods & algorithm for selection Does treatment selection ONLY depends on the observed data? OVERLAP: Are treatment groups comparable? QuEENS checklist for critical appraisal. Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Objective: obtain the true (unbiased) effect of treatment. Methods Selection on observables Regression adjustment Inverse probability weighting Matching (±regression; w/o propensity score) Regression on propensity score Doubly robust Objective: obtain the true (unbiased) effect of treatment. Selection on unobservables Instrumental variables Natural experiments Differences-in-Differences Regression discontinuity Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Critical appraisal: QuEENS checklist Selection on observables General issues Robustness of results Consistency across studies Statistic validity Valid assumptions Matching methods Methods using propensity score Instrumental variable methods Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.

Final recommendations Remember the objective: obtain the true (unbiased) effect of treatment. Data Assumptions Methods Sensitivity analysis Reporting Analysing RWE for HTA: Challenges, methods and critique. Rita Faria, Research Fellow, University of York.