1. Advancing Health Economics, Services, Policy and Ethics Collecting Real World Evidence: HTA’s perspective Dr. Kelvin Chan, MD FRCPC MSc (Clin Epi) MSc (Biostats) Clinical Lead, Provincial Drug Reimbursement Programs, CCO Co-Director, Canadian Centre for Applied Research in Cancer Control Medical Oncologist, Sunnybrook Odette Cancer Centre

2. What types of uncertainty do HTA committees encounter in their deliberations? How often does pCODR Expert Review Committee (pERC) request collecting evidence to reduce uncertainty? What types of information can be potentially obtained by collecting real world evidence (RWE)? Contents

3. Disclaimer Member of HTAs pCODR Expert Review Committee Committee to Evaluate Drugs Ontario Steering Committee of Cancer Drugs Personal opinion Does not represent the views of U of Toronto, pCODR, CED, OSCCD, CCO, Ministry etc. Information from publicly available source

4. Ontario Cancer Plan IV (2015-2019)

5. Health Technology Assessment (HTA) Committee’s Recommendations

6. Recent Example

7. In general, HTA committees (pCODR, CED, OSCCD) consider the following inputs: Clinical benefit (from clinical trial data) –Efficacy (survival) data –Safety data –Quality of Life (QOL) data Patient values Cost-effectiveness and budget impact Adoption feasibility Things that HTA committees consider before making a recommendation to fund (or not to fund)

8. Uncertainty in clinical data –Survival data Missing or limited Surrogate of overall survival Non-comparative Short term data –Safety data Missing data/Late data –Quality of life data Missing data (numerous examples) Uncertainty in clinical data

9. Non-Comparative Data Romidepsin in Peripheral T-Cell Lymphoma “It was noted that due to the limitations of relying on non-comparative, non-randomized evidence and the heavy reliance on extrapolation of overall survival data, there was substantial uncertainty in the magnitude of the net clinical benefit associated with romidepsin.” - pCODR Expert Review Committee Final Recommendation Adapted from Piekarz et al. BLOOD 2011Adapted from Coiffier et al. JCO 2012

10. Missing Quality of Life Data Vismodegib in Basal Cell Carcinoma “… quality of life and functional outcomes are very important in this population. Patients with BCC who are inappropriate for surgery may experience severe disfigurement, leading to extreme social isolation and decreased quality of life.” - pCODR Expert Review Committee Final Recommendation Survival is main a concern for patients with locally advanced or metastatic disease Disease progression may lead to facial disfigurement, and thus a decreased QoL This study presents no QoL data, however researchers connected a response and decreased tumour size to an improvement in overall QoL Adapted from Sekulic et al. NEJM 2012

11. Cost-effectiveness analysis and budget impact analysis Model structure and methods –Comparator and long term clinical efficacy Uncertainty in the inputs of the model –Number of patients –Duration of drug treatments –Resource utilization Underestimation of ICER (sometimes 1-2 fold difference) Underestimation of BIA Economic evidence: Estimation or “guess-timation”

12. Short Term Survival Data Pembrolizumab in Unresectable Metastatic Melanoma “In the absence of longer term data, pERC was unable to accept this assumption of prolonged benefit and agreed with the EGP’s use of alternative data sources to extrapolate survival in both settings.” - pCODR Expert Review Committee Final Recommendation Adapted from Robert et al. NEJM 2015

13. Trial patients are different from real world patients –e.g. age, co-morbidities, performance status, diffusion –Intensive monitoring on trial Different practice patterns in the real world –Duration of treatment (treatment until progression vs. discontinuation before progression) –Dose intensity of treatment in the trials vs. in the real world –Management of side-effects (e.g. febrile neutropenia are managed mostly as in-patient in Ontario) Sequencing of subsequent lines of available therapy –Different from what was available on the trial Changes in drug price over time (e.g. generics) –Drop in price in the older drugs will make the new drug less cost- effective Loss in “Translation”: Uncertainty about Translating clinical trial evidence to the real world

14. EXPERIENCE OF pCODR EXPERT REVIEW COMMITTEE (pERC)

15. Requesting Real World Evidence pCODR 60 reviews (Up to Feb 2016) Total of 21 pCODR reviews requested Real World Evidence 13 pCODR reviews explicitly requested Real World Evidence 10 pCODR reviews potentially requested Real World Evidence Potential RWE Request: Unclear if pERC requested RWE, but it could be beneficial

16. Next Steps for Real World Evidence Collection 23 requests for RWE for 21 studies

17. Breakdown of pCODR Reviews 60 Final Recommendations (4 requested RWE) (17 requested RWE)

18. Temporal Trends in RWE Requests 60 pERC Final Recommendations

19. Breakdown by Tumour Site 60 pERC Final Recommendations

20. Breakdown by Study Characteristics: 21 Studies DSV (Decrease in Spleen Volume), MCyR (Major Cytogenetic Response), MaHR (Major Haematological Response)

21. WHAT REAL WORLD EVIDENCE CAN WE COLLECT?

22. Potential Deliverables (effectiveness, safety, quality of life, cost-effectiveness)

23. Potential Deliverables (verify economic models)

24. Discussions HTA committees commonly make best possible recommendations based on substantial uncertainties pCODR ERC commonly requests for the collection for further evidence to reduce uncertainties –How can further evidence be collected? Routinely collected data (population-based admin data) Evidence building program (prospective collection of data) Real world experiments (real world pragmatic randomized trials) –Who will use this evidence once collected? Practitioners and patients – informed treatment decisions Policy decision-makers and payers Re-HTA (recommendation-makers)