Observational Health Data Sciences and Informatics (OHDSI): An International Network for Open Science and Data Analytics in Healthcare Patrick Ryan, PhD Janssen Research and Development Columbia University Medical Center 30 May 2016

Odyssey (noun): \oh-d-si\ A long journey full of adventures A series of experiences that give knowledge or understanding to someone http://www.merriam-webster.com/dictionary/odyssey

A journey to OHDSI

What is the quality of the current evidence from observational analyses? August2010: “Among patients in the UK General Practice Research Database, the use of oral bisphosphonates was not significantly associated with incident esophageal or gastric cancer” Sept2010: “In this large nested case-control study within a UK cohort [General Practice Research Database], we found a significantly increased risk of oesophageal cancer in people with previous prescriptions for oral bisphosphonates”

What is the quality of the current evidence from observational analyses? April2012: “Patients taking oral fluoroquinolones were at a higher risk of developing a retinal detachment” Dec2013: “Oral fluoroquinolone use was not associated with increased risk of retinal detachment”

What is the quality of the current evidence from observational analyses? BJCP May 2012: “In this study population, pioglitazone does not appear to be significantly associated with an increased risk of bladder cancer in patients with type 2 diabetes.” BMJ May 2012: “The use of pioglitazone is associated with an increased risk of incident bladder cancer among people with type 2 diabetes.”

What is the quality of the current evidence from observational analyses? Nov2012: FDA released risk communication about the bleeding risk of dabigatran, based on unadjusted cohort analysis performed within Mini-Sentinel Dec2013: “This analysis shows that the RCTs and Mini-Sentinel Program show completely opposite results” Aug2013: “However, the absence of any adjustment for possible confounding and the paucity of actual data made the analysis unsuitable for informing the care of patients”

Lessons from the OMOP experiments Database heterogeneity: Holding analysis constant, different data may yield different estimates Parameter sensitivity: Holding data constant, different analytic design choices may yield different estimates Empirical performance: Most observational methods do not have nominal statistical operating characteristics Empirical calibration can help restore interpretation of study findings Madigan D, Ryan PB, Schuemie MJ et al, American Journal of Epidemiology, 2013 “Evaluating the Impact of Database Heterogeneity on Observational Study Results” Madigan D, Ryan PB, Scheumie MJ, Therapeutic Advances in Drug Safety, 2013: “Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies” Ryan PB, Stang PE, Overhage JM et al, Drug Safety, 2013: “A Comparison of the Empirical Performance of Methods for a Risk Identification System” Schuemie MJ, Ryan PB, DuMouchel W, et al, Statistics in Medicine, 2013: “Interpreting observational studies: why empirical calibration is needed to correct p-values”

Lesson 5: Reliable evidence generation isn’t (just) a data/analysis/technology problem Understanding the problems requires input and perspective from multiple stakeholders: government, industry, academia, health systems Research and development of novel solutions require multi-disciplinary approach: informatics, epidemiology, statistics, clinical sciences Adoption and application requires active participation and buy-in from all interested parties (both evidence producers and evidence consumers) Major outstanding need: to establish a community of individuals based on shared attitudes, interests and goals where everyone has equal opportunity to participate and contribute

What’s the core problem? We have lots of DATA we’d like to learn from… ….and very little EVIDENCE we can actually trust

Why large-scale analysis is needed in healthcare All health outcomes of interest All drugs

Introducing OHDSI The Observational Health Data Sciences and Informatics (OHDSI) program is a multi-stakeholder, interdisciplinary collaborative to create open-source solutions that bring out the value of observational health data through large-scale analytics OHDSI has established an international network of researchers and observational health databases with a central coordinating center housed at Columbia University http://ohdsi.org

OHDSI’s mission To improve health, by empowering a community to collaboratively generate the evidence that promotes better health decisions and better care.

What evidence does OHDSI seek to generate from observational data? Clinical characterization Natural history: Who are the patients who have diabetes? Among those patients, who takes metformin? Quality improvement: what proportion of patients with diabetes experience disease-related complications? Population-level estimation Safety surveillance: Does metformin cause lactic acidosis? Comparative effectiveness: Does metformin cause lactic acidosis more than glyburide? Patient-level prediction Precision medicine: Given everything you know about me and my medical history, if I start taking metformin, what is the chance that I am going to have lactic acidosis in the next year? Disease interception: Given everything you know about me, what is the chance I will develop diabetes?

What is OHDSI’s strategy to deliver reliable evidence? Methodological research Develop new approaches to observational data analysis Evaluate the performance of new and existing methods Establish empirically-based scientific best practices Open-source analytics development Design tools for data transformation and standardization Implement statistical methods for large-scale analytics Build interactive visualization for evidence exploration Clinical evidence generation Identify clinically-relevant questions that require real-world evidence Execute research studies by applying scientific best practices through open-source tools across the OHDSI international data network Promote open-science strategies for transparent study design and evidence dissemination

OHDSI ongoing collaborative activities Methodological research Open-source analytics development Clinical applications Observational data management Data quality assessment Common Data Model evaluation ATHENA for standardized vocabularies WhiteRabbit for CDM ETL Usagi for vocabulary mapping HERMES for vocabulary exploration ACHILLES for database profiling Clinical characterization Phenotype evaluation CIRCE for cohort definition CALYPSO for feasibility assessment HERACLES for cohort characterization Chronic disease therapy pathways Population-level estimation Empirical calibration LAERTES for evidence synthesis CohortMethod SelfControlledCaseSeries SelfControlledCohort TemporalPatternDiscovery HOMER for causality assessment Patient-level prediction Evaluation framework and benchmarking PatientLevelPrediction APHRODITE for predictive phenotyping PENELOPE for patient-centered product labeling

OHDSI in action #1: Establishing open community standards for observational data management

The journey of the OMOP Common data model OMOP CDMv2 OMOP CDM now Version 5, following multiple iterations of implementation, testing, modifications, and expansion based on the experiences of the community who bring on a growing landscape of research use cases. OMOP CDMv4 OMOP CDMv5 Page 18

One model, multiple use cases Drug safety surveillance Comparative effectiveness Device safety surveillance Health economics Vaccine safety surveillance Quality of care Clinical research Standardized clinical data Person Standardized health system data Standardized meta-data Observation_period Location Care_site CDM_source Specimen Provider Death Concept Payer_plan_period Standardized health economics Standardized vocabularies Vocabulary Visit_occurrence Domain Visit_cost Procedure_occurrence Concept_class Procedure_cost Concept_relationship Drug_exposure Drug_cost Relationship Device_exposure Concept_synonym Device_cost Concept_ancestor Condition_occurrence Standardized derived elements Cohort Source_to_concept_map Measurement Cohort_attribute Drug_strength Note Condition_era Cohort_definition Observation Drug_era Dose_era Attribute_definition Fact_relationship

OHDSI community in action OHDSI Collaborators: >140 researchers in academia, industry, government, health systems >20 countries Multi-disciplinary expertise: epidemiology, statistics, medical informatics, computer science, machine learning, clinical sciences Databases converted to OMOP CDM within OHDSI Community: >50 databases >660 million patients

OHDSI in action #2: Clinical characterization of treatment pathways

ADA T2DM Guidelines, 2015

What treatments do patients in Canada actually use for their diabetes, hypertension, or depression? OHDSI asked this question around the rest of the world…

Network process Join the collaborative Propose a study to the open collaborative Write protocol http://www.ohdsi.org/web/wiki/doku.php?id=research:studies Code it, run it locally, debug it (minimize others’ work) Publish it: https://github.com/ohdsi Each node voluntarily executes on their CDM Centrally share results Collaboratively explore results and jointly publish findings

OHDSI in action: Chronic disease treatment pathways Conceived at AMIA Protocol written, code written and tested at 2 sites Analysis submitted to OHDSI network Results submitted for 7 databases 15Nov2014 30Nov2014 2Dec2014 5Dec2014

OHDSI participating data partners Code Name Description Size (M) AUSOM Ajou University School of Medicine South Korea; inpatient hospital EHR 2 CCAE MarketScan Commercial Claims and Encounters US private-payer claims 119 CPRD UK Clinical Practice Research Datalink UK; EHR from general practice 11 CUMC Columbia University Medical Center US; inpatient EHR 4 GE GE Centricity US; outpatient EHR 33 INPC Regenstrief Institute, Indiana Network for Patient Care US; integrated health exchange 15 JMDC Japan Medical Data Center Japan; private-payer claims 3 MDCD MarketScan Medicaid Multi-State US; public-payer claims 17 MDCR MarketScan Medicare Supplemental and Coordination of Benefits US; private and public-payer claims 9 OPTUM Optum ClinFormatics US; private-payer claims 40 STRIDE Stanford Translational Research Integrated Database Environment US; inpatient EHR HKU Hong Kong University Hong Kong; EHR 1 Hripcsak et al, PNAS, in press

Treatment pathways for diabetes T2DM : All databases Only drug First drug Second drug Hripcsak et al, PNAS, in press

Population-level heterogeneity Type 2 Diabetes Mellitus CCAE Hypertension CUMC Depression MDCD CPRD INPC GE JMDC MDCR OPTUM Hripcsak et al, PNAS, in press

OHDSI in action #3: Population-level estimation through open-source analytics

OHDSI’s approach to open science Generate evidence Data + Analytics + Domain expertise Open source software Enable users to do something Open science is about sharing the journey to evidence generation Open-source software can be part of the journey, but it’s not a final destination Open processes can enhance the journey through improved reproducibility of research and expanded adoption of scientific best practices

Standardizing workflows to enable reproducible research Open science Population-level estimation for comparative effectiveness research: Is <intervention X> better than <intervention Y> in reducing the risk of <condition Z>? Generate evidence Database summary Cohort definition Cohort summary Compare cohorts Exposure-outcome summary Effect estimation & calibration Compare databases Defined inputs: Target exposure Comparator group Outcome Time-at-risk Model specification Consistent outputs: analysis specifications for transparency and reproducibility (protocol + source code) only aggregate summary statistics (no patient-level data) model diagnostics to evaluate accuracy results as evidence to be disseminated static for reporting (e.g. via publication) interactive for exploration (e.g. via app)

Feasibility enabled in near-real-time through open-source applications

Community invitation to participate… that means you!

Protocol and source code made freely available to public BEFORE the analysis is completed

OHDSI in action #4: Patient-centered evidence dissemination

ALT TEXT: Baby in neonatal intensive care unit.

ALT TEXT: Product brochure for Synagis® (palivizumab) ALT TEXT: Product brochure for Synagis® (palivizumab). The brochure lists the indication “used to help prevent serious lung disease caused by respiratory syncytial virus (RSV) in children at high risk for severe lung disease from RSV”. It also lists ‘select safety information’: “common side effects of Synagis include fever and rash. Other possible side effects include skin reactions around the area where the shot was given (like redness, swelling, warmth, or discomfort)”. Doctor X: “This paper says there’s side effects, but I’ve never seen them happen”

ALT TEXT: Screenshot of the first page of the FDA-approved product labeling for Synagis® - palivizumab injection. The label lists under the ‘Warnings and Precautions’ section that “Analyphylaxis and anaphylactic shock (including fatal cases), and other severe acute hypersensitivity reactions have been reported.”

Pediatric patients across US observational databases source age group (at database entry) persons avg years of observation CCAE 1. newborn 0 to 27d 3,360,896 2.23 MDCD 1,862,651 1.74 Optum 1,473,940 1.82 2. infant and toddler 28d to 23mo 3,275,604 2.34 1,379,760 1.70 963,770 1.97 3. children 2 to 11 yo 14,904,293 2.46 4,037,836 1.77 4,951,888 2.18 4. adolescants 12 to 18 yo 12,218,224 2.41 2,565,515 1.57 3,805,609

Exploring palivizumab exposure and hypersensitivity in observational data data source age group (at time of exposure) persons exposed follow-up time (yrs) persons with outcome risk (events / 1000 persons); 95%CI CCAE 1. newborn 0 to 27d 1839 4829 0 (0 - 2.59) MDCD 381 760 0 (0 - 12.41) OPTUM 2610 5916 1 0.38 (0 - 2.45) 2. infant and toddler 28d to 23mo 42843 106320 27 0.63 (0.43 - 0.92) 19910 41196 17 0.85 (0.53 - 1.38) 22365 48632 11 0.49 (0.27 - 0.9) 3. children 2 to 11 yo 544 1525 1.84 (0 - 11.68) 265 706 0 (0 - 17.78) 204 574 0 (0 - 23.01) 4. adolescants 12 to 18 yo 38 93 0 (0 - 115.33) 33 28 0 (0 - 131.21) 44 0 (0 - 334.22) Back of the envelope: Assuming CCAE+MDCD+OPTUM represents 10% of US and exposures are evenly distributed across ~1000 NICUs, doctor would have seen ~50 newborns with exposure... even if the true event rate was 1%, there’s >60% chance they’d never see one case

Enter PENELOPE Personalized Exploratory Navigation & Evaluation Of Labels for Product Effects

Let’s Take a Look!

OHDSI: what does it mean to me? Methodological research Open-source analytics development Clinical applications Observational data management Where is there reliable data about the health of children? Clinical characterization Who are the children who are exposed to palivizumab? Population-level estimation Does palivizumab cause anaphylaxis in newborns? ALT TEXT: OHDSI focused on four analytical use cases: observational data management (to answer questions like: ‘where is there reliable data about the health of children?’), clinical characterization (descriptive analyses to answer questions like: ‘who are the children who are exposed to furosemide?’ and ‘how often to children exposed to furosemide experience acute renal disorder?’), population-level estimation (analytical studies for drug safety surveillance and comparative effectiveness research that address causal inference questions estimating average treatment effects for questions like: ‘does furosemide cause nephrocalcinosis in newborns?’), and patient-level prediction (modeling individualized risk for questions like ‘will my daughter by the one to develop nephrocalcinosis?’). To address these analytical use cases, OHDSI is collaborating on three fronts: methodological research (to learn and evaluate scientific best practices), open-source analytics development (to embed those best practices learned into standardized, reproducible statistical routines), and clinical applications (to apply those best practices through standardized tools to important clinical questions that patients and providers want the answers to). Patient-level prediction Will my daughter be the one to develop anaphylaxis?

Concluding thoughts Observational databases can be a useful tool for generating evidence to important clinical questions in… Clinical characterization Population-level estimation Patient-level prediction …but ensuring that evidence is reliable requires developing scientific best practices, and transparent and reproducible processes to conduct analyses across the research enterprise An open science community allows all stakeholders to contribute to and benefit from a shared solution…anyone can get involved…that mean’s YOU! Every patient, caregiver, parent and child deserves to know what is known (and what remains uncertain) from the real-world experience of others in order to inform their medical decision-making

Interested in OHDSI? Questions or comments? ryan@ohdsi.org Join the journey Interested in OHDSI? Questions or comments? ryan@ohdsi.org