Outline I have trouble working with ODBs Patient  Measure  DB Input & Processing  Analyses Each step has noise and bias Patients report differently based on background characteristics Measurements a well known error and sometimes bias source DBs have structures and conventions for purposes different from the ones our queries may fit with DB input & processing (coding conventions, reimbursement policy) Analyses try to anticipate these but may make it worse

DB Input & Processing A database poorly approximates the reality that we purport it captures (given our query) because … databases arise in a larger system with rules that can be at odds with both the overt reason for its existence and the question we want to answer people alter inputs to counter the larger system (say hosptital or payer) rules so they can get their work done chronological trends such as changes in reimbursement policies or, in the case of drugs, prescribing rules

Why worry? Analyses will fix it! Stakeholders may care: patients, providers, regulators … Many observational database studies (ODS) published each day Safety of a drug requires knowledge gained from multiple sources Electronic health records (EHR), claims data, and spontaneous report systems (SRS) provide sources beyond clinical trials (CT/RCT) – with >>> N’s

Besides, RCTs are overated, and unable to answer questions of sample size or ethical sensitivity https://www.sciencedirect.com/journal/social-science-and-medicine/vol/210/suppl/C

Level of problem Simulation work by Franklin et al (2014) drug studies in claims data may suffer from bias due to residual confounding (Brookhart et al 2010) Methods for adjustment need testing Covariates can help to a point

Exploring EHRs in their own right Hripcsak & Albers (2013) Propose studying EHRs as a subject unto itself Review of data collection protocols Testing of new methods to correct data Step one – harmonize at a basic level

OHDSI: OMOP common data model To allow the systematic analysis of disparate observational databases Transform data contained within various databases into a common format (data model) Use common representation for terminologies, vocabularies, coding schemes Additionally have a library of standard analytic routines that have been written based on the common format for systematic analyses of the combined data

Sentinel Establishment of the Sentinel Operations Center Sentinel’s Major Accomplishments (from https://www.sentinelinitiative.org/background) Establishment of the Sentinel Operations Center Creation of a common data model and distributed data approach that enables FDA to monitor the performance of medical products while securing and safeguarding patient privacy Development of a distributed database with more than 300 million person-years of high quality, unduplicated, curated data

Sentinel Development of processes for turning FDA’s safety concerns into queries of the Sentinel distributed data that can be responded to rapidly by Data Partners, often within weeks, in support of FDA’s regulatory needs Substantial progress toward development of a mature data analytics system Recruitment of a broad group of scientific collaborators who regularly provide the FDA with valuable technical support in evaluating electronic health data

Sentinel Development of focused surveillance efforts around vaccine safety with the Postmarket Rapid Immunization Safety Monitoring (PRISM) system and around blood and blood products with the Blood Surveillance Continuous Active Network (BloodSCAN)

Some other initiatives http://www.stratos-initiative.org/ Focus on design and analysis https://pcornet.org/ Has a common data model Network of data sources Appears many other individuals and groups focus on some of these problems, but not clear how many go into policy and initial database purpose issues

Levels of harmony 1) data are in the same form and coding 2) policies are the same 3) patients and HCPs act similarly Next example has 1 and maybe 2

Obvious test interpretations? Agneil et al Two hospital EHR systems Use 272 common lab tests’ data to predict survival in ~670000 treated patients over 1 year Compare predictive ability of actual test lab value versus the time of the day, day of the week, and ordering frequency of the test

Not obvious lab test interpretations “The presence of a laboratory test order, regardless of any other information about the test result, has a significant association (P<0.001) with the odds of survival in 233 of 272 (86%) tests. Data about the timing of when laboratory tests were ordered were more accurate than the test results in predicting survival in 118 of 174 tests (68%).”

How bad can biases get? Illinois Wellness Study Population – employees in the UI system Methods – randomized to get invited to “wellness” program or not Health outcomes followed for a year (and more) Analyzed two ways – based on randomization and using an observational subgroup

How bad can biases get? Randomized study analyses found no effect of program on health outcome one year after initiation When analyzed using only the “observational” cohort, results favored the wellness program

example in my work Coding Processing Interpretation Automation

Back to things to try Hripcsak & Albers (2013) Propose studying EHRs as a subject unto itself Review of data collection protocols Testing of new methods to correct data