1. Constructing OHDSI network studies
Marc A. Suchard

2. A journey from data set to paper
Most epidemiologists view a study as a journey from data set to paper.
The protocol might be your map
You will come across obstacles that you will have to overcome
Several steps will require manual intervention
In the end, it will be impossible to retrace your exact steps

3. Current epi studies are non-reproducible
How do we know what happened?
How do we know if it was done correctly?
How do we know how well it worked?
How could we be more efficient?
How can we deal with more complex studies?
How can multiple people work together on the same analysis?
How could other reproduce this study on a different database?

4. What should OHDSI studies look like?
Paper
Database
A study should be like a pipeline
A fully automated and reproducible process from database to paper
‘Performing a study’ = building the pipeline

5. Distributed research network
Many observational databases in OHDSI
large numbers
large diversity
We cannot share patient-level data
Solution:
analysis code ‘visits’ the data
only population-level data is shared

6. Hub and spoke network Data site A Data site F Data site B
Coordinating center
Data site E
Data site C
Data site D

7. OHDSI network Stanford UCLA IMS Columbia University
University of
Hong Kong
Taipei Medical University
Regenstrief
University of South Australia
Janssen
Ajou School of Medicine

8. Treatment pathway study
Stanford
UCLA
IMS
Columbia University
University of
Hong Kong
Taipei Medical University
Regenstrief
University of South Australia
Janssen
Ajou School of Medicine

9. Everyone can initiate and lead a study
See the Wiki Collaborative Study FAQ:
Post preliminary protocol on Wiki
Invite community review
Post final protocol on Wiki
can be used for IRB approval
Develop study code, post on GitHub
Test code at at least 2 sites
Invite sites to join

10. Implementation Standards:
Study coordinator
Data site
Standards:
PostgreSQL, Oracle, SQL Server, RedShift, or APS
OMOP Common Data Model
Windows, macOs, Linux R

11. Implementation Mini-Sentinel: SAS EU-ADR: Java application (Jerboa)
Study coordinator
Data site
Content:
R package
Delivery:
GitHub (StudyProtocols repo)
E.g.
Mini-Sentinel: SAS
EU-ADR: Java application (Jerboa)
Why R?
Open source
Efficient in deploying advanced computing code
Easy to integrate different modules
Can we written by person ≠ Marc

12. Implementation Content: zip file containing Delivery: Plain text
Study coordinator
Data site
Content: zip file containing
Plain text
CSV (comma-separated values)
PNG (plots) …
Needs to be:
Non-identifiable information
Human reviewable
Delivery:
Amazon S3

13. Study package User experience Under the hood
Examples: Keppra-angioedema study packages

14. Study package user experience
Steps for the user:
Install the package (and dependencies)
Run the analyis
Review the results
Share the results

15. Study package user experience
Steps for the user:
Install the package (and dependencies)
Run the analyis
Review the results
Share the results

16. Installing the package
Using R’s package infrastructure to deploy packages Not all packages are in CRAN. There’s an OHDSI package repository (using “drat”)

17. Study package user experience
Steps for the user:
Install the package (and dependencies)
Run the analyis
Review the results
Share the results

18. Running the analysis
Ideally a single command to run the entire analysis This should create a folder called ‘export’ with all files that will be shared

19. Study package user experience
Steps for the user:
Install the package (and dependencies)
Run the analyis
Review the results
Share the results

20. Review the results

21. Study package user experience
Steps for the user:
Install the package (and dependencies)
Run the analyis
Review the results
Share the results

22. Share the results Push the StudyResults.zip file to an S3 bucket
S3 is Amazon’s storage service with secure up and download
One bucket per study
Two accounts per bucket:
Study coordinator can read and write
Study participants can write
Contact the coordinating center for your study-specific bucket

23. Under the hood Executing the analysis typically entails:
Instantiating cohorts
Running some SQL against the CDM database
Do some further processing in R
Write results to the export folder

24. Example: Keppra – angioedema study
OHDSI study:
Does exposure to Keppra (levetiracetam) lead to an increased risk of angioedema?
Compared to phenytoin

25. Full traceability Study package contains
Cohort definitions (e.g. angioedema definition)
OhdsiRTools::insertCohortDefinitionInPackage(2193, "Angioedema")
All analysis details for the CohortMethod package
CohortMethod package describes data extraction
Code to generate tables and figures
Code to generate full report

26. We can check for correctness
We can review the study code
We should make the study code publicly available as part of the paper
Large parts of the study are automatically checked using unit tests

27. We can evaluate how well the study worked
Included 100 negative control outcomes
Results show little residual confounding when using propensity score matching

28. Writing the study was very efficient
Reuse of R code in CohortMethod, DatabaseConnector, SqlRender, EmpiricalCalibration, etc.
Implementation took days instead of months
Next study will be faster

29. Complexity is not a problem
Use software engineering approaches to deal with complexity:
Abstraction
Encapsulation
Writing clear code
Re-use

30. Several people can work on the same analysis through version control
Mary
Version 2
History of changes: How did the final analysis come about?
Version
control
system
Version 1
Version 2
Use for everything:
Protocol
Analysis code
Paper X Me
Version 1
Version 2
Version 3
Version 2
Time

31. Commit log

32. Easy to rerun on different data
The Keppra – Angioedema study was run on:
Columbia University EHR
Stanford EHR
Cerner (University of Texas)
Pharmetrics Plus (IMS)
Optum
Truven CCAE
Truven MDCD
Truven MDCR …

33. Viewing a study as a pipeline has many advantages
Full traceability
Ability to check for correctness
Ability to evaluate using controls
More efficient
Ability to deal with complexity
Ability to work with several people on one analysis
Easy to rerun on different data

34. Study package structure
R contains all user-executable R code

35. Study package structure
extras contains code used by the study coordinator:
Code for importing CIRCE definitions
Testing code
Code for aggregating data across sites

36. Study package structure
inst is for non-R content of the payload
SQL
Settings files …

37. Study package structure
DESCRIPTION tells R about package dependencies
(E.g. methods in the methods library)

38. Study package structure
README.md is what is shown in the GitHub front page

39. Essentials
Reading
OHDSI Collobarative Studies FAQ:
R packages manual:
SqlRender vignette:
OHDSI R and SQL code styles:
Sites:
SqlRender test site:
Contacts:

40. Conclusions Anyone can initiate a study and be a study coordinator
Follow the steps described in the FAQ
Need to have a clear research question
Need someone with R and SQL skills
OHDSI standards for distributed analysis are evolving
Check latest study package for most recent ideas
Martijn and Marc can help with the writing of the package
End-product: fully reproducible (and replicable) study with demonstrated operating characteristics