1. Martijn Schuemie, PhD Janssen Research and Development
Method evaluation
Martijn Schuemie, PhD
Janssen Research and Development

2. Selected research topics
Topic 1: Method evaluation
Have initiated the Method Evaluation Task Force
Topic 3: Smooshed comparators
Collaboration between UPenn and JnJ
Topic 5: Heterogeneous treatment effects
Will be put on hold until 1 is completed

3. Method evaluation Population-level effect size estimation:
What is the relative risk of outcome X when using drug A?
What is the relative risk of outcome X when using drug A compared to drug B?
How accurate and reliable are estimates of a particular method?

4. Method evaluation state-of-the-art
Main evaluation of HDPS:
Selective Cox-2 inhibitors versus nsNSAIDs for GI complications
Unadjusted: RR = 1.09 (0-91 – 1.30)
Age-sex-race-year: RR = 1.01 (0.84 – 1.21)
+ HDPS: RR = 0.87( )
Statins and 1 year mortality
Unadjusted: RR = 0.56 (0-51 – 0.62)
Age-sex-race-year: RR = 0.77 (0.69 – 0.85)
+ HDPS: RR = 0.80 (0.70 – 0.90)
Limitations:
n = 2 (1 showing no effect at all)
Effects were well known so could affect practice
What was the true effect size?
Was the evaluation tailored to the method?

5. OMOP 2011/2012 experiment Drug-outcome pairs Methods Case-Control
New User Cohort
Disproportionality methods
ICTPD
LGPS
Self-Controlled Cohort
SCCS
Observational data
Thomson MarketScan GE

6. Lessons learned from OMOP
Issues with the OMOP experiment(s):
Problem with know positive controls (especially contra-indication) that artificially favors self-controlled designs
Limitations in methods library
Forgot to censor in SCC
MSCCS applied shrinkage to exposure of interest
CohortMethod did not correct for differences in length of follow up
Uncertainty around positive and negative status
Don’t know true RR if RR != 1

7. Method Evaluation Task Force
Objectives:
Develop the methodology for evaluating methods
Use the developed methodology to systematically evaluate a large set of study designs and design choices

8. Task force members Alejandro Schuler Mike Goodman Anthony Sena
Nicole Pratt
Brian Saur
Niham Shah
Chan YouSeng
Patrick Ryan
David Madigan
Peter Rijnbeek
Eldar Allakhverdiiev
Rae Woong Park
George Hripcsak
Sara Dempster
Jamie Weaver
Teng Liaw
Marc Suchard
Vojtech Huser
Martijn Schuemie
Yuxi Tian

9. Getting philosophical
Suitability of a method
In general?
For a specific clinical question?

10. Must specify 3 components
Gold standard
(Exposure-Outcome pairs)
Methods
Data

11. Gold standard
Real negative controls
Synthetic positive controls
RCTs

12. Negative control evaluation
Select trials from clinicaltrials.gov that are
Randomized
Placebo-controlled
Report number of (potential adverse) events
Apply criteria for negative controls to intervention-event pairs
Compute odds ratios for negative controls

13. RCT estimates for negative controls

14. Negative controls Pro: Probably unbiased
Includes measured and unmeasured confounding
Con:
Null effects only

15. Synthetic positive (and negative) controls
Two options:
Injecting outcomes on top of real negative controls
Generating all outcomes through simulation

16. Injecting outcomes on negative controls
Target
Patient 1
Comparator
Patient 2
Target
Patient 3
Comparator
Patient 4
Target
Predictive model of outcome indicates this is a high-risk patient
Patient 5
Comparator
Patient 6
New RR = 2 (but with same observed confounding)
Ingrowing nail
Injected ingrowing nail

17. Simulating all outcomes
Target
Patient 1
Comparator
Patient 2
Target
Patient 3
Comparator
Patient 4
Target
Patient 5
Comparator
Patient 6
RR = 2 (with defined confounding structure)
simulated ingrowing nail

18. Comparisons made Injection on negative controls
new set of outcomes during risk period vs
original set of outcomes during risk period
Generating all outcomes
new set of outcomes during risk period target vs
new set of outcomes during risk period comparator

19. Pros and cons Injection on negative controls Pro Con
Real unmeasured confounding in negative controls
One evaluation set for all methods (CohortMethod, SCCS, Case-Control, etc.)
By picking appropriate negative controls can generate gold standard relevant for specific clinical questions
Con
No unmeasured confounding in injected outcomes
Assumes negative controls are really negative
Simulating all outcomes
Certain that simulated effect is exactly true
We can simulate unmeasured confounding by dropping measured variables
Can use non-negative exposure-outcome pairs as basis
Simulate gold standard relevant for specific clinical question
Need separate simulation strategies for each method
Replaces real confounding structure with simplified one

20. RCTs Selection criteria
Large trials only, so the result of the trial has some accuracy which is nice in a gold standard 
Recent trials, so we can restrict our observational data to time prior to when the results of the trial were known
(Related to the previous criterium:) Only trials of drugs that were already on the market at the time of the trial, so we actually have observational data prior to the trial
Trials with inclusion/exclusion criteria we can also apply to our replication.

21. Method evaluation tasks
Identify exposures of interest and negative controls 
Decide approach to positive control synthesis 
Identify RCTs and implement inclusion criteria 
Implement case-crossover / case-time-control 
Define universe of methods to evaluate 
Identify list of databases to run on 
Develop evaluation metrics 
Implement and execute evaluation 
Write paper

22. Next workgroup meeting
Western hemisphere: April 13
6pm Central European time
12pm New York
9am Los Angeles / Stanford
Eastern hemisphere: April 5
3pm Hong Kong / Taiwan
4pm South Korea
4:30pm Adelaide
9am Central European time
8am UK time