1. Michael Putman, MD Rheumatology Fellow RWCS 2019
The Quality of Randomized Controlled Trials in the Rheumatologic Literature from
Michael Putman, MD
Rheumatology Fellow
RWCS 2019

2. Background-imumab
Rheumatology has been inundated by new therapies…

3. Background-failimab?
…yet somehow the majority of our guideline recommendations are level C
JAMA Intern Med. 2018 Jan 1;178(1):

4. Guidelines Background There are a LOT of these too
How can we have so many therapies yet so few high level recommendations?
Guidelines
RCTs
Obs. Studies
Expert Opinion
There are a LOT
of these too
There are a LOT of these
Old adage: “if you ask five rheumatologists for an opinion you’ll get 7 answers back”
(I’m not one
of them)

5. What kinds of questions
Background
Let’s focus on the randomized controlled trials (RCTs)
How good are they?
Are they
getting better?
…or worse?
RCTs
How strong are
their findings?
Who funds them?
What kinds of questions
are they asking?

6. Background
How do we know what’s a good trial and what’s not?

7. Identification of primary outcomes
Background
How do we know what’s a good trial and what’s not?
Blinding
Identification of primary outcomes
Patient reported outcome measures (PROMs)
Power calculation
Sensitivity analysis
Adjustment for multiple hypothesis testing
Intention to treat analysis (ITT)

8. Methods
Overview
We identified the top 3 clinical rheumatology journals by h5-index, Scimago Journal and Country Rank, and Emeunet 2016 Top Journals assessment

9. Methods
Overview
All clinical RCTs reporting primary analysis of a pharmacologic intervention against a comparator in 1998, 2008, and 2018 from selected journals were included
Intervention, metrics of RCT quality, absolute risk estimates, p-values, and funding were recorded
A “quality scale” was constructed from the quality metric variables and normalized to a value from 1-10
Bivariate associations between variables were assessed using Chi2 testing for categorical variables and either independent samples t-test or one way ANOVA for continuous

10. 3,338 Titles Identified 3,338 Titles Identified
Methods
Biting off more than I meant to chew
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included

11. 3,338 Titles Identified 3,338 Titles Identified
Methods
Journals and years
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included

12. 3,338 Titles Identified 3,338 Titles Identified
Results
Has the quality of journals changed over time?
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included

13. 3,338 Titles Identified 3,338 Titles Identified
Results
Does industry involvement impact the quality of a trial?
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included

14. * * * 3,338 Titles Identified 3,338 Titles Identified
Results
If the quality hasn’t changed much over time, what has?
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included * * *

15. * * * 3,338 Titles Identified 3,338 Titles Identified
Results
Bigger is better? Rheumatology rocks? Publication bias?
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included * * *

16. * * * 3,338 Titles Identified 3,338 Titles Identified
Limitations
“Project does not currently meet our standards for publication”
Only assessed 3 years out of 20, may not be generalizable
Review by co-author has not been completed
Not powered to detect small differences between groups
Interpretation of quality metrics may be subjective
Limited by reporting from studies; actual study quality may not reflect that described in study methods
Publication patterns may vary over time and high impact general interest publications were not assessed
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included * * *

17. * * * 3,338 Titles Identified 3,338 Titles Identified
Conclusions
There are a whole lot of RCTs out there
Intention to treat (ITT) analysis has become more common, but RCT quality by other metrics has remained constant.
Industry funds the vast majority of studies. Industry studies are significantly more likely to be appropriately blinded, report PROMs, use ITT, and have a higher overall quality.
Over time, fewer studies have included an active comparator and studies with significant primary outcomes (p < 0.05) have become significantly less common.
Smaller studies were associated with larger effect sizes. Overall, rheumatologic interventions have a high ARR.
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included
3,067 Titles Did Not Meet Inclusion Criteria
186 Did Not Meet Criteria After Full Review
3,338 Titles Identified
85 Papers included * * *

18. Thank You