Michael Putman, MD Rheumatology Fellow RWCS 2019 The Quality of Randomized Controlled Trials in the Rheumatologic Literature from 1998-2018 Michael Putman, MD Rheumatology Fellow RWCS 2019

Background-imumab Rheumatology has been inundated by new therapies…

Background-failimab? …yet somehow the majority of our guideline recommendations are level C JAMA Intern Med. 2018 Jan 1;178(1):146-148.

Guidelines Background There are a LOT of these too How can we have so many therapies yet so few high level recommendations? Guidelines RCTs Obs. Studies Expert Opinion There are a LOT of these too There are a LOT of these Old adage: “if you ask five rheumatologists for an opinion you’ll get 7 answers back” (I’m not one of them)

What kinds of questions Background Let’s focus on the randomized controlled trials (RCTs) How good are they? Are they getting better? …or worse? RCTs How strong are their findings? Who funds them? What kinds of questions are they asking?

Background How do we know what’s a good trial and what’s not?

Identification of primary outcomes Background How do we know what’s a good trial and what’s not? Blinding Identification of primary outcomes Patient reported outcome measures (PROMs) Power calculation Sensitivity analysis Adjustment for multiple hypothesis testing Intention to treat analysis (ITT)

Methods Overview We identified the top 3 clinical rheumatology journals by h5-index, Scimago Journal and Country Rank, and Emeunet 2016 Top Journals assessment

Methods Overview All clinical RCTs reporting primary analysis of a pharmacologic intervention against a comparator in 1998, 2008, and 2018 from selected journals were included Intervention, metrics of RCT quality, absolute risk estimates, p-values, and funding were recorded A “quality scale” was constructed from the quality metric variables and normalized to a value from 1-10 Bivariate associations between variables were assessed using Chi2 testing for categorical variables and either independent samples t-test or one way ANOVA for continuous

3,338 Titles Identified 3,338 Titles Identified Methods Biting off more than I meant to chew 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included

3,338 Titles Identified 3,338 Titles Identified Methods Journals and years 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included

3,338 Titles Identified 3,338 Titles Identified Results Has the quality of journals changed over time? 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included

3,338 Titles Identified 3,338 Titles Identified Results Does industry involvement impact the quality of a trial? 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included

* * * 3,338 Titles Identified 3,338 Titles Identified Results If the quality hasn’t changed much over time, what has? 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included * * *

* * * 3,338 Titles Identified 3,338 Titles Identified Results Bigger is better? Rheumatology rocks? Publication bias? 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included * * *

* * * 3,338 Titles Identified 3,338 Titles Identified Limitations “Project does not currently meet our standards for publication” Only assessed 3 years out of 20, may not be generalizable Review by co-author has not been completed Not powered to detect small differences between groups Interpretation of quality metrics may be subjective Limited by reporting from studies; actual study quality may not reflect that described in study methods Publication patterns may vary over time and high impact general interest publications were not assessed 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included * * *

* * * 3,338 Titles Identified 3,338 Titles Identified Conclusions There are a whole lot of RCTs out there Intention to treat (ITT) analysis has become more common, but RCT quality by other metrics has remained constant. Industry funds the vast majority of studies. Industry studies are significantly more likely to be appropriately blinded, report PROMs, use ITT, and have a higher overall quality. Over time, fewer studies have included an active comparator and studies with significant primary outcomes (p < 0.05) have become significantly less common. Smaller studies were associated with larger effect sizes. Overall, rheumatologic interventions have a high ARR. 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included 3,067 Titles Did Not Meet Inclusion Criteria 186 Did Not Meet Criteria After Full Review 3,338 Titles Identified 85 Papers included * * *

Thank You