1. Improving the Standards of Reporting of Clinical Trial Data
2018 JSM
Jitendra Ganju
Clinical trials consultant (statistics)

2. Why this talk?
Why not a talk on better statistical methods?
Tamping down reporting bias as important as developing more sophisticated methods

3. John Ioannidis on low reproducibility (several important papers)
Updated CONSORT guidelines (BMJ 2010)
“[Q]uality of reporting of randomized controlled trials is not optimal.”  This results in “biased results” and can “mislead decision making in health care at all levels”
Fleming (Ann Intern Med, 2010)  
Bias arising from the “interest to find evidence suggesting that an intervention…has a favorable benefit-to-risk profile.”

4. The Problem From My Perspective
Trial Planning
Phase 2 and 3 randomized, blinded trials require a lot of planning
Inherent hierarchy in trial design. Analyses are
α-controlled and pre-specified (power known)
not α-controlled but pre-specified (power unclear)
not pre-specified, i.e. post hoc
Trial Reporting
Hierarchy not maintained when results reported
Point estimates from non-α controlled and post hoc analyses
lead to exaggerated sense of efficacy when p < 0.05

5. More p-values, more Bias
Multiple
- endpoints
- definitions of endpoints
- methods of analysis per endpoint
- time points analyzed per endpoint
- analysis populations
Suppose for a trial 5 out of 100 p-values α controlled
Article includes α controlled and a few non-controlled p-values.
What are the selection criteria?
Big question: what should get included in abstract?
“96% of abstracts with p-values had at least one ‘statistically
significant’ p-value” (John Ioannidis and team finding)

6. Ask your non-stat colleagues
Which outcome would they prefer to report?
Hazard ratio < 1 implies benefit. Same data, different models
HR = 0.81, p = 0.008, or
HR = 0.75, p = 0.020

7. NEJM publications (typical examples)
Examples next
NEJM publications (typical examples)

8. Example 1: Typical style
Comingling pre-specified primary with other related
NEJM 2007
Primary endpoint: patients receiving red cell transfusion
between days 1 and 29. Secondary: no. of units transfused

9. Example 2: More commingling
NEJM 2007
Secondary endpoint: mortality at days 29 & analyses (4 not in table).
No primary method. Adjusted for 8 or 9 covariates. No α adjustment.

10. Separate table, separate discussion
How to improve it
Suppose mortality at Day 29 is primary endpoint. Suppose unadjusted
analysis pre-specified. Separate presentation and discussion of
only the red box. The rest in separate tables and sections
Separate table, separate discussion

11. Example 3: Estimate in abstract
NEJM 2016
Analysis by stratified Wilcoxon rank sum test (tests shift in distribution)
45.3% = (difference in medians)×100 / placebo median
What is 45.3% an estimate of? (post hoc)
What is estimate of variability?
Report estimate on which inference was based

12. Example 4: Another estimate in abstract
NEJM 2018
Report both mean and median or neither in abstract

13. Improvements in Reporting:
Present and discuss hierarchically
Top: α-controlled and pre-specified
Most reliable. Separate presentation
Middle: Not α-controlled but pre-specified
Do not mix with top tier. Less rigorous
Bottom: Post hoc
Do not mix with top and middle tier. Speculative,
may or may not be important

14. Other Improvements
For non-α-controlled but pre-specified analyses: use 99.5% CI (no p-values). OR, no CI, just point estimate, std dev
For post hocs: Show point estimate, std dev (highly preferred), no CIs, no p-values. If included, use 99.9% CI. Post hocs findings in body of report, not abstract
What to include in abstract?
1. Criterion should be that result would be included regardless
of whether statistical significance achieved
2. Point estimate should be based on method of analysis (not a
reaction to seeing the data)

15. What You Can Do To reduce bias, need for rigor in reporting
Actively participate in making the change
(editor, reviewer, author, tweet, letter to editor, persuade,…)
Additional proposals in manuscript

16. Example 5: Post hoc Improvements
Table title should note post hoc analysis
Drop 95% CI. Show point estimate and std dev
If CI included, explain what it says

17. Example 6: Multiplicity
NEJM 2017 (emphasis added)
“Although on the basis of the prespecified hypothesis testing sequence
the renal outcomes are not viewed as statistically significant, the results
showed a possible benefit of canagliflozin with respect to the progression
of albuminuria (hazard ratio, 0.73; 95% CI, 0.67 to 0.79)…”
Risk of type II error?
How to do better
Abstract and Conclusion: note “possible benefit”
(no point estimate, no p-values, no CI)
Body of report: point estimate, std dev (no p-values, no CI)

18. Example 7: α splitting and CI
Align α and CI
Lancet 2016

19. Very Important Point
“If a rare event for Ho occurs that also is rare for typical H1 values, it provides little evidence for rejecting Ho in favor of H1”
CN Morris, JASA 1987
Comment relevant when α not controlled, post hocs