1. 1 Changing Trial Designs on the Fly Janet Wittes Statistics Collaborative ASA/FDA/Industry Workshop September 2003

2. 2 Context Trial that is hard to redo Serious aspect of serious disease Orphan

3. 3 Statistical rules limiting changes To preserve the Type I error rate To protect study from technical problems arising from operational meddling

4. 4 Challenge sense rigor

5. 5

6. 6 Challenge senseless rigor mortis

7. 7 Scale of rigor Over rigid Rigorous Prespecified methods for change – preserves Unprespecified but reasonable change Invalid analysis responders analysis outcome-outcome analysis completers

8. 8 Consequences No change during the study OR Potential for the perception that change caused by effect

9. 9 Prespecified changes Sequential analysis Stochastic curtailing Futility analysis Internal pilot studies Adaptive designs Two-stage designs

10. 10 Problems Technical Solved Operational Risks accepted EfficiencyUnderstood

11. 11 Add a DMC What if it acts inconsistently with guidelines? Something really unexpected happens? DMC initiates change Steering Committee initiates change

12. 12 Reasons for unanticipated changes Unexpected high-risk group Changed standard of care Statistical method defective Too few endpoints Assumptions of trial incorrect Other

13. 13 Examples 1.Too much censoring; DMC extends trial 2.Boundary not crossed but DMC stops 3.Unexpected adverse event 4.Statistical method defective 5.Event rate too low; DMC changes design

14. 14 #1 Endpoint-driven trial Trial designed to stop after 200 deaths Observations different from expected Recruitment Mortality rate At 200 deaths, fu of many people<2 mo DMC: change fu to minimum 6 mo P-value: 0.20 planned; 0.017 at end

15. 15 #2. Boundary not crossed Endpoint Primary: 7 day MI Secondary: one-year mortality Very stringent boundary

16. 16 What DMC sees Very strong result at 7 days No problem at 1 year Clear excess of serious adverse events

17. 17 Haybittle-Peto bound (10%)

18. 18 Haybittle-Peto bound (30%)

19. 19 Haybittle-Peto bound (50%)

20. 20 Haybittle-Peto bound (70%)

21. 21 Haybittle-Peto bound (70%)

22. 22 #3. Unexpected adverse event: PERT study of the WHI Prespecified boundaries for BenefitHarm Heart attackStroke FracturePE Colon cancerBreast cancer

23. 23 Observations BenefitHarm -----Stroke FracturePE Colon cancerBreast cancer Heart attack

24. 24 Actions Informed the women about increased risk of stroke, heart attack, and PE Informed them again Stopped the study

25. 25 #4. Statistical method defective Neurological disease 20 question instrument Anticipated about 20% would not come Planned multiple imputation- results: Scale: 0 to 80 Value for ID 001: 30 38 ? 42 28 ? MI values: -22, 176

26. 26 #5. Too few endpoints Example: approved drug Off-label use associated with AE Literature: SOC event rate: 20 percent Non-inferiority design - = 5 Sample size: 800/group

27. 27 Observation 400 people randomized 0 events What does the DMC do?

28. 28 Choices Continue to recruit 1600 Stop and declare no excess Choose some sample size Tell the Steering Committee to choose a sample size What if n=1? 2? 5? 10?

29. 29 Conclusions Ensure that DMC understands role Separate decision-making role of DMC and Steering Committee Distinguish between reasonable changes on the fly and cheating Expect fuzzy borders

30. 30 Technical Changing plans can increase Type I error rate We need to adjust for multiple looks How do we adjust for changes?

31. 31 Operational Unblind assessments Subtle change in procedures In clinical trials, the FDA and SEC