1. Statistical & Design Considerations for Non-inferiority trials Andrew Nunn MRC Clinical Trials Unit London

2. TB Forum December 20052 Outline What is a non-inferiority trial? How do they differ from superiority trials? What do the regulators say? How large do the trials need to be? How should the trials be conducted and analysed?

3. TB Forum December 20053 What is a non-inferiority trial? How is it different from an equivalence trial? Does non-significant imply non-inferior? Does non-inferior imply non-significant? Are non-inferiority trials always larger than superiority trials? Can a failed superiority trial be turned into a non- inferiority trial? Why do we need these trials in TB?

4. TB Forum December 20054 A little bit of history - 35yrs ago Study R, the first East African/BMRC trial of short course chemotherapy could be regarded as a non-inferiority trial. 2STH/16TH worked well under strict trial conditions. The main objective was to see if a six month regimen was at least as good as the standard treatment, better - would be a bonus. S = streptomycin, T = thiacetazone, H = isoniazid

5. Study R – 30 month relapse free rates 0 -35% 5% -5% A possible δ Confidence intervals for difference from control 2STH/16TH regimen 6SHR 6SHZ 6SHT 6SH No difference

6. 6 A point to remember “It is never correct to claim that treatments have no effect or that there is no difference in the effects of treatments. It is impossible to prove … that two treatments have the same effect. There will always be uncertainty surrounding estimates of treatment effects, and a small difference can never be excluded.” Alderson P, Chalmers I. BMJ 2003:326:1691-8.

7. TB Forum December 20057 Why non-inferiority for new TB drugs? Under a wide variety of trial conditions the gold standard 2EHRZ/4HR regimen is at least 95% effective. –Nomads in the Algerian Sahara –Recently published IUATLD study in Africa and Asian centres. We will be very unlikely to better it. –we would require a total of 2600 evaluable patients to demonstrate a reduction from 5% to 2.5% relapses

8. TB Forum December 20058 Our goal Our goal is to reduce treatment duration to a maximum of 4 months and preferably less. How much are we prepared to pay, if anything, for such a reduction? –must the new regimen be as good as the standard? –would we be satisfied with a regimen that was almost as good? –if so, how good is almost?

9. TB Forum December 20059 EMEA quote “If no degree of possible inferiority of the test [new regimen] to the reference [control] is acceptable, then the development of products with equal efficacy to a comparator by means of non-inferiority trials would become impossible.” EMEA /CPMP /EWP /2158 /99

10. TB Forum December 200510 Does non-significant = non-inferior? No! definitely not. Common sense will tell us that a non- significant result from an under-powered study is, in the extreme case of little value. BUT, non-inferior does not necessarily mean non-significant!

11. TB Forum December 200511 How do we do it? We need a null hypothesis. The situation is the reverse of what is required in a superiority design. For superiority –H 0 is there is no difference. For non-inferiority –H 0 is there is a difference. The alternative hypothesis is also reversed.

12. TB Forum December 200512 Equivalence & non-inferiority What’s the difference?

13. TB Forum December 200513 Determining equivalence First step in establishing equivalence - define ‘limits of equivalence’ (± δ) Having conducted the trial, calculate the 95% confidence intervals for the difference between the control and the new treatment If the confidence interval is entirely within ± δ then equivalence is established

14. TB Forum December 200514 Non-inferiority Equivalence requires that the difference control - new intervention is both > -δ and < δ, the new treatment must be neither worse nor better than the control by a fixed amount. In contrast to equivalence with non-inferiority we are only interested in determining whether new treatment is no worse by an amount δ.

15. TB Forum December 200515 Non-inferiority 0-δ No difference The 95% CI for the difference between the control and the intervention are all > -δ, i.e. non-inferiority demonstrated.

16. TB Forum December 200516 Non-inferiority 0-δ No difference The lower 95% CI for the difference between the control and the intervention are all > -δ, i.e. non-inferiority demonstrated. The lower 95% CI is < -δ, non-inferiority has not been demonstrated.

17. TB Forum December 200517 Non-inferiority and superiority 0-δ No difference The 95% CI for the difference between the control and the intervention are all >-δ, i.e. non- inferiority demonstrated. In this case both non- inferiority and superiority have been demonstrated

18. TB Forum December 200518 Non-inferiority and inferiority 0-δ No difference The 95% CI for the difference between the control and the intervention are all >-δ, i.e. non- inferiority demonstrated. In this case both non- inferiority and superiority have been demonstrated In this case both non- inferiority and inferiority have been demonstrated

19. TB Forum December 200519 Choosing δ The value of δ must be chosen before the trial begins. It’s value will depend on clinical, statistical and possibly regulatory considerations.

20. TB Forum December 200520 Example: 2NN Study van Leth, Phanuphak et al (Lancet 2004), a study of first-line antiretroviral therapy in HIV Main comparison between nevirapine twice daily and efavirenz (plus stavudine and lamivudine) in terms of ‘treatment failure’ (based on virology, disease progression, therapy change) Primary objective was to establish the non-inferiority of nevirapine twice daily ( δ =10%)

21. TB Forum December 200521 Example: 2NN Study Confidence intervals for failure rates (E-2NN) –All data (-12.8%, 0.9%) –Only those starting med. (-14.6%, -0.8%) –Concurrently randomised (-11.9%, 3.4%) Non of these intervals are completely above δ value of -10%; one interval also excludes zero

22. TB Forum December 200522 Example: 2NN Study BUT, the authors concluded: ‘Antiviral therapy with nevirapine or efavirenz showed similar efficacy, so triple-drug regimens with either … are valid for first-line treatment’ Lancet 2004, 363:1253-63

23. TB Forum December 200523 Does it matter? A non-inferiority trial can demonstrate significant benefit from the new treatment - (cf Study A). But is it possible to have non-inferiority and a significantly worse outcome in the new treatment? Yes! provided δ is acceptable to clinicians. –if N is large enough any difference can be shown to be significant!

24. TB Forum December 200524 Adverse effects Assessment of adverse effects is particularly important in equivalence trials. It is not enough to prove non-inferiority in terms of efficacy. A new treatment must be as safe, or safer, than the old one.

25. TB Forum December 200525 Choosing δ On 27 th July 2005 the European Medicines Agency (EMEA) issued a new European “Guideline on the choice of the non-inferiority margin” This guideline comes into effect in January 2006. EMEA /CPMP /EWP /2158 /99

26. TB Forum December 200526 Quote from EMEA document “The lower limit of the confidence interval [of the difference between the new regimen and the control]... represents a lower bound and is usually interpreted as the degree of inferiority to the reference that can be excluded based on the data presented….. EMEA /CPMP /EWP /2158 /99

27. TB Forum December 200527 Quote from EMEA document “Of course this is not an actual lower bound and the magnitude of inferiority could be greater. However it is generally considered that the chance of the true difference being worse than that suggested by this bound is acceptably small.” EMEA /CPMP /EWP /2158 /99

28. TB Forum December 200528 General EMEA recommendations If possible three study arms should be included, test, reference and placebo - allows validation of the non- inferiority margin. The margin should be such there is assurance that the test arm has a clinically relevant effect. The primary focus is the relative effect of the test arm and the reference arm. The choice of the margin should be justified in the protocol The choice of the margin should be independent of power considerations.

29. TB Forum December 200529 Design consideration It is important to ensure that the design of equivalence trials, including definitions of a favourable response, should be as similar as possible to earlier trials assessing the control regimen.

30. TB Forum December 200530 Internal validity In a superiority trial there is a strong incentive to ensure high quality of conduct. In contrast in an non-inferiority trial the conclusion of non-inferiority could be reached because of poor discriminatory power. In a TB trial this could occur if follow-up rates were poor and/or there was failure in the lab to detect all relapses.

31. TB Forum December 200531 But If there are already many treatments being used interchangeably for the disease under consideration a possible approach might be to consider the information available from all of them. From this a delta may be constructed which summarises the information known about the relative efficacy of these products, and the new trial can be designed to provide a similar level of knowledge of the relative efficacy of the new product.

32. TB Forum December 200532 Accepting a larger δ “In the situation where the test product is anticipated to have a safety advantage over the reference it is likely that a larger delta could be justified as some loss of efficacy might be accepted in exchange for the safety benefits”

33. TB Forum December 200533 Is there a case for a larger δ if treatment can be shortened? “It may be possible to justify a wider non- inferiority margin for efficacy if the product has an advantage in some other aspect of its profile. This margin should not, however, be so wide that superiority to placebo is left in doubt”

34. TB Forum December 200534 How large a δ would you accept? If treatment could be shortened from 6 to 4 months would an increase in the failure/relapse rate from 5% to 10% be acceptable?

35. TB Forum December 200535 How large a δ would you accept? If treatment could be shortened from 6 to 4 months would an increase in the failure/relapse rate from 5% to 10% be acceptable? - provided that the failures and relapses could be satisfactorily retreated.

36. TB Forum December 200536 FDA position FDA (as described in FDA’s 1992 Points to Consider document) originally used a ‘step function’: Cure Rate δ  90%10% 80 - 89 % 15% < 80 % 20% A more flexible approach has since been adopted

37. 37 Example : Pediatric Meningitis Trial Investigational Drug vs. Active Control Sponsor’s FDA Proposal Proposal Projected response rate80% 80% Delta15% 10% Evaluable total sample size224 504 Projected % evaluable70% 70% Total to be enrolled320 720 Projected enrollment time 2-4 years 4-6 years FDA proposed study considered not to be feasible Note:  = 5%, power = 80%

38. 38 What confidence level? Traditionally we use 95% confidence in superiority trials (thanks to RA Fisher!) Guidelines for pharmacokinetic equivalence have traditionally used 90% CI. In regulatory situations the choice is based on level of risk regulators are prepared to accept. Could be appropriate to use 90%, 95% or even 99%. Need for flexibility.

39. TB Forum December 200539 Calculating power - an example Given the expected range of, say 3-6% relapse rates in the control, 2EHRZ/4HR regimen. What study size would we require for a range of δ?

40. 40

41. 41 5% relapse, δ = 10%, 100 per arm

42. 42 5% relapse, δ = 10%, 100 per arm 5% relapse, δ = 5%, 400 per arm

43. TB Forum December 200543 But... These power calculations do not allow for additional numbers required for a Per Protocol analysis, or patients excluded because they do not have TB, or because they have MDR disease. Neither do they allow for losses to follow-up.

44. TB Forum December 200544 How should we analyse non- inferiority trials? Superiority trials are analysed by ITT because it is the most conservative and least likely to be biased. ITT analysis of non-inferiority trials is not conservative - there is a bias towards no difference. PP biased since not all randomised patients included. It is recommended that non-inferiority trials should be analysed by both ITT and per protocol (PP).

45. 45 Defining ITT and PP Definitions vary. –For ITT some definitions exclude patients who either do not have confirmed diagnosis, or who never received treatment. –PP includes all receiving full course of treatment with no major protocol violations. What definitions are appropriate for TB trials? CPMP: ‘similar conclusions from both the ITT and PP are required in a non-inferiority trial.’ ‘Sample size computations should ensure sufficient numbers in the PP population’. CPMP: Committee on Proprietary Medical Products (2000)

46. TB Forum December 200546 CAVE! Drop-outs from the two regimens need to be carefully evaluated. Suppose patients not responding dropped out early from one treatment arm, or Possibly because of differential withdrawal rate for adverse events - This would suggest there may be important differences between the treatments.

47. TB Forum December 200547 Interim analyses Do we need them? Probably not if it is to consider stopping early for strong evidence of non-inferiority. Such evidence would support a case for the possible superiority of the new treatment to the control - a strong incentive to keep on.

48. TB Forum December 200548 Conclusions 1 A major concern among regulators in many NI trials is that the efficacy of the control is not well established. This is NOT the case with the control regimen 2EHRZ/4HR. One advantage of no new drugs for 40 years!! In the event of establishing a 4 month regimen to be non-inferior it would be unwise to use that regimen as the control in the next NI trial - biocreep. Biocreep - slightly inferior treatment becomes the control for next generation of NI trials

49. TB Forum December 200549 Conclusions 2 NI trials must be conducted with rigour The value of δ needs to be determined before the start of the trial and should take into account both clinical and statistical considerations. Both the value of δ and other aspects of design need to be discussed with regulators Non-inferiority needs to be demonstrated not only for efficacy but also for safety.

50. TB Forum December 200550 Regulatory Guidance ICH E9 ‘Note for Guidance on Statistical Principles for Clinical Trials’, September 1998 ICH E10 ‘Note for Guidance on Choice of Control Group’, July 2000 CPMP ‘Note for Guidance on the Investigation of Bioavailability and Bioequivalence’, July 2001 CPMP ‘Points to Consider on Switching between Superiority and Non-Inferiority’, July 2000 CHMP ‘Guideline on the Choice of the Non-Inferiority Margin’, July 2005

51. TB Forum December 200551 Selected references D’Agostino RB, Massaro JM et al: Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics. Statist Med 2003; 22: 169-186. Altman DG, Bland JM: Absence of evidence is not evidence of absence. BMJ 1995; 311:485. Blackwelder WC: Current issues in equivalence trials. J Dent Res 2004; 83:C113-115. Jones B, Jarvis P et al: Trials to assess equivalence: the importance of rigorous methods. BMJ 1996; 313:36-9.

52. Study R – 30 month relapse free rates 0 -35% 5% -5% A possible δ Confidence intervals for difference from control 2STH/16TH regimen 6SHR 6SHZ 6SHT 6SH No difference

53. TB Forum December 200553