1. Borrowing Prior Information: What to Borrow, How Much to Borrow, and the Effective Communication between the Sponsor and FDA Laura Lu, Ph.D Division of Biostatistics, CDRH
Along review experience

2. Bayesian Statistics: Submissions to CDRH
Since 1999, at least 25 original PMAs and PMA supplements have been approved with a Bayesian analysis for the primary endpoint.
Most are for therapeutic devices
Several are for diagnostic devices
Many IDEs have also been approved with a Bayesian design, quite a few EAPs
Several applications for “substantial equivalence” (510(k)s)
First I am going to briefly describe the applications of bayesian statstics in CDRH
Notate PMA, IDE and EAP, 510K

3. Why Prior Information in Device Studies?
The mechanism of action of the device is often very well understood:
Local as opposed to systemic
Physical as opposed to pharmacokinetic
Treatment effect of device is relatively easier to be predicted with pre-clinical results (e.g., bench testing)
Devices evolve while drugs are discovered
Slight changes to a device may lead to small changes in its effects
Results of previous versions of device may serve as good prior information
Getting more prevalent. In many prior information are available for.. More often than drug.. We know… compare with other ,,,,,, as oposed to drugs, a little
Useful prior info are avialble for device due to the mature of devices and its development …….These lead to better understanding and prediction on device effect before cany clincal data are avaialbe.

4. 07/16/96
Finalized by CDRH and CBER. Reflected the thinking of two centers in Bayesian applications for devices fromboth stat and clin aspects

5. Considerations in Regulatory Setting
Prior planning of the design and analysis is crucial. Before the new study begins
Prior information identified
The amount of prior information borrowed to be agreed upon
Not a substitute for good science
Need of randomization
Handling of missing value
Assessment of bias
Adequacy of analysis population
The guidance document emphasizes …As the frequestist trial

6. Source of Prior Information
RWE, e.g. data from clinical registry
Pilots studies
Studies conducted overseas
Data of very similar devices
Subgroups from a failed study
Adult population to be extrapolated for pediatric population
Simulated patient outcomes based on clinical/engineering models
Next I am going to describe the source of prior and appraoches in borrwoing prior information . Many sources of prior information being proposed in submission

7. Use of Prior Information
Borrow information for the same treatment
Supplemental control data from historical trials or registries
Borrow information between treatment arms (treatment effect)
Adult data for pediatric indication
Trials for a previous generation device
Two categories for the use of prior info, which means we borroween the imformation for the difference

8. Bayesian Models for Borrowing Prior Information (1)
Hierarchical model
Estimates device effect in current study by borrowing strength from related studies
The amount of borrowing depends on the similarity of the study outcomes and the parameters specified in the hyper priors
Power prior model (Chen and Ibrahim, 2006)
Borrow historical data with a power prior
ϴ: treatment effect D0: historical data D1: new clinical data
α0: power parameter ranges 0-1
Math form for normal situation
yij ~ N(ϴi, σi2),
ϴi ~ N(ϴ, υ2),
hyper prior for ϴ and υ are usually specified
α0 in [0, 1], the proportion of information borrowed from the historical study
Power prior discounts prior information, not effect size in the historical study
We also see other approaches
Adam
π(ϴ/D1, D0, α0) ∝ L(ϴ/D1)L(ϴ/D0)αoπ0(ϴ)

9. Bayesian Models for Borrowing Prior Information (2)
Both the hierarchical and power prior model could incorporate multiple prior datasets
When there is only one prior study,
Power prior approach is considered a straight forward way to borrow historical information with a discount
Power prior approach has a more intuitive interpretation for the clinicians
Challenge in choosing α0

10. How Much to Borrow? -Clinical Judgment
Clinical judgment on the relevance of prior information
Similarity in protocol design (treatment arms, endpoints, target population, etc.), and
Similarity of clinical practice
In many situations we do not wish the prior information to dominate the posterior distribution, especially
Patient population
A subgroup selected from a failed study without a strong clinical rationale
We have discussed … now face the critial cecision a decision to be agreed upon with extensive discussion between … for example

11. How Much to Borrow? -Statistical/Regulatory Considerations
Patient level data including baseline characteristics to assess exchangeability (similarity) between patients and studies
All information should be considered (favorable and non-favorable)
Cherry picking favorable outcomes (instead of using clinically sound rationale) will lead to biased inferences for posterior distribution of the treatment effect
Type I error rate to keep the consistency/connection with the evidence level required for trials with frequentist design
Besides .. The following should be considred from stat/regulatory point of view

12. Example 1 Device A was approved in Europe
A randomized US trial was proposed for a more inclusive patient population with the same control
Plan to borrow both treatment and control information from the European study
Next I am going to give a few tio illustrate our experience with baysina design

13. Mock Example 1 (cont.)
Sponsor proposed power prior models and provided Type I error rate and power with different α0 values and assumptions for treatment effect
True Treatment Effect as % observed in an European Study α0 0%
(No Borrowing) 5%
15%
25%
35%
50%
2.5% 3% 4% 6% 9%
12%
26%
30%
37%
44%
52%
58%
75%
55%
66%
67%
70%
76%
100%
73%
77%
83%
88%
91%
Propose to enroll 100 patients

14. FDA’s Considerations
Due to the more inclusive patient population and potential difference in clinical practice, decision for approval should be mainly based on the information in US population
Reasonable Type I error rate
At the time of PMA submission, assess
Consistency of US data and European data
Effective sample size
A minimum number of patients should be enrolled to evaluate safety endpoints

15. Example 2 The first study for Device B failed
Based on subgroup analyses (1 in >10 subgroups), the result was more promising in:
A secondary endpoint among patients enrolled in Subgroup A
The sponsor proposed
To enroll patients similar to those in Subgroup A
Borrow data from Subgroup A data in the first study

16. FDA’s Considerations
A better treatment effect in Subgroup A is supported by a clinical rationale. However, ‘better’ subgroups can be defined by other clinical rationales and based on different cut-points.
Could not exclude the possibility of a spurious finding among the outcomes of all subgroups and endpoints
Statistical model should appropriately take into account the relationship of results in Subgroup A and other subgroups
The number of subgroup analyses
A more convincing clinical rationale leads to relatively more borrowing from Group A and less borrowing from Group Ac
All subjects in the study should be borrowed if Group A was only a chance finding
Ongoing discussion on the amount to borrow

17. Example 3
A very large trial is usually conducted to estimate Device C’s long term failure rate
An engineering model is proposed to predict clinical outcomes (virtual patient model)
The sample size of the new study is reduced when incorporating virtual patient data with a power prior model
π(ϴ/D1, D0, α0) ∝ L(ϴ/D1)L(ϴ/D0)αoπ0(ϴ)

18. FDA’s Considerations Advantages: Challenges: Easy to interpret
Regret control
More powerful than hierarchical models and the power prior model with pre-specified α0
Challenges:
Determination of maximum sample size for D0 (virtual patients): How reliable of the engineering model?
Determination of α0
Definition of the ‘similarity’ function
Double use of virtual patient data violates likelihood principle
𝜆 and 𝑘 are specified such that the type I error and power can be reached at desired values. Do and alpha0, both FDA/MDIC are putting in effort in improving the methodology, the original

20. Challenges with Q/IDEs Submissions with Bayesian Design
A trial with Bayesian design needs more input from the agency at the beginning stage: adequacy of prior information and how much can be borrowed.
Analysis with Bayesian design often involves customized coding – leading to spending more time on writing (sponsor) and checking (FDA) the codes
Parameters in the statistical model may often change based on the Agency’s input on the winning rule and desired operating characteristics.
Next , I will talk So far we have seen that also, and more…

21. Stage/Step-Wise Submission Is Recommended
The high level features of a protocol should be discussed before the statistical details are proposed.
For complex statistical model, thorough communication between sponsor and FDA is needed to clearly understand the clinical implication of the model.
Programs and simulation codes should be started after the statistical model is agreed with FDA to avoid repetitive work.

22. Stage I (preliminary)
1a) Determine the primary endpoint(s) and other parameters of clinical significance for study design
Treatment arms
Primary endpoints
Inclusion exclusion criteria
Follow-up schedule
Types of stopping rule
prior information (external control-historical, concurrent registry data)
Adequacy of prior
1b) Amount and methods of borrowing
The maximum amount of prior information to borrow and new information needed
Proposal for borrowing methods (power prior, hierarchical model and others)
Winning criteria (threshold for the posterior probability)

23. Stage I (preliminary)
1a) Determine the primary endpoint(s) and other parameters of clinical significance for study design
Treatment arms
Primary endpoints
Inclusion exclusion criteria
Follow-up schedule
Types of stopping rule
prior information (external control-historical, concurrent registry data)
Adequacy of prior
1b) Amount and methods of borrowing
The maximum amount of prior information to borrow and new information needed
Proposal for borrowing methods (power prior, hierarchical model and others)
Winning criteria (minimum clinically significant effect and threshold for the posterior probability)

24. Stage II (detailed)
2a)
Simulation algorithm for obtaining the posterior distribution
Write module of codes for simulation
Trustable/standard software or packages
Well organized with annotated codes
Displaying intermediate output with mock data can improve readability
Other statistical aspects such as analyses population, handling of missing data, multiplicity issues, poolability of data
2b)
Evaluate operating characteristics based on different design parameters and stopping rules
Type I error rate, power, probability of success/futility at interim analyses
Fix design parameters and stopping rule to meet desired design properties from clinical, statistical and regularity points of view.
Details at Q sub. Low level language, not enough time to check at IDE stage. 2b can bin in IDE

25. Stage II (detailed)
2a)
Simulation algorithm for obtaining the posterior distribution
Write module of codes for simulation
Trustable/standard software or packages
Well organized with annotated codes
Displaying intermediate output with mock data can improve readability
Other statistical aspects such as analyses population, handling of missing data, multiplicity issues, poolability of data
2b)
Evaluate operating characteristics based on different design parameters and stopping rules
Type I error rate, power, probability of success/futility at interim analyses
Fix design parameters and stopping rule to meet desired design properties from clinical, statistical and regularity points of view.
Details at Q sub. Low level language, not enough time to check at IDE stage. 2b can bin in IDE

26. Summary and Conclusions
Bayesian method is a useful approach in combining prior information and current information to make a inference for the treatment effect, especially in medical device trials.
Prior planning of the design and analyses is crucial.
What to borrow and how much to borrow depend on clinical, statistical and regulatory considerations.

27. Summary and Conclusions (cont.)
Cherry picking prior information could lead to a biased inference if the Bayesian model does not take into account the relationship between the selected prior information and rest of the available data.
Extra effort is needed in communicating with the Agency due to the challenges with Bayesian design:
Decision on what and how much to borrow
The choice of Bayesian model
Complexity in writing and validating the code for analysis and simulation
Stage/step-wise submission is recommended to avoid repetitive work.
Subtantial amount of time and effor neeed to resutsve issues and reach agreement

28. Acknowledgement
Thanks to my colleagues Ram Tiwari, Yun-Ling Xu, and Sherry Yan for their valuable contributions to this presentation

29. Back-up Slides

30. Choosing the More Promising Subgroup: Two Subgroups Situation
Paired design n=100.
Sham vehicle applied to both eyes
For each subgroup, randomly dissect the sample into two subgroups with equal number ni=50
Yi iid ~ N(0, 1)
Group with the higher mean ymax selected as the promising group

31. More rationale, the better

32. Statistical Considerations
Biased prior information for treatment effect µmax~0.1
Chance of identifying a subgroup with statistical significance is 5% (number of independent subgroups x 2.5%) instead of 2.5% for a non-effective device
Risk of wasting resource due to the regression to the mean effect
Both groups A and Ac need to be weighed equally to correct the bias