Lessons learned from the MDIC mock submission CRT, February 20, 2017 Rajesh Nair, Ph.D. Team Leader, Therapeutic Statistics Branch II Division of Biostatistics CDRH, FDA 1

No Disclosures

Outline Overview of the MDIC mock submission Proposed pivotal clinical trial for mock device which leverages simulated clinical performance based on engineering models Novel methodology using Bayesian methods for incorporating good engineering models Benefits of proposed approach Lessons learned from mock submission First, I will provide an overview of the MDIC mock submission to FDA. The collaboration proposed a pivotal clinical trial for a mock device which leverages simulated clinical performance based on engineering models to make the trial more efficient. I will then discuss some benefits of our proposed approach and the lessons learned.

MDIC computational modeling and simulation working group Going from bench-to-bedside: Device manufacturers increasingly use engineering models to predict safety and effectiveness outcomes during the product development process Can we leverage simulated clinical performance of a device to improve efficiency of clinical trials? Work group brought together scientists from many device companies and FDA under the umbrella of MDIC Collaboration spanned a 2 year period For some devices we may have good engineering models that enable us to reliably predict the clinical performance of the device. Can we leverage this to improve the efficiency of clinical trials? MDIC set up the computational modeling and simulation work group in 2014 to address this problem. The work group brought together scientists from many device companies and FDA under the umbrella of MDIC.

MDIC computational modeling and simulation working group Framework for augmenting clinical trial with simulations Utilize engineering models to simulate clinical performance of device in a virtual patient (VP) population Novel Bayesian method combines virtual patients with prospective clinical data from real patients Potentially smaller and cost-efficient clinical trials Seek feedback from FDA on regulatory implications of proposed framework Here are some of the key steps in our framework for augmenting clinical trials with simulation. First, the engineering model is used to simulate the clinical performance of the device in a virtual patient population. When clinical outcomes are simulated in a target population we call these patients virtual patients. Second, a novel Bayesian method developed by the collaboration is then used to combine the virtual patients with prospective data from the trial. The proposed framework can potentially result in smaller and shorter clinical trials. We also sought frequent feedback from FDA on regulatory implications of the proposed framework.

Regulatory feedback obtained via CDRH presubmission (PreSub) PreSubs provide regulatory feedback to sponsors prior to an intended submission Submitted detailed clinical trial protocol for a mock device utilizing Bayesian framework for augmenting clinical trial data with virtual patients (VP) Mock submission reviewed by independent team within CDRH comprising medical officers, engineers and statisticians Multiple rounds of interaction with CDRH reviewers provided timely regulatory feedback Helped work group to refine methods The CDRH presubmission process is intended to provide opportunity for sponsors to obtain FDA feedback prior to an intended submission. We used the presubmission process to submit a detailed clinical trial protocol for a mock device utilizing a Bayesian framework for augmenting clinical trial data with virtual patients. The submission was reviewed by an independent team comprising of medical officers, engineers and statisticians within the center for devices at the FDA.

Presubmission: Lead Fracture study for mock ICD lead In our presubmission we proposed an augmented clinical trial which combines virtual patients with clinical trial data for a mock ICD lead. The figure on the left shows an ICD implant. The cardiac lead follows a convoluted path between the generator site and the heart. The figure on the right shows a cross section of the ICD lead. The curvature of the lead may vary considerably due to patient anatomy and how the surgeon implants the ICD. The curvature experienced by the lead can result in bending stresses on the lead which can result in lead fracture. Moreover, depending on the patients activity levels the lead can experience a lot of stress near the shoulder region which can cause the lead to fracture. Demonstrating an acceptable lead fracture rate is critical for demonstrating the safety of a new ICD lead. Leads

Presubmission: Lead Fracture study for mock ICD lead Investigational model: 2014VP Previous generation 2010 model market approved Updated design to improve handling characteristics Design changes could affect fatigue performance and lead failure in a patient. Bench testing can be used to measure fatigue strength and bending stiffness of the new lead. We have in-vivo use conditions on the predicate lead. We have distributions of patient characteristics for the predicate lead. Using these inputs we build a virtual patient model, predicting lead failure for the 2014VP lead As a case study we considered a scenario where the existing 2010 lead model has been redesigned to improve handling characteristics . These design changes could affect fatigue performance and lead survival of the updated 2014VP model. Generally, sponsors have to provide clinical data to obtain marketing approval for a new lead. Due to advances in predictive engineering we can reliably model the potential failure modes that can result in lead fracture. We leveraged this to design a more efficient clinical trial for the 2014VP lead. We can use bench testing to measure fatigue strength and bending stiffness of the new lead. Also, for the 2010 lead, we have in-vivo use conditions and distributions of patient characteristics. We used these inputs to build a virtual patient model to predict lead failure for the 2014VP lead.

Leveraging engineering models to predict clinical outcomes Physical Modeling Probabilistic Modeling Clinically Relevant Predictions Well Characterized Physics: Mechanical Electrical Heat Knowledge of Physiology: Local device ↔ tissue interactions Failure modes Tissue remodeling Patient and Device Variability: Age, Gender Activity level Implant factors Physical tolerances Safety & Reliability Related End Points: Cardiac lead failure Orthopedic implant fracture Pacemaker housing cracks Cardiac rhythm detection The engineering model uses a combination of physical and probabilistic modeling to simulate clinically relevant outcomes. To develop the model it is important that the device physics is well understood. The model also has to account for patient and device variability due to covariates like patient age and gender, activity levels and implant factors like surgeon effect. The model must have undergone extensive validation so that clinical outcomes can be reliably predicted.

Simulating Virtual Patients: ICD lead fracture modeling INPUT OUTPUT in-vivo bending patient activity Freedom from lead fracture fatigue strength This slide shows the input to the stochastic engineering model and model output. The figure on the top left shows the in-vivo bending of the lead seen in x-ray imaging The middle shows various types of patient activities which could result in bending stresses on the lead. The bottom figure shows the bench testing results for fatigue strength of the new lead. The engineering model uses these as input to predict the fracture survival of the investigational 2014VP lead model which is shown in the figure on the right. Years Lead fracture predicted using bench testing, use conditions, patient activity level Simulate many combinations of virtual patients to predict lead survival Haddad, Himes, Campbell, Reliability Engineering and System Safety, 123 (2014)

Highlights of proposed mock clinical study for 2014VP model Single arm study with a Bayesian adaptive design Primary endpoint: intra-cardiac lead fracture rate at 18-months Goal: Probability of experiencing intra-cardiac fracture is less than performance goal Bayesian adaptive design enrolls patients until a sufficient sample size is achieved with High probability of meeting the endpoint High probability of a futile study To establish the safety of the 2014VP lead we proposed a prospective single arm study with a Bayesian adaptive design. The primary endpoint is the intra-cardiac lead fracture rate at 18-months. This performance goal of 3% was chosen for the intracardiac lead fracture rate based on data from historical trials. Our Bayesian adaptive design enrolls patients until a sufficient sample size is achieved with either a high probability of meeting the endpoint or a high probability of a futile study.

Incorporate virtual patients (VP) using Bayesian methods Simulate lead survival for 2104VP lead in Virtual Patients (VP) based on engineering model Use Bayesian methods to integrate VP into clinical trial VP incorporated using power prior methodology How much should we borrow from VP data? If we borrow too much, results will be driven by model If we borrow too little, no efficiencies will be achieved Our proposed method first simulates lead survival for the 2014VP lead in virtual patients based on the engineering model. Then, Bayesian methods are used to combine the virtual patient data with prospective data from the clinical trial. The previous speakers have described how Bayesian methods can incorporate prior information from historical trials. We use the power prior methodology described by the previous speakers to combine the simulated data with the data from the prospective trial. How much information should we borrow from the simulated virtual patients? If we borrow too much, results will be driven by the model. If we borrow too little, no efficiencies will be achieved in the trial.

Control borrowing based on consistency with experimental data Achieving the right amount of borrowing We can simulate an infinite number of VP so down weighting needed Achieved using power prior methodology Use loss function to control borrowing Borrow more when data agree Borrow less when data do not agree Maximum borrowing capped in consultation with regulators Our proposed flexible loss function achieves desired amount of borrowing Ensure good frequentist operating characteristics In principle, we can simulate an infinite number of virtual patients so it is important to down weight the amount of borrowing from virtual patients. We achieved this using the power prior methodology. We also used a loss function to control the amount of borrowing from the virtual patients depending on the level of agreement between the virtual patient data and the experimental data. The neat feature of the loss function is that it enables us to borrow more from the virtual patients when the simulated outcomes are similar to the experimental data. Conversely, we borrow less when the data do not agree. Our flexible loss function controls the amount of borrowing to ensure good frequentist operating characteristics.

Benefits of augmented trial with borrowing from virtual patients This figure shows the percent of virtual patients as a fraction of the total number of patients. When the level of agreement between the simulated data and real data is high, the virtual patients contribute up to 45% of the total number of patients in the trial. When the level of agreement is low, we borrow little from the virtual patients and only about 5% of the total patients are virtual patients. With the proposed method, depending on level of agreement between the simulated data and real data there is potential for substantial reduction in sample size and in the trial duration. Level of agreement High Low Up to 45% reduction in required enrollment Up to 10 months reduction in trial duration

Lessons learned Due to heavy reliance on modeling early interaction with regulators is critical Model credibility and context of use critical for regulatory acceptance Is the model reliable? Does model account for engineering uncertainties as well as patient to patient variation? Have all relevant covariates been included? Can virtual patients be considered exchangeable with real patients in the trial? Due to heavy reliance on modeling talking to regulatory agencies early in the development process is essential. Model credibility and context of use have to be demonstrated for the proposed methodology to be acceptable to regulators. The model has to be able to account for engineering uncertainties as well as patient to patient variation. All relevant patient-level covariates must be included in the model. For the Bayesian methodology to be applicable the virtual patients have to be exchangeable with real patients in the trial.

Lessons learned Be prepared to answer questions from regulators Bias and type I error control Weight of prior information and choice of loss function Results robust to misspecification of virtual patient model High simulation burden to address regulatory concerns Method likely to be useful for mature device areas which are in 2nd or 3rd generation of product development The FDA review team raised a large number of questions related to bias and type I error control using the proposed method. The review team also had questions regarding the weight of the prior information, choice of loss function and whether results are robust to misspecification of virtual patient model. To address these questions from regulators a large number of simulations had to be carried out which could be a high burden for trial sponsors. The method is most likely to be useful for mature device areas which are in the 2nd or 3rd generation of product development.

Lessons learned Challenging to review for regulators Requires collaboration between clinicians, statisticians and engineers at the FDA Multiple face-to-face meetings may be required during review process on account of added complexity Extensive interaction with sponsor needed to identify areas where there are gaps in understanding Mock submission gave FDA reviewers advance opportunity to understand potential regulatory issues related to use of simulations for regulatory approval Enable development of regulatory science A submission utilizing simulations is going to be challenging to review for regulators. It will require a lot of collaboration between clinicians, statisticians and engineers at the FDA. On account of the added complexity, multiple face-to-face meetings may be required during the review process. The mock submission provided FDA reviewers advance opportunity to understand potential regulatory issues related to the use of simulations for regulatory approval.

Summary Mock PreSub and interaction with FDA available online through MDIC site Promote development of industry proposals leveraging these methods Fosters a collaborative approach to developing methods for improving efficiency of medical device clinical trials Benefits the broader ecosystem through publications Vehicle for creating innovation in regulatory science http://mdic.org/computer-modeling/virtual-patients/ The mock presubmission as well as the interaction with the FDA are available through the MDIC website. We hope the methods developed by the collaboration will be utilized by industry in future submissions. By bringing together industry and regulatory scientists, the MDIC working group was able to develop innovative methods for improving the efficiency of medical device clinical trials.

References Visit MDIC.org for more information on the MDIC computational modeling & simulation project T. Haddad, A. Himes and M. Campbell, "Fracture prediction of cardiac lead medical devices using Bayesian networks," Reliability Engineering and System Safety, vol. 123, pp. 145-157, 2014 T. Haddad, et al “Incorporation of stochastic engineering models as prior information in Bayesian medical device trials" to appear in Journal of Biopharmaceutical Statistics Here are some references for the Bayesian methodology described in my talk. 19

Acknowledgements Thanks for attending! MDIC team Tarek Haddad Adam Himes Laura Thompson Telba Irony Valentin Parvu FDA team Sherry Yan Xuefeng Li Jack Zhou Robert Kazmierski I would like to acknowledge contributions from both the MDIC and FDA teams. This collaborative work was done under the Medical Device Innovation Consortium (MDIC) public-private partnership. For more information please visit http://mdic.org Thanks for attending! 20