1. FDA/INDUSTRY STATISTICS WORKSHOP: Washington, D. C. Sept
FDA/INDUSTRY STATISTICS WORKSHOP: Washington, D.C. Sept. 29, Statistical Issues in Medical Device Trials
George Koustenis, FDA-CDRH
David Breiter, Boston Scientific
Roseann White, Abbott Vascular
George Woodworth, Univ. of Iowa
9/20/2018

2. The Use of Objective Performance Criteria (OPC) in Medical Device Evaluation
FDA/Industry Statistics Workshop
September 29, 2006
Washington, D.C.
George Koustenis
Division of Biostatistics
Office of Surveillance and Biometrics
Center for Devices and Radiological Health
U.S. Food and Drug Administration
Note that the Commentary contains more detail and specifics and references…so feel free to look it over at your leisure. Any and all comments are appreciated
9/20/2018

3. OBJECTIVE
The purpose of this presentation is to provide a brief introduction to the possible use of Objective Performance Criteria (OPC) in the evaluation of medical devices during the regulatory approval process.
From my perspective as FDA statistical reviewer
9/20/2018

4. DEFINITION OF OPC Fixed Target(s) Objective and Meaningful Standard
Provides Comparison in Evaluating Safety and Effectiveness
Usually a Rate
Surrogate for Control Group
Benchmark for Minimally Acceptable Values
Not a Control Group
As a first step, let us establish an operational definition of the term Objective Performance Criteria. The essence of an OPC is that it is designed to be used as a fixed target(s), a value that should be an objective and meaningful standard to provide a comparison when evaluating the safety and effectiveness of a medical device. It is typically expressed as a rate (1-2). Thus, the OPC is used as a surrogate for traditional control groups and the associated rigorous scientific and analytical methodology typically observed in medical device clinical trials. An OPC serves as a benchmark, a minimally acceptable value used in a pass/fail approach in determining if a particular device application is ultimately approved for marketing. By definition then, an OPC is not a control group.
9/20/2018

5. CONTROLS Standard Comparison for an Experimental Treatment
Group of Patients with Same Condition, Demographics, & Prognostic Values
By Controlling for as Many Variables as Possible – Any Differences are Presumed Due to New Intervention
A control group serves as the standard comparison for some intervention. It is typically a group of patients with essentially the same condition, demographics, and prognostic values as those in the treatment group, except that they do not receive the intervention under investigation. The control group may receive no intervention, a different intervention, or the current standard therapy regimen. By attempting to control for as many factors as possible during the study, then any differences observed between treatment and control groups on selected patient outcome measures are presumed to be due to the introduction of the intervention during the clinical trial.
9/20/2018

6. TYPES OF CONTROLS Concurrent Randomized Controlled Trial (RCT)
The Gold Standard
Effectively Minimizes Bias
Effectively Balances Demographics & Unknowns
Supports Basic Assumptions of Standard Statistical Methodology
Note that Greg has covered Controls but a brief re-cap will be useful here
There are different types of control groups that may be employed in device clinical trials. The active, concurrent randomized control group is considered the gold standard for clinical trials. This type of control can effectively minimize bias in allocating patients to the control or experimental group, effectively balances the two groups with respect to critical demographic and study variables, and will support the assumptions associated with standard statistical methodology (3).
9/20/2018

7. Historical Controls
Compares Current Therapy & Patients Against Others Studied in Previous Investigations
Not Randomized to Current Conditions
May Be Too Far Removed in Time
Very Difficult to Validate the Data from Historical Trials
Tend to Produce More Positive Results
Another type of control is the historical control group. A historical control study compares current patients using a new intervention with a group of patients that have been previously studied in other investigations. There are a number of problems associated with historical controls. They are not randomized with respect to the current intervention group, they are not concurrent with the current intervention group, they may be so far removed in time from the current study that the patient population and/or standard of care for patients has significantly changed. Further, it is almost impossible to validate the data entered from historical trials, and in fact significant baseline data may not even be available for the historical controls (3).
9/20/2018

8. ESTIMATES OF OPC’S Necessarily Driven by Historical Data
Requires Appropriate Pooling of Different Investigations
OPC’s Inherit All of Problems Seen with Historical Controls
Even Sophisticated Pooling/Analytical Techniques Cannot Eliminate Basic Problems
The estimate for an OPC is necessarily derived from historical data. This requires pooling of data across previously published results using meta-analysis or similar approaches (10-15). Thus, the OPC inherits all of the difficulties associated with historical controls, plus some other problems as well. Approaches have been reported that employ more sophisticated statistical modeling techniques, including Bayesian approaches, (16-21), in an effort to derive OPC’s that are a more precise estimate of the true historical rate of interest. However, even these techniques include the caution associated with deriving any value from historical data (17). Further, even after an OPC has been derived, there may not be consensus agreement in the established value of the OPC. When the single arm, self as control design is used as the treatment group to be compared to the OPC, problems with confounding increase because this design has problems with potential investigator and patient selection bias, along with the inability to mask patient or investigator. These issues further confound the effort to provide a valid comparison of the OPC against the device with regard to safety and effectiveness.
9/20/2018

9. USE OF OPC’s In CDRH Some Current Use of OPC’s
Cardiac Ablation Catheters
Replacement Heart Valves
Ophthalmics
Hip Replacement Systems
In CDRH, we are already seeing the limited use of OPC in the device approval process. Some examples include cardiac ablation catheters, replacement heart valves, ophthalmic criteria for certain refractive procedures, and there has been significant preliminary work done in an effort to establish OPC for hip replacement systems (22-25). Given that the use of OPC represents a significant departure from the standard scientific approach to the design and analysis of medical device clinical trials, the question now becomes: when might it be appropriate to use this type of non-control comparison in the medical device approval process?
9/20/2018

10. USE OF OPC’s In DEVICE TRIALS
Given that the use of OPC represents a significant departure from the standard scientific approach to the design and analysis of medical device clinical trials, the question now becomes: when might it be appropriate to use this type of non-control comparison in the medical device approval process?
Given that the use of OPC represents a significant departure from the standard scientific approach to the design and analysis of medical device clinical trials, the question now becomes: when might it be appropriate to use this type of non-control comparison in the medical device approval process?
9/20/2018

11. CITED ADVANTAGES Of OPC’s
Smaller Sample Size
Standard Value for All Sponsors
Save Time and Money
Easier to Execute
I.E. – “Least Burdensome”
The cited general advantages of using an OPC are that they require a smaller sample size, provide a standardized comparison for all sponsors, they will save considerable time and money, and are logistically simpler to execute. In other words, they may be less burdensome to conduct than traditional medical device trials. However, as we will examine below, the appropriate derivation of an OPC may not be a trivial exercise. Thus it is possible in some instances that a randomized controlled trial may in fact be the less burdensome investigation. While the spirit of least burdensome requirements plays a role, the preeminent concern of CDRH is to promote and protect the public health by assuring that valid scientific evidence is used in determining the safety and effectiveness of medical devices during the regulatory approval process (26).
9/20/2018

12. DISADVANTAGES Of OPC’s
All the Problems Associated with Historical Controls
Problems with Validity of Data & Analysis
Problems with Advances in Practice of Medicine
May be Disagreement on Final OPC Value
Problems with Single Arm Trials
Selection Bias
May Not See Pre-Post Benefits
9/20/2018

13. DISADVANTAGES Of OPC’s
May Not Be Least Burdensome
Smaller Sample Size? N=100 or 150?
Time and Resource Intensive to Develop
Sets a Minimum Standard
Superiority?
Older and Older Data
Protect and Promote the Public Health?
9/20/2018

14. WHEN MIGHT OPC’s Be USED?
History of OPC’s in Medical Devices
Non-Experimental or Quasi-Experimental Designs May be Valid in Certain Situations
O’Malley et al., and others provide descriptions of the history and rationale for the use of OPC’s in the device approval process (1,12-13,16,24,27). Others have discussed the fact that under certain conditions, valid and useful data may be obtained when using non-experimental designs in the design, conduct, and analysis of clinical trials (3-4,6,14).
The salient observations from the above citations will be briefly summarized here. In clinical trial design, there are situations where a great deal may be known about the natural history of the disease, the underlying patient population is well described and relatively stable, there is extensive clinical history and experience, a stable and well known standard of care, and the appropriate technology is relatively stable.
9/20/2018

15. MINIMUM REQUIREMENTS WHEN OPC’s MIGHT Be USED
Great Deal is Known About the Natural History of the Disease or Condition
Underlying Patient Population is Well Described & Relatively Stable
Extensive Clinical History & Experience with This Device
9/20/2018

16. MINIMUM REQUIREMENTS WHEN OPC’s MIGHT Be USED (cont.)
Stable and Well Known Standard of Care
Appropriate Ancillary Technology is Relatively Stable
No Significant New Questions of Safety or Effectiveness
Consensus Among FDA, Industry, Clinical, Academic and Patient Communities
Expectation of Significantly Positive Treatment Effect
9/20/2018

17. HOW SHOULD OPC’s BE DERIVED?
Data Driven
Rigorous And Scientifically Valid Methodologies
Valid Databases
Appropriate Statistical Modeling
Appropriately Designed and Powered Pivotal Trial
Periodical Re-Evaluation and Updating the OPC’s
Further, rigorous and scientifically valid methodologies should be developed and employed in the derivation of any OPC for use in the medical device approval process. While there is clearly room for appropriate clinical input, as well as other relevant evidence in the discussion of OPC development, the fundamental derivation an OPC must be data driven. This implies that an OPC must be derived from recognized and generally complete historical datasets, and be the product of appropriate statistical modeling and analytical techniques. In addition, the pivotal trial that will be conducted to collect patient data for comparison with the selected OPC must incorporate an appropriately sized and powered sample size. Further, there should be a designated provision for periodically evaluating, and updating the OPC based on more recent experience and data (27-29).
Thus, given the extensive methodologies, logistics, resources, and time that must be devoted to the appropriate development and derivation of an OPC, this approach may not always be the least burdensome path to device approval. For example, a recent CDRH guidance document indicates that the FDA believes that, in general, randomized, controlled trials reflect the least burdensome means of collecting data in support of safety and effectiveness for catheter devices intended to treat atrial fibrillation (30).
Still think they are least burdesome compared to a relatively small study?
9/20/2018

18. OTHER IMPORTANT OPC ISSUES
All Parties Must Have a Clear Understanding of the OPC in Medical Device Trials – Success and Failure
OPC’s Already in Limited Use in Device Trials
No Current General Policy of OPC Use in CDRH
Need for Comprehensive, Coherent, and Consistent OPC Policy for Device Trials
Equally important, all concerned parties must clearly understand the implication of OPC use in the approval or disapproval of a medical device application. Currently, there is no general CDRH policy statement regarding the development and use of objective performance criteria in the medical device approval process. Since OPC’s are already in limited use, and the expectation is that the use of OPC’s will continue to proliferate in the device approval process, it is clear that a comprehensive, coherent, and consistent CDRH policy on OPC development and use is required.
9/20/2018

19. WHAT SHOULD THIS POLICY CONTAIN?
Clear Definitions of All Appropriate Terms
Established Guidance on Minimum Standards for a Device or Device Class
Specific Roles Played by FDA, Industry, Clinical Community, Academia and Other Interested Parties
Provisions for Periodic Updating of OPC
Specific Guidance on Methodology to Derive an OPC
Unambiguous Policy Regarding Failure to Meet OPC
Such a policy should include, but is not limited to the following considerations:
1. Clear definitions for all appropriate terms.
2. Established guidance on the minimum conditions and criteria for a device, or class of device, as to when an OPC might be considered an appropriate vehicle for device approval.
3. Specific roles to be played by FDA, industry, the clinical community, academia, and other interested parties in the development of a particular OPC.
4. Provisions for the periodical updating of the OPC including who will be responsible for the revisions, specific methodology, and timelines for these updates.
5. Specific guidance on appropriate methodology for deriving an OPC.
6. A clear policy on agreement between FDA and sponsors as to the limitations of device approval under an OPC.
7. An unambiguous policy regarding the regulatory consequences of any investigation that fails to meet a pre-defined OPC.
9/20/2018

20. SUMMARY
An introduction to the history and use of Objective Performance Criteria (OPC) in medical device evaluation
Advantages and disadvantages of OPC’s
When and how OPC might be used & developed
Discussed the need for developing a CDRH policy regarding use of OPC in medical device trials
General outline of this policy is presented in an effort to begin dialogue between CDRH, industry, clinical community & other interested parties
9/20/2018

21. REFERENCES Please see accompanying brief commentary
Request electronic copy from
9/20/2018

22. Q&A
9/20/2018