1. Risk and Credibility Assessments for Computational Modeling of Medical Devices
Tina Morrison, PhD
Advisor of Computational Modeling
Office of Device Evaluation, FDA
Vice Chair
ASME V&V40 Subcommittee
Member
MDIC CM&S Steering Committee

2. Role of V&V for Computational Models of Medical Devices
If computational models are to be increasingly relied upon in the development and evaluation of medical devices, the consistent application of V&V must be applied to establish model credibility.
Need to establish …
If the model is correct and credible
Demonstrated predictive capabilities to justify use beyond domain of validation
Predictive confidence is commensurate with model risk

3. ASME Subcommittee on V&V
Standards Subcommittee
Provide procedures for assessing and quantifying the accuracy and credibility of computational modeling and simulation
V&V Standards Committee in Computational Modeling and Simulation
V&V-10 - Verification and Validation in Computational Solid Mechanics
V&V-20 - Verification and Validation in Computational Fluid Dynamics and Heat Transfer
V&V-30 - Verification and Validation in Computational Simulation of Nuclear System Thermal Fluids Behavior
V&V-40 - Verification and Validation in Computational Modeling of Medical Devices

4. Verification & Validation in Computational Modeling of Medical Devices
V&V-40 Charter
Provide procedures to standardize verification and validation for computational modeling of medical devices
Charter approved in January 2011
Medical device focus
Regulated industry with limited ability to validate clinically
Want increased emphasis on modeling to support device safety and/or efficacy
Use of modeling is hindered by lack of V&V guidance and expectations within medical device community
V&V Standards Committee in Computational Modeling and Simulation
V&V-10 - Verification and Validation in Computational Solid Mechanics
V&V-20 - Verification and Validation in Computational Fluid Dynamics and Heat Transfer
V&V-30 - Verification and Validation in Computational Simulation of Nuclear System Thermal Fluids Behavior
V&V-40 - Verification and Validation in Computational Modeling of Medical Devices

5. Guide for Verification and Validation for Computational Models of Medical Devices
Regulated industry with limited ability to validate clinically
Want increased emphasis on modeling to support device safety and/or efficacy
Use of modeling is hindered by lack of V&V guidance and expectations within medical device community
Focus of the Guide
Instead of focusing on how to perform V&V (established elsewhere) …
We developed a common V&V framework to standardize definitions, processes, and documentation requirements between industry, researchers, software developers and regulators.

6. Overall V&V Flow Assess Model Risk Establish Credibility Requirements
Establish Work plan for VV
Define
COU
Purpose NO
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
YES
Execute pre-defined M&S and V&V plan
Is the
CM&S Credible for COU? NO
YES
Document M&S and VV Plan and Findings

7. Overall V&V Flow Risk Assessment Matrix Assess Model Risk
Establish Credibility Requirements
Establish Work plan for VV
Define
COU
Purpose
Risk Assessment Matrix NO
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
YES
Execute pre-defined M&S and V&V plan
Is the
CM&S Credible for COU? NO
YES
Document M&S and VV Plan and Findings

8. Risk Assessment Matrix (RAM)
Establish Context of Use
Model Risk: combination of decision influence and consequence
Decision Influence: contribution of the model outcome to the decision being made
Consequence: impact if the model outcomes prove incorrect
Model risk assessment
Directs/guides V&V activities
Defines model credibility requirements
HIGH
MEDIUM
INFLUENCE
LOW
CONSEQUENCE

9. Overall V&V Flow Credibility Assessment Matrix Assess Model Risk
Establish Credibility Requirements
Establish Work plan for VV
Define
COU
Purpose
Credibility Assessment Matrix NO
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
YES
Execute pre-defined M&S and V&V plan
Is the
CM&S Credible for COU? NO
YES
Document M&S and VV Plan and Findings

10. Credibility Assessment Matrix (CAM)

11. Credibility Assessment Matrix (CAM)
Establish Target Credibility Requirements based on the Context of Use ● ● ● ● ● ● ● ● ● ● ● ● ●

12. Overall V&V Flow Assess Model Risk Establish Credibility Requirements
Establish Work plan for VV
Define
COU
Purpose NO
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
YES
Execute pre-defined M&S and V&V plan
Is the
CM&S Credible for COU? NO
YES
Document M&S and VV Plan and Findings

13. Credibility Level Determination√ √ √ √ √ √ √ √ √ √ √ √ √

14. Jeff Bischoff Mehul Dharia Zimmer, Inc.
Example
Jeff Bischoff
Mehul Dharia
Zimmer, Inc.

15. Force on tibial spine of a knee implant
Posterior Tibial Spine Force in Deep Flexion
May Create Posterior Liftoff
Anterior Tibial Spine Force
in Hyperextension
May Create Anterior Liftoff
Locking mechanism between the (metal) tibial tray and (polyethylene) articular surface is intended to prevent disassembly (poly lift-off) of the modular tibial component during activities of daily living

16. Anterior Lift-off Test
Context of use of a test for anterior lift-off
Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than expected physiological loading, and therefore demonstrate that the new (locking mechanism) design sufficiently mitigates that risk.

17. Contexts of use of a model for anterior lift-off:
Determine the size of component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test relative to a predicate.
FEA followed by Physical Test, Comparison to Predicate
Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than that required for a clinically successful predicate, and therefore demonstrate that the new (locking mechanism and/or geometry) design sufficiently mitigates that risk.
FEA only, Comparison to Predicate
Determine the size of a component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test without reference to predicate device.
FEA followed by Physical Test, No Predicate
Demonstrate through analysis alone that the worst case size can sustain physiological loading without liftoff, without reference to a predicate device.
FEA only, No Predicate

18. Risk Assessment Matrix
Model influence  
LOW: Results from the computational model are a negligible factor in the decision associated with the question being answered.
MEDIUM: Results from the computational model are a moderate factor in the decision associated with the question being answered.
HIGH: Results from the computational model are a significant factor in the decision associated with the question being answered.
Patient consequence
LOW: A poor decision would not adversely affect patient safety or health, but might result in nuisance to the physician or has other negligible impacts.
MEDIUM: A poor decision would result in minor patient injury and potentially requiring physician intervention or has other moderate impacts.
HIGH: A poor decision would result in severe patient injury or death or has other significant impacts.

19. Risk Assessment Matrix
Predicate
No predicate
FEA + testing
COU1
COU3
FEA alone
COU2
COU4

20. Risk Assessment Matrix
Predicate
No predicate
FEA + testing
COU1
COU3
FEA alone
COU2
COU4
COU4
COU1: Worst case
determination
COU2: Absolute evaluation
COU3
COU1,2

21. CAM – Elements of Computational Models
Verification
Code (Column B) – 4
Used commercially available validated FEA software
Solution (Column C) – 4
Mesh convergence study was performed
Numerical effects are determined to be small on all important quantity of interests at conditions/ geometries directly relevant to the context of use
All inputs and outputs based on independently reputable source

22. CAM – Elements of Computational Models
Validation: Computational Model
System Configuration (Column D) – 2
Used mean/nominal geometry (no LMC/MMC)
Major and minor features captured
Two sizes considered
Governing Equations (Column E) – 4
Used nonlinear material (constitutive) model for UHMWPE
Key physics (press-fit, resistance against force) was captured
Material model did not need re-calibration/tuning
System Properties (Column F) – 1
Nominal physical properties that are representative of the comparator from literature
Sensitivity analysis on material properties was not performed
Boundary Conditions (Column G) – 3
Load applied through assumed contact patch on spine, rather than directly modeling the femoral component - Representative but simplified BCs with non-quantified effect on QOI

23. CAM – How Well Is The Comparator Understood?
Validation: Evidence-Based Comparator
System Configuration (Column H) – 3
Prescribed location
Geometries matched to machine tolerance (production parts)
Signal to noise ratio is high
System Properties (Column I) – 3
Off-the-shelf parts were tested
Environmental effects on the material are known (testing speed was modified, environment was kept the same for both groups: in air).
Boundary Conditions (Column J) – 3
No sensitivity analysis was performed.
Known (recorded) loading (perturbations) was applied and boundary condition variability (e.g. posterior slope) is known.
Sample Size (Column K) – 3
Statistically relevant sample size (n = 5)
Component size, a key parameter for lift-off, variation was considered.

24. CAM – How Appropriate is CM to Comparator?
Validation: Model-to-Comparator
Discrepancy (Column L) – 4
Equivalent input parameters, equivalent quantity of interest

25. CAM – How Appropriate is CM to Comparator?
Validation: Model-to-Comparator
Discrepancy (Column L) – 4
Equivalent input parameters, equivalent quantity of interest
CAM – How Rigorously Are Outputs Compared?
Validation: Qualitative or Quantitative
Comparison (Column M) – 3
Quantitative comparison, with single set of input parameters, without predictive accuracy or uncertainties available
No quantitative comparison with broad range of cases

26. CAM – How V&V activities relates to COU?
Validation: V&V to COU
Applicability (Column N) – 3
Validation activities embody relevant characteristics of the CoU sufficient overlap between the validation domain and the CoU space)

27. What can we conclude? COU4 COU3 COU1,2 Predicate No predicate
FEA + testing
COU1
COU3
FEA alone
COU2
COU4
COU3
COU1,2

28. Overall V&V Flow Assess Model Risk Establish Credibility Requirements
Establish Work plan for VV
Define
COU
Purpose NO
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
YES
Execute pre-defined M&S and V&V plan
Is the
CM&S Credible for COU? NO
YES
Document M&S and VV Plan and Findings

29. Public Meeting - FDA/NIH/NSF Workshop on Computer Models and Validation for Medical Devices, June 11-12, 2013
Additional resources on RAM and CAM

30. For More Information Please Contact:
Tina Morrison, PhD
Advisor of Computational Modeling
Office of Device Evaluation, FDA OR
Michael Liebschner, PhD
Pre-ORS Symposium Chair
Baylor College of Medicine; Exponent Failure Analysis