Break-out Session II Group III: Certification HCMDSS November 16-17, 2004 Arlington, Virginia

Challenges and Opportunities Certification guidelines/policies are often too general and do not sufficiently guide vendor development process expectations are unclear and vary depending on the "certifying agent” Positive & negative aspects positive: freedom for vendors to follow their own process negative: lack of standard processes leads to lack of incentive to develop tools that aid the certification process confusion/frustration from vendors as they try to anticipate requirements of certifying agent process and validation techniques vary (even within same organization) Imprecise/underspecified certification guidelines

Challenges and Opportunities Systems are certified (not software units/components) lack of officially sanctioned mechanism for certifying components/units/infrastructure no official process for reusing certification efforts/artifacts even when software components themselves are reused must re-certify entire system again for each configuration/assembly of components prohibits “software product-line architectures” which are proving effective in reducing costs in a variety of domains reduces the incentive to develop standard interfaces and infrastructure (e.g., middleware) for hooking devices together barrier to “plug-and-play” vision Non-composibility of certifiable systems

Challenges and Opportunities Effective development of “Systems of systems” of medical devices requires well-defined standards for interfaces and data exchange formats Need definitions of standard interfaces and automated tools that can establish that, e.g., services conform to (correctly implement) the interface clients correctly use the interface composed components don’t inadvertently interfere Lack of standards (especially for interfaces)

Challenges and Opportunities FDA: no certification authority mechanism for introducing/sanctioning tools that reduce certification obligations (however, this concept does exist in FAA certification) reduces the motivation to develop tools to assist in certification reduces the motivation to move to “model- driven development” methods No mechanism for introducing officially sanctioned tools

Challenges and Opportunities Inclusion of COTS components in certified systems OS: dealing with security updates to Windows XP (should update to close security holes, but updating can lead to crashes – Brian Litt (Level 2 devices)) Middleware Compilers Imprecise/underspecified certification guidelines

Challenges and Opportunities What is appropriate “evidence” of high- confidence that allows an agency to confirm that a system is “certified”? currently, process-oriented (not quality- oriented or correctness-oriented) does not admit paradigm of independent verifiable “certificates” or “proofs” of (partial) correctness/authenticity Limited view of certification “evidence”

Challenges and Opportunities Once systems are certified in US, one would hope that they could be deployed in other world areas without recertification according to a different set of guidelines Currently Europe – basically OK – sharing of guidelines Asia – more problematic Certification for other world areas

Current/Existing Solutions Certifiers are smart conscientious people that take their job very seriously Establish that new system to be certified is substantially equivalent to an existing marketed device Certification – state of the practice

Research and Development Needs Support Plug-and-Play vision for medical devices “Modular Certification”? should be able to… certify components, assemble a system from a collection of components while checking “compatibility/non-interference” of components complete recertification of assembled system should not be necessary What are testing requirements in this context? infeasible to test components assembled in every possible configuration Composable certified entities

Research and Development Needs Need notions of “trusted” component or ability to establish authenticity of “certificate” provided with a component e.g., proof-carrying-code paradigm component comes with certificate, and checking the certificate is simpler than generating certificate or verifying the component directly problem similar to that faced with Microsoft device drivers (3 rd party code being inserted into larger framework) Need motivations and frameworks for vendors to develop certifiable infrastructure (OS, middleware, component libraries) Inclusion of COTS and “third-party” components

Logistics to Meet Research Needs “Open Experimental Platform” for High Confidence Medical Devices Funding agency pays for… …vendors to develop representative examples of certified systems …vendors to provide description of development process …vendors to provide “challenge problems” to researchers …vendors to provide “face time” to meet with researchers and evaluate developed tools against base-line development processes Positive Examples of DARPA-funded OEPs… Avionics Mission Computing OEP from Boeing Automotive Control System OEP from Ford

Logistics to Meet Research Needs Encourage teaming of standards/certification bodies ISO, FDA, with vendor representatives motivation: each of these are stakeholders in certification effort Example task Evolution of HL 7 standard (protocol for device communication) good support for exchange of static data poor support for exchange of dynamic/object- based data

Workshop Recommendations Invite someone to talk about DICOM (medical imaging standard) how did this standard arise (war stories) Use this insight to determine how best to arrive at standards for other interfaces and infrastructure Insight on development of standards

Workshop Recommendations Invite third party reviewers for FDA certification to share experiences gleaned from being involved in a wide range of certification efforts Insight on experience with certification efforts

Workshop Recommendations NEMA – National Electrical Manufacturing Association Invite members of trade organizations for presentations & interactions

Workshop Recommendations Invite members of agencies that are able to guide/form policy, e.g., standards incorporation of verification/validation tools to reduce certification obligations Suggestions for FDA included… Dan Schultz Phil Philips Invite high-level members of relevant agencies

Workshop Recommendations Talk about how the use of formal methods tools is used to reduce obligations/efforts associated with validation Invite EDA representative (chip developer)

Workshop Recommendations Invite someone to talk about overall costs of developing certifiable systems Use this information to help researchers understand where they should target & prioritize their efforts if they aim to reduce costs Invite someone give a broad perspective on certification costs