EHR-S Conformance Considerations Lynne S. Rosenthal National Institute of Standards and Technology August 2004

2 Todays Objectives Present basic information about conformance and conformance related topics Address: –What is Conformance? –What does it mean to conform? –How do you know a system conforms? –What do you need to do testing? –Who does the testing?

3 Background Standards not enough to ensure interoperability –Standards only meaningful if implemented in a consistent way Need to ensure that implementations adhere to the standard –What is expected of implementations in order to claim conformance – i.e., what are the requirements? –How will we know if an implementation conforms? test suites, test tools Different ideas of what conformance is –Past experience may have affected view of conformance

4 Terminology Conformance Specification Conformance Clause Conformance Testing test suite, test tools Validation process, reporting Certification certificate, brand

5 Terminology - Conformance CONFORMANCE – the fulfillment of a product, process or service of specified requirements. (ISO Guide 2) –The requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

6 Terminology - Conformance Testing CONFORMANCE TESTING – a way to determine directly or indirectly that relevant requirements are fulfilled. –Serves as a communication between buyer and sellers Buyers increased confidence Sellers substantiate claims –Necessary, but not sufficient, for interoperability Note: referred to as conformity assessment in ISO Guide 2

7 Terminology - Testing Process VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite –ensures that testing can be repeatable and reproducible –ensures that conclusions are consistent with facts presented in the evaluation CERTIFICATION - acknowledgement that a validation was completed and the criteria established for issuing certificates (brands) was met.

8 Testing Ingredients Standard Conformance clause Conformance Testing Test assertions Test suite (test software, test scripts, test criteria) Validation Process - policy and procedures for testing Certification qualified bodies to do the testing and certification control board - advisory and arbiter

9 Conformance Issues Conformance Clause should address 1.What Needs to Conform 2.How to conform and claim conformance 3.Subdividing a specification: Profiles, Levels, Modules 4.Variability a.Extensions b.Options c.Implementer defined values d.Alternative approaches

10 What needs to conform? Identify who or what will implement the specification What class of products will be developed –What products are considered an EHR-S? –Does it make sense to mention these? Why care? –It helps in defining the scope of the specification –It helps the reader know what is being targeted by the specification What are the conditions that need to be met in order to claim conformance

11 Profiles, Levels, Modules Profile is a subset of the overall specification that includes all the functionality necessary to satisfy the requirements of a particular community of users. –Use to target needs of specific constituencies or purpose (care setting) Levels are nested, hierarchical subsets of the specification. Level 1= core that all must implement, subsequent levels include lower level + added functionality –Use to facilitate incremental development and implementation (system core) Modules are discrete collections of semantically-related units of functionality that do not necessarily fit into a simple hierarchical structure. –Use when functionality can be implemented independently of one another (ePrescribing, CPOE, PHR) Reference: Subdivide a Specification (extracted from W3C Quality Assurance documents)

12 Profiles, Levels, Modules EHR-S DSTU Nursing Home Profile Ambul. Care Profile System Core Care Setting Core Care setting core XX ZZYY 3 System Core Care Setting Core CPOE ePrescibe PHR

13 Extensions Extensions incorporate additional features beyond what is defined in the specification If allowed: –shall not re-define semantics of existing functions –shall not break existing functions Standardized method for extending the specification –Define how to simplify or add functions for care settings –Provide wording so vendors can articulate their extensions in a consistent way Why? –Helps ensure extensions defined in a consistent manner – leading to predictable understanding and handling aids interoperability

14 Options = discretionary choices Provide the freedom to make choices about: –Whether or not to support a function and/or features of that function –Which functions, features, or values to choose from a list of possibilities –Implementation dependent features or values –Alternative approaches Indicated by choice of words: Should, May, Recommend Make sure need for the option is real If the option is implemented, it must be implemented correctly (as defined in the specification)

15 More on options Implementation Defined Features –may be necessary to address environmental conditions (code sets and vocabulary, locality differences - e.g., language, time zones, dependencies on other technologies, or just to be flexible) Alternative Approaches –specification may describe several different ways to accomplish an operation (e.g., choice of bindings) –specify whether or not all approaches need to be implemented

16 Conformance Testing Requires: a test tool/method and testing program Test method: –Way to determine requirements are met –E.g., test tool, test suite, checklist, etc Testing is Falsification Testing –Can show presence of errors not their absence –Prove non-conformance, can never prove conformance Testing Program = Doing the testing –1 st party (self-test), 2 nd party (buyer), 3 rd party (independent) CONFORMANCE TESTING – a way to determine directly or indirectly that relevant requirements are fulfilled.

17 Testing and Certification Testing and Certification Considerations Not all specifications need a testing program –Risks of nonconformance vs. cost of creating and running testing program How formal should testing be? –Determined by the market or regulation –Self testing, 2 nd party (buyer), 3rd party testing –Self-certifying vs. independent certification Testing program –Who will own the testing program –What are the policy and procedures –Who will do the testing – testing organizations? –Who will issue the certificates? –Who will handle disputes Control board –Who will do maintenance?

18 Testing and Certification Enables quality products, competitive markets, more choices Must be acceptable to sellers and credible to buyers Reference: Conformance Testing and Certification Model for Software Specifications

19 Conformance Guidance All specs to contain a conformance clause –Identifies: what and how to conform –Any special designations of conformance (e.g., conform to core levels) –Ways that conforming implementations can vary –How to make a claim of conformance Traceability to the specification Specification profiles EHR-S Conformance statements test cases tests

20 Conformance Clause Examples Example 1 –Conformance of Generators is defined in terms of conformance to a particular profile of CGM. –A Profile conforms if it meets all the requirements in ISO xxx and is structured in accordance with the Rules for Profiles Example 2 –There is no concept of conformance to this standard because there is no normative content. This approach is taken for several reasons, chiefly … Example 3 –Within the context of this specification, conformance can be claimed for markup, document types, module implementations, documents, generators, and interpreters. In most of these cases, two levels of conformance are available: simple conformance and full conformance. Simple conformance means that the conforming object supports the minimal content model of the ruby element in section 2.1, i.e. only simple ruby markup. Full conformance means that the conforming object supports the maximal content model of the ruby element in section 2.1, i.e. that both simple and complex ruby markup are supported. Example 4: –DICOM 1 (PS ) points to PS , DICOM Part 2: Conformance –Each part of the 1073 family of standards contains a conformance section.

21 References 1. Conformance Testing Defines conformance and conformance testing and describes the components of a conformance testing program. 2. Conformance Testing and Certification Model for Software Specifications Overview of testing and certification, including identifying key roles, activities and products involved in any conformance testing and certification program. 3. W3C Quality Assurance Framework: Specification Guidelines and W3C Quality Assurance Variability in Specifications Public versions available in September How to write better specifications with a focus on conformance issues