Part 1: HL7 Testing Tools (Supporting Meaningful Use and IHE) Part 2: Towards a Testing Infrastructure (Lessons Learned and Leveraging Resources) Robert Snelick March 21, 2011

Outline: Key Theme is Automation and Leveraging Tools HL7 Testing Toolkit HL7 is the most widely used messaging standard to exchange administrative and clinical data (>90% of U.S. hospitals) Meaningful Use Testing Data Instance Validation Reportable Lab Results to Public Health Immunization to Public Health Patient Identification Management and Query Testing Isolated System Testing IHE PIX/PDQ Testing Tool Towards a Testing Infrastructure Expand the scope to other protocols End-to-end Interoperability Test bed (Peer-to-peer System Testing) Sustainability/Transfer/Exit Strategy

NIST HL7 Testing Toolkit APIs and components to support tool development Applications and Services HL7 Artifacts Web Services Message Generation Message Validation Message Encoding Profile Validation Generation HL7 Communication Data Sources Test Agents Java APIs Message Maker 3rd Party Applications and Testing Environments Java Database Message Validation Profile Validation Testing Framework Web Applications IHE PIX/PDQ IHE PCD MU EHRPH MU ImmunizationPH NHIN PIX/PDQ CCHIT Lab Messages Message Generation Message Validation NIST HL7 Test Data HL7 Data Repository MU Validation Profile Registry

Meaningful Use HL7 V2 Validation Testing Reportable Lab Results and Immunization Robert Snelick March 21, 2011

ARRA Meaningful Use Certification Testing HL7 V2 Tools used for certifying EHR systems and models (Stage 1) Process Test Procedure Data elements and message structure defined Test Data Sets Validation Tool (http://xreg2.nist.gov:8080/HL7V2MuValidation2011/) EHR Reportable Lab Results to Public Health HL7 V2.5.1 Implementation Guide and LOINC value set EHR Immunization to Public Health HL7 V2.3.1, HL7 V2.5.1, and CVX value set Leverages the HL7 V2 Toolkit Project Timeline Research/Development: March 2010 to October 2010 Maintenance/Support: October 2010  Current Has general industry applicability Current Use: Accredited Testing Certification Bodies Stage 2?

Lab Results to Public Health MU Test Procedure HL7 ELR IG 2010 Meaningful Use Scope (Stage 2) LAB (or any sending system or utility) EHR (System under Test) Public Health Meaningful Use Scope (Stage 1) ORU Message ORU Message ORU Elements Direct Data Entry Message Captured NIST Validation Tool Validation Report ORU Message Data The EHR is the system being tested. The EHR system is required to send (create) messages that conform to the referenced standards (HL7 V2.5.1 Lab IG and LOINC). Test data can be entered into EHR directly via the EHR’s user interface or be imported via an incoming message The EHR is expected to process the test data to create a message. This message is captured and uploaded into the testing tool for validation. The test can be conducted using NIST supplied test data or vendor supplied test data. When using NIST data sets, the appropriate validation context needs to be selected. When choosing vendor supplied data an appropriate generic validation context needs to be selected.

Immunization Message Validation

Validation Report

Supporting Pre-connectathon and Connectathon Testing IHE PIX/PDQ Testing Supporting Pre-connectathon and Connectathon Testing Robert Snelick March 21, 2011

IHE PIX/PDQ Testing Patient Identification Management and Patient Demographics Critical integration profiles PIX Source, PIX Manager, PIX Consumer, PDQ Client, PDQ Supplier Used for North American and European IHE Affiliates Supports both Connectathon and Pre-connectathon Testing Events Supports both HL7 V2 and V3 Messages Standards Testing Tool Isolated System Testing Interactive on-line web application tool Test Event Management Capabilities Leverages the HL7 Toolkit Completely automated Template/Mold for other use cases based on HL7 transactions Timeline Research/Development: June 2009  October 2009 (Prototype) Research/Development: June 2010  October 2010 (Production) Upgrade/Maintenance: March 2011  October 2011 (Version 3)

Patient Identification Matching and Linking East Side Hospital System Under Test ADTA04 – Register Patient (Master Patient ID - 101) PIX Source Simulator (Patient Registration) 1 PIX Manager Application XDS 1a ACKA04 – Acknowledgement RSPK23– Get Corresponding Identifier Response (Validate for response with master ID 101) 4 ADTA04 – Register Patient (Local Patient ID – F432) QBPQ23 – Get Corresponding Identifiers (Query with local ID F-432) 3 2 PIX Source Simulator (Patient Registration) PIX and Document Consumer Simulator 2a ACKA04 – Acknowledgement Retrieve Document St. Mary’s Medical Center

PIX/PDQ Test Tool: Operational Process INTERACTION/REPORTS END-USER (VENDOR) MANUAL OR AUTOMATED SUT Web Application Interface NIST PIX/PDQ HL7 v2/v3 TEST TOOL MESSAGES (TEST OBJECTS) SYSTEM UNDER TEST (SUT) STIMULUS OR RESPONSE (MESSAGES) (via the communication protocol) SPECIFICATIONS (test material that defines test assertions)

Test Case : Patient Match, Link , and Retrieve PIX/PDQ Test Tool System Under Test 1 ADTA04 – Register patient “ALAN ALPHA” (ID = PIX10501) in domain “HIMSS2005” End User (Vendor) NIST PIX Source Test Agent Vendor PIX Manager Application 1a ACKA04 – Send acknowledgement ADTA04 – Register patient “ALAN ALPHA” (ID = XYZ10501) in domain “XREF2005” 2 2a ACKA04 – Send acknowledgement ADTA04 – Register patient “CARL SIMPSON” (ID = ABC10501) in domain “HIMSS2005” 3 3a ACKA04 – Send acknowledgement Web Application Interface NIST PIX Consumer Test Agent 4 QBPQ23 – Send query message for patient “ALAN ALPHA” in domain “HIMSS2005” seeking the patient ID in domain “XREF2005” 4a RSPK23– Send response indicating that the identifier (“XYZ10501”) for patient “ALPHA” was found in domain “XREF2005” QBPQ23 – Send query message for patient “CARL SIMPSON” in domain “HIMSS2005” seeking the patient ID in domain “XREF2005” 5 5a RSPK23 – Send response message indicating no data was found “NF”

Select Test

View Test Steps and Run

View Interactive Log

Summary Results

North American Pre-Connectathon Results (Fall 2010)

Summary of Vendor Test Results

Vendor Saved Test Results

Community Involvement: PIX/PDQ Google Group 2009/2010 Prototype: ~60 Group Members; ~25 vendors completed tests 2010/2011: > 200 Group Members; ~70 vendors completed tests so far

Testing Infrastructure Robert Snelick March 21, 2011

Motivation Group has built tooling for various healthcare IT domains HL7 V2, HL7 V3, IHE XDS, CDA, Patient Care Devices, NHIN, IHE PIX/PDQ Typically built test tools one at a time (independently) We have had many successes and have positively impacted industry However, we can’t build tools fast enough one at time Problem MU reportable lab tool took 4 man months (immunization the same) In some instances we needed to become domain experts (e.g., lab); probably not the best use of NIST resources Muddled down in administration of tests/maintenance of tools—instead of innovation Demand greatly outpacing industry/NIST capabilities to produce tools Recognized a need to work more efficiently Need to “build tools to build tools”, we’re heading in that direction… V2 Tool Kit (leveraged for CCHIT Message Generation, PIX/PDQ, PCD, connectathon, NHIN, Meaningful Use) External validation services (V2 and CDA); common interface/reporting More and more to come… ARRA funding allocated to build TI in 2009 (but not yet awarded) Group has traditionally worked with and built tooling for various healthcare IT domains HL7 V2, HL7 V3, XDS, CDA, Patient Care Devices, NHIN Built test tools one at a time (independently) At the end of the day you’ve built one test tool (which is great, but…) In the process we need to gain domain expertise; is this really a NIST function? Necessary to some extent to test our tooling methodology (and the IT part) Many shortcomings with specifications and testing process became apparent Demand greatly outpacing industry capabilities to produce test tools Recognized a need to work more efficiently Need to “build tools to build tools” Don’t duplicate tooling (not only at NIST but in the community), we’re starting… V2 Tool Kit (leveraged for CCHIT Message Generation, PIX/PDQ, PCD, connectathon) External validation services (V2 and CDA); common interface/reporting More and more to come… Easier said then done—more applicable in some areas then others TI is not a do over; we’ll build on or adapt existing toolkit design/code base—not everything can be assimilated (cost/benefit)

Testing Infrastructure Heading in this direction with the HL7 Projects Proceeding with pilot HL7 to better understand requirements Testing model is widely applicable HL7 V2 IG, CDC , IHE, PHIN, NHIN, HITSP, ISDS, etc Flood of implementation guides For example, MU to add 3 more HL7 IGs and also more elaborate testing scenarios Important that our tools are leveraged and we need to further engage industry experts Easier said then done—more applicable in some areas then others TI is not a do over (migration); we’ll build on or adapt existing toolkit design/code base—not everything can be assimilated (cost/benefit) Build TI with targeted (production) tools in mine build out TI components for specific tools (develop abstract design along the way) End-to-End PIX/PDQ End-to-End PCD (collaborate with working group from start) NHIN (Patient Discovery) ONC MU (Lab, Immunization, Bio-surveillance, Patient Management) Transfer technologies to SDOs (HL7, IHE, ONC, etc)

How can we Address the Increased Demand for Tools? Implementation Guide Authoring Tools (Create more “testable” specifications!) Implementation Guides (Number is increasing rapidly—no testing tools available) HL7 v2 MU Bio-Surveillance NHIN HL7 V3 Patient Discovery HL7 v2 MU incorporate Lab Results & Reportable Lab Results MU Immunization Reporting IHE HL7 v2 PCD Technical Framework PIX/PDQ IHE HL7 v3 IHE HL7 v2 and v3 In General HL7 v2 and v3 Guides in General (HL7, CDC, PHIN, ISDS, ONC, etc.) 20+ IHE integration profiles using V2! Test Case Management Tool (Let domain experts help write the test cases!) V2 Implementation Guide Authoring Tool Many HL7 V2 Implementation Guides are being written IHE, CDC-public health, some specified for meaning use All with different style, conformance requirement interpretations, etc. Written in text; hard to extract conformance requirements V2 is a different beast then newer XML standards But V2 is here to stay and MU/IHE will continue to use Tools are lacking—MU test development made that clear Provide a mechanism to produce consistent HL7 V2 IG Models V2 message definition and constraints Explicitly defines conformance requirements that can be processed automatically Publish Text (English) version of the Implementation Guide Pre-defined IG Skeleton Modifications made in tool and republished Test Case Management Tool Creates, Maintains, and Manages Test Plans Embedded Standard Constraint Model or Imports Test Artifacts Supports Multistep Test Cases (Test Scenarios) Defines Each Test Case (Test Steps) Builds Test Environment Defines transactions, actors, and sequence to execute tests Defines generation context files and validation context files Defines configuration parameters Creates executable test script in XML format Independent of any execution engine Generates English Document Test Plan Describes test cases and expected results Provides traceability to specification reference Overall has to incorporate/use results of the other tools TI Goal: Tool Building Development Environment Build testing tools more rapidly and with better correctness IHE has 40/50 integration profiles with 100’s of transactions 30/40 actors; V3, V3, DICOM, XDS, CDA New specifications being developed each year New tests being developed each year NIST can build Testing Infrastructure and Demonstrate Usefulness (PIX/PDQ as one example) then replicate Industry/Domain Experts can use TI to develop Test Tools Bulk of work completed with TI Industry supplies test agent functionality, writes test cases, test data, validation criteria Show industry “how to fish” Applies to other industry organizations and SDOs CDC/Public Health(PHIN), HL7, Meaningful Use, etc. Test Cases and Testing Artifacts Test System (Allow NIST/others to build tools quickly)

“Reusable” Test System Phase 3 HL7 Test Tool Phase 1 SDOs Implementation Guide Authoring Tool Test Flow Engine User Interface Test Agent Support Testers HL7 v2 Test Agent Core Test Cases Test Case Management Tool Phase 2 End-User (Vendor) Test Agent Support Message Generation and Validation Test Engine Web Application Test Management Test Agent Core Communication Real- Time Log Vendor System Under Test Test Agent Support Current/Planned Projects with Testing Infrastructure To be developed under the testing infrastructure umbrella IHE PIX/PDQ V2/V3 End-to-end Specify the integration using the Implementation Guide Authoring Tool Build/manage Test Cases using the Test Case Management Tool Execute Tests using components of the Testing Infrastructure V2/V3 Message Generation Services V2/V3 Message Validation Services Data Service—Test Data Development Registry/Repository Services PIX/PDQ Test Agents SOAP Validation Service Test Harness Logging Service Test Analyzer Service Reporting Service Meaningful Use 2011 (Lab Results and Immunization) Meaningful Use 2013 Test Agent Core Reports HL7 v3 Special Validation MLLP (v2) or SOAP (v3) NIST Server/Other Server Tool Builder Tasks

Broader View: Conceptual View of TI System Components Specification (IG) Authoring Tools STAKEHOLDERS NIST TEST INFRASTRUCTURE SYSTEM SUT Test System Instance Resource Repository System(s) Under Test HITSP User Interface Test Cases, Machine-Readable Test Scripts Templates, Common File Format Specifications Test System Control Certification Body Test Environments Supported Instance Testing Test Harness Isolated System Testing Workflow Management /Execution Engine Test Analyzer Peer-to-Peer System Testing NHIN Test System Development Components Services Test Management Tools and Test Systems Test Agent Proxy Evaluation Agent Vendors Test Initiation and Test Identification Management Test Framework Test Framework Test Framework Test Case Creation Implementers Testing Infrastructure Key Objectives Provide a variety of testing services to health IT stakeholders Support conformance and interoperability testing for numerous healthcare messaging and document data exchange standards Provide a set of reusable/flexible components and services “A Framework for Building Test Systems—an SOA Approach” Assortment of testing infrastructure uses Enables user customization Support different delivery mechanisms including APIs, desktop applications, web applications, and web services Provide a feedback loop to enhance the healthcare standards Integrated existing tools when appropriate Collaborate and harmonize with other tooling efforts Roll out tools and resources incrementally Generation Test Data Validation Report Remote Services Configuration Settings Management Existing Tool Time Security Logging Log Analyzer Other Industry Consortia User Repository Management Registry/ Repository Aggregated Specialized Other Services External Service Real-Time Monitor

Tool Building Development Environment Meaningful Use Web Application Validation Tool PIX/PDQ Pre-Connectathon Test Tool NHIN Test Tool IHE Connectathon Test Tool General CDA Validation IHE XDS V2 Toolkit (includes communication framework) V3 Tools General HL7 V2 tools (Message Maker, Validation W/S, app) IHE CDA (General CDA + more) IHE PCD (HL7 V2 Toolkit/Validation + more) IHE PIX/PDQ (HL7 V2 Toolkit + Test Agents + more) IHE PIX/PDQ (V3 Tools + CDA + XDS + more) MU CDA (General CDA + more) MU HL7 V2 Immunization (HL7 V2 Toolkit/Validation + more) MU HL7 V2 Lab Results (HL7 V2 Toolkit/Validation + more) NHIN (XDS + V3 PIX/PDQ + more) Modularized/Flexible/Reusable Components Able to react quickly to test tool needs, e.g., Meaningful Use Enable the healthcare community/domain experts to build tools Provide a low barrier of use; high return

Summary What have we done? Lessons Learned/Industry Needs HL7 Tool Kit (V2 and V3) Tools General HL7 Utilities Message Maker CCHIT Incorporate Lab Message certification Meaningful Use (Stage 1: Reportable Lab Results and Immunization) IHE PIX/PDQ Pre-connectathon and connectathon IHE PCD Pre-connectathon and connectathon Lessons Learned/Industry Needs Demand for tools increasing; need a community effort Need tools to build tools Need to react quickly to requests for tools Need more automated tools What are doing? Improving HL7 conformance standards (HL7 V2 conformance co-chair) Building core components and infrastructure Building prototypes and tools to work smarter HL7 V2 End-to-End Test System (PIX/PDQ and PCD) Implementation Guide Authoring Tool Test Case Management Tool Test System Support NHIN Testing Building reference test data (messages) Where are we headed? Build out broader testing infrastructure components Support more elaborate meaningful use scenarios Virtual Interoperability Test bed Engage industry to adopt tools Transfer technology such that others can build the tools (e.g., ONC, IHE, etc.)