1. LRI Validation Suite LRI Validation Suite Meeting Rob Snelick—NIST March 27th, 2012

2. Agenda Tool status and plans Tool updates Test Case/Messages Update Development of “Negative” and Scenario test cases Schedule -- Tuesdays, 1:00 PM EST – April 24 th – May 22 nd – Additional meetings to be added if necessary – HL7V2_LRI_Testing Google Group will be used for notification

3. Tool Status and Plans Validation Tool for LRI (Prototype for trial use) LRI IG Version 15 (will remain) – lri-phase-2– – http://lri.sipilotdevelopment.org/lri-phase-2/ http://lri.sipilotdevelopment.org/lri-phase-2/ LRI IG In-progress – lri-phase-2b – Updating tool according to latest “published” version Moving target Some updates are in and others are not Tool has latest features and validation rules – http://lri.sipilotdevelopment.org/lri-phase-2b/ http://lri.sipilotdevelopment.org/lri-phase-2b/ LRI IG In-progress – lri-phase-ballot2 – Target is to make available by April 24th, 2012 – Still will be a work in-progress but validation should be current and stable (“production version”  by July 2012) – Start to move into the phase of adding test cases – Changes to the tool will still be made without notice – lri-phase-2b will show incremental updates

4. Resources Wiki Page—Validation Suite Resources – Latest version of test plans, messages, etc Google Group – Discussion forum for LRI IG, Pilots, Validation Tool, Tool and Resources Updates, Release Notes, etc – http://groups.google.com/group/hl7v2_lri_testing

5. Tool Updates Profile/Tooling updates to the latest IG version Configuration and Regeneration – Local test cases Expanded/modified views – Profiles – Juror Documents – Data Sheets Division of NG/GU Profiles/Test Cases Spreadsheet Processor Tool – Documentation to come

6. Phase 1 Core Test Messages Erythrocyte sedimentation rate – GU and NG versions now available – 30341-2 OBR.4 LN code – Valid Minimally Populated Message – Final Results – Valid Typically Populated Message – Final Results – Valid Typically Populated Message – Corrected Results – Valid Maximally Populated Message – Final Results – Valid Specimen Reject Message CBC W Auto Differential panel in Blood – NG version coming soon – 57021-8 OBR.4 LN code – Valid Typically Populated Message – Final Results – 27 Results (OBXs) Lipid 1996 panel in Serum or Plasma – GU and NG version now available – 24331-1 OBR.4 LN code – Valid Typically Populated Message – Final Results – 4 Results (OBXs) Culture Results Reporting (Relating OBX segments) – Final review needed – 625-4 OBR.4 LN code – GU and NG versions available; RU/RN coming soon Culture and Susceptibility Results Reporting (Parent/Child Relationships) – Final Review Needed – 625-4 OBR.4 LN code – GU and NG versions available; RU/RN coming soon A subset of these messages will be included in the final version of the LRI IG Start to develop scenario and “negative” tests

7. “Negative” Testing Intentionally implant errors in messages Messages sent to the EHR Test EHR How should EHR handle the Errors What are the cases that should be tested: Examples: – Missing required elements (e.g., Patient ID, OBX.3, ORC.3) – Observation value indicates an abnormal value; abnormal flag indicates a normal result – Message Date/Time sequence incorrect; e.g., result time is earlier than specimen collection time – Invalid code values (e.g., Invalid LONIC code, Results Flag of “Z”) – What are the other cases that should be included?

8. Scenario Testing Scenario Testing: a sequence of messages valid or invalid to test EHR behavior Results state machine: – Valid: P  F  C – Invalid: P  C Snapshot Processing – Valid: Partial Results  More Results (all known sent)  all results sent – Invalid: Partial Results  Additional results sent (previous known results not resent) Reportable Lab Results to Public Health – LAB Message  EHR  Reportable Lab result to PH – Validate Lab result message to PH based on MU-1 IG – Correlate to original message sent to EHR Others?

9. LRI Validation Suite WG Charter Overview Ambulatory EHR ONC S&I Framework Test Data LIS LRI Validation Tool EHR LRI Test Harness NIST LRI Test Tool 1.Suite scope limited to the requirements specified in the LRI IG 2.Develop test data 1.provided & verified by the S&I Framework community 3.LIS LRI Validation Tool 1.Used to validate vendor LIS systems & Test Harness 2.Develop a test plan 3.Capture and validate LIS LRI messages 4.EHR LRI Test Harness 1.Used to validate EHR systems 2.Develop a test plan 3.Simulation of an LIS system 4.Manage, Send, Receive LRI Messages 5.Inherently provides CLIA Testing 6.Targets 1.S & I Framework EHR Pilots 2.Vendor EHR Products 3.MU Certification LRI ORU R01 LRI ACK R01 S & I Framework EHR Pilots Vendor EHR Products Ambulatory EHR Systems 1.EHR Pilots 1.Draft standard trial use (implementation) 2.Verifies LRI IG can be implemented—provide feedback 3.Use LRI test harness to perform conformance testing 4.Function of the S&I Framework LRI Pilots WG 2.Vendor Product EHRs 1.Can use LRI test harness to perform conformance testing 2.LRI Test Harness expected to be used for MU certification Part of validation suite deliverablesAnticipated users of validation suite products

10. LRI Validation – High Level Overview LIS Mode – Context-Free – Context-Based (Generated Data Sheets from Test Cases – Scenarios) EHR Mode – Generated Data Sheets and Juror Documents from Test Cases – Scenarios Conformance Profile Browser Vocabulary Browser

11. LIS (or Proxy) LIS Test Tool  Load  Cut/paste  Send HL7 V2 Lab Results Message Validation Report Context Free Testing: 1.Context free testing will validate a LRI message created by the LIS 2.The context (e.g., the type and results of the lab test) is unknown to the validation tool 3.Therefore not all conformance requirements of the LRI implementation guide can be assessed 4.However, the validation provides a simple and convenient method for testing message structure and most vocabulary Process: 1.An LRI message is created by the vendor’s LIS 2.The message is sent, pasted, or loaded in the test tool 3.A validation is performed LIS Context-Free Testing

12. LIS LIS Test Tool select test case Test Data Sheet Manual entry of test data  Load  Cut/paste  Send HL7 V2 Lab Results Message Use Case Test Case Validation Report EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data Lab Results Data Sheet John Doe DOB: 05/23/1959 Gender: M Result: 13.7 g/dL Range 13.2 – 16.2 Status: Final and more… A lab test is ordered for a patient The specimen is collected, and is received and processed in the lab The lab result is produced and stored in the LIS database The lab result message is created The lab result is transmitted to an ambulatory EHR The lab result is viewed in the ambulatory EHR Process: 1.A technician enters lab results data into LIS based off the data sheet provided 2.The message is sent, pasted, or loaded in the test tool 3.A validation is performed Context-based Testing: 1.Context-based testing will validate a LRI message created by the LIS 2.The context (e.g., the type and results of the lab test) is known to the validation tool 3.Therefore all conformance requirements of the LRI implementation guide can be assessed 4.Supports various scenarios (F  C, rejected specimen) LIS Context-based Testing

13. EHR Lab Message ACK Validation LRI EHR Test Harness Inspection Testing can be performed by: 1.On-site inspection 2.Over a webex like technology 3.Screen-scraper or screen-capture (include clock) 4.Printed Reports Inspection Testing Techniques 1.EHR display screens 2.Database access 3.Configuration files Communication Automated Testing Acknowledgement Message Limited Utility LRI Test Message Validation Report Test Data Test Case Use Case Juror Document ACK LRI EHR Testing – Test Harness

14. LIS LIS Test Tool select test case Test Data Sheet Manual entry of test data  Load  Cut/paste  Send HL7 V2 Lab Results Message Use Case Test Case Validation Report EHR transmits Hemoglobin blood test order to the LIS for John Doe along with pertinent demographic and order request data Lab Results Data Sheet John Doe DOB: 05/23/1959 Gender: M Result: 13.7 g/dL Range 13.2 – 16.2 Status: Final A lab test is ordered for a patient The specimen is collected, and is received and processed in the lab The lab result is produced and stored in the LIS database The lab result is transmitted to an ambulatory EHR The lab result is viewed in the ambulatory EHR 1.Local code 2.No code—text 3.LOINC code No LOINC given for results Test tool inspects OBX.3 for a specific LOINC code Inspect for 718-7 only or inspect for one of 718-7, 20509-6, 30313-1, 30352-9, 30351-1, and 30350-3? For Discussion: 1.No suggested method given (Should we?) 2.With no method given we should expect one from a set of valid LOINC codes for this order (Lab dependent?) 3.If a specific method is given (is this typical/possible?) then should we look for a specific LOINC code in the sent message? For Discussion: 1.Does the LAB expect a local code for the order in a electronic order request? 2.Use Cases to consider: agreement/no agreement between LIS and EHR for local codes It is assumed Lab has capabilities for manual entry Data Entry DB Message

15. EHR Lab Message ACK Validation LRI EHR Test Harness Communication LRI Test Message Validation Report Test Data Test Case Use Case Juror Document ACK 718-7 (LN) SNameValueREF Range FHemoglobin13.713.2 – 16.2 g/dL For Discussion: 1.What are the requirements on the EHR with regards to LOINC— given the LRI IG, CLIA, and meaningful use? 2.What are the EHR display (GUI) requirements for the received lab results? 3.What are the EHR storage requirements for the received lab results? 4.Given that the LOINC code is likely to be translated into the EHR internal representation, what lab test names need to be displayed in exact form and what lab test names can be displayed in an equivalence text (same concept)? 5.What qualifies as “equivalence—same concept”? How is that determined? Can it be? Expert inspector only? Should displaying of the LOINC text (only) be the requirement—i.e., no mapping? 6.What is the impact of sending only the LOINC code (i.e., no local code) on testing? Translated into internal representation Displayed as internal display text representation