1. HL7 V2 Test Case Authoring and Management Tool Robert Snelick National Institute of Standards and Technology May 14 th 2012 Contact: rsnelick@nist.gov

2. 2 Specification to Testing: An End-to-End Approach Proposed methodology and tools to support the development HL7 V2 implementation guides and testing of systems that implement the implementation guides End-to-End Scope –Implementation Guide Authoring and Management –Test Data Authoring and Management –Message Generation and Inspection Document Generation –Test Case Authoring and Management –Testing Infrastructure Test Engine –Choreography Validation Engine –Messaging Validation –Functional Validation Communication Platforms Leverage and Extend Existing Concepts and Tools –VA Messaging Workbench (MWB), NIST Message Maker and Testing Infrastructure

3. 3 What will the tools address? Standards Test Cases Test Execution Test Scripts Results Analysis Test Material  specification of test cases  implementation guide development  processable test cases  automated or human analysis  juror document or test report  execution of the test scripts  healthcare use case development  create requirements to guide the standards development  create inspection document  provide validation test report  create inspection document  provide validation test report  XML message profiles  vocabulary  validation context files  XML message profiles  vocabulary  validation context files  test or certification managers determine pass/fail based on results  ensure use case requirements are satisfied  create testable assertions criteria  ensure use case requirements are satisfied  create testable assertions criteria  tests cases should test all requirements specified in the implementation guide  methods and tools to transform test cases into machines process- able instructions Use Cases  artifacts to support testing  test engine interprets test scripts  validate message against test material  test engine interprets test scripts  validate message against test material Implementation  implementation  application implements the standard Test Data  development of test messages  messages support test cases  valid and invalid  messages support test cases  valid and invalid Implementation Guide Authoring and Management Tool (IGAMT) IGAMT and TCAMT Test Case Authoring and Management Tool (TCAMT) Message Maker Tool Testing Infrastructure (Test Engine) Test Case Authoring and Management Tool (TCAMT) Vendors Testing Infrastructure (Validation) and TCAMT Testing Organizations

4. 4 How can we Address the Increased Demand for Tools? Test System (Allow NIST/others to build tools quickly) Implementation Guide Authoring Tools (Create more “testable” specifications!) Test Cases and Testing Artifacts Test Case Management Tool (Let domain experts help write the test cases!) Implementation Guides (Number is increasing rapidly—no testing tools available) HL7 v2 MU Syndromic Surveillance Implementation Guide ELINCS HL7 v2 Lab Orders and Results Implementation Guide HL7 v2 MU incorporate Lab Results & Reportable Lab Results Implementation Guides HL7 v2 MU Immunization Reporting Implementation Guides IHE HL7 v2 PCD Technical Framework IHE HL7 v2 PIX/PDQ Technical Framework HL7 v2 MU Lab Results Interface (LRI) Implementation Guide IHE HL7 v2 Technical Framework In General HL7 v2 Implementation Guides in General (HL7, CDC, PHIN, ISDS, ONC, etc.) 20+ IHE integration profiles using V2!

5. 5 Proposed Use and Goals Integrated Tooling Develop tooling to automate process as much as possible –Component integration—one tool output flows into the next E.g., Conformance profile and conformance statements created during implementation guide development can directly be used by validation engine –Machine processable message profile specification Used and built upon throughout and the development and testing cycle Constraints as specified in Implementation Guide Authoring Tool are used directly by validation engine Goal is to enable domain experts to develop artifacts –Tooling will better enable domain experts to produce quality implementation guide requirements and test criteria HL7 Balloted Implementation Guides should use the tool to create (template) –Ensure consistent definition and use of HL7 V2 Conformance Concepts –Ensure conformance constructs specified correctly and completely –Ensure compatibility with the standard or document variances Implementation Guides should have companion test plans complete with example/test messages –A quality level for implementation guides can be establish –Test plans can be balloted by HL7 –Developing test plans in coordination with the implementation guide development promotes better understanding of the requirements and force clear conformance requirements –Establish Implementation Guide Best Practices—Enforce with tooling –With HL7 CGIT WG participation the ONC S&I Framework Lab Results Interface (LRI) Implementation Guide and NIST’s development of the LRI Test Plans and Test Tool is an example that is heading in the right direction

6. 6 Tool Component Overview Implementation Guide Authoring and Management –Messaging Profiling and IG Document Generation Test Data Authoring and Management –Test Data Organization, Library Creation, and Management Message Generation and Inspection Document Generation –Assembly of Test Data, Message Generation, and Juror Inspection Documents Test Case Authoring and Management –Support complete Test Plan Life Cycle –Incorporates Conformance Profiles, Test Messages, Inspection Documents –Support Scenario and Application Functional Behavior Testing (Actor based) Testing Infrastructure –Test Engine Choreograph a sequence of transactions –Validation Engine Messaging Validation Functional Validation –Communication

7. 7 Implementation Guide Authoring & Management Tool The Big Picture: HL7 V2 End-to-end Testing Support HL7 v2 Standard Message Definition General Transaction Constraints (e.g., IG, TF, or user input)¹ HL7 v2 Standard Value Sets Test Case Requirements Vocabulary Library (XML) Conformance Profile (XML) Test Case Management and Authoring Tool Message Profiling Test Plan Execution Script (XML) Test Plan Template HL7 V2 Impl. Guide Template Test Plan English Document Data Sheets Impl. Guide English Document Specification Validation Context File (XML) ¹ Support existing specifications Use Cases Vocabulary Profiling Use Case Development Disposition Traceability Utilities IG Template Wizard Constraint Generator Test Case Development Test Script Generator Data Sheet and Juror Document Generator Test Plan Development Document Generator Message Maker Data Management Message Generation Testing Infrastructure Document Generator Execution Engine Validation Engine Test Agents Report Generation Communication Testing Environments Capabilities Artifact Repository Testing Tools Web Applications Testing Infrastructure Components Test Plan Integration Validation Resource Bundle Management Message Generation Engine Vocabulary Libraries (XML) Conformance Profiles (XML) Test Case Specific Context Files (XML) Domain Specific Tools Juror Documents Utilities Documentation Communication Resource Bundle Test Plan Execution Script (XML) R. Snelick May 8 th, 2012 Domain Experts Domain Experts and Test Case Developers User InputTool Generated Software Engineers Tool Developers Import Value Sets DITA XML

8. 8 Test Case Management and Authoring Tool Creates, Maintains, and Manages Test Plans Embedded Standard Constraint Model or Imports Test Artifacts Supports Multistep Test Cases (Test Scenarios) –Defines Each Test Case (Test Steps) –Builds Test Environment –Defines transactions, actors, and sequence to execute tests –Defines generation context files and validation context files –Defines configuration parameters Generate Data Sheets Generate Juror (Inspection) Documents Incorporate Message Maker 2 Creates executable test script in XML format –Independent of any execution engine Generates English Document Test Plan –Describes test cases and expected results –Provides traceability to specification reference Overall has to incorporate/use results of the other tools

9. 9 TCAMT Design Overview Test Case Requirements Vocabulary Library (XML) Conformance Profile (XML) Test Case Management and Authoring Tool Test Plan Execution Script (XML) Test Plan Template Test Plan English Document Data Sheets Impl. Guide English Document Specification Validation Context File (XML) Constraint Generator Test Case Development Test Script Generator Data Sheet and Juror Document Generator Test Plan Development Message Maker Data Management Message Generation Document Generator Vocabulary Libraries (XML) Conformance Profiles (XML) Test Case Specific Context Files (XML) Juror Documents R. Snelick May 8 th, 2012 Domain Experts and Test Case Developers User InputTool Generated

10. 10 Conceptual View of System Components Registry/ Repository Test Data Test Data Aggregated Validation Proxy Security Generation Logging Log Analyzer Log Analyzer Services Time Specialized Other Services Other Services Test Agent Evaluation Agent NIST TEST INFRASTRUCTURE SYSTEM Test Harness Workflow Management /Execution Engine Resource Repository Templates, Common File Format Specifications Test Cases, Machine- Readable Test Scripts Test System Development Components Test Case Creation Test Initiation and Test Identification Management Configuration Settings Management User Repository Management Certification Body NHINOther Industry Consortia Implementers HITSP STAKEHOLDERS Vendors Test Management Tools and Test Systems Report Test Environments Supported Peer-to-Peer System Testing Test Analyzer System(s) Under Test SUT Real-Time Monitor Test Framework Test System Instance Instance Testing Isolated System Testing User Interface Test System Control Remote Services Remote Services Existing Tool External Service

11. 11 Isolated System Test Environment E.g., IHE PIX Testing using a Web Application Client Report IHE PIX Consumer Test Agent IHE PIX Consumer Test Agent HL7 V2 Message Generation HL7 V2 Message Generation IHE PIX Source Test Agent IHE PIX Source Test Agent HL7 V2 Message Validation HL7 V2 Message Validation ServicesTest Management Router/Logger/Proxy Vendor System Under Test Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files Test Artifacts Conformance Profiles HL7 Tables Validation Context Files Generation Context Files IHE PIX Client Test Scenario IHE PIX Client Test Scenario Results HL7 V2 Message Validation Reports Results HL7 V2 Message Validation Reports Test Harness (Java Code) Test Harness (Java Code) Test Execution Web Application Client

12. 12 Test Plan Structure and Format Introduction and Overview Test Plan  Actors  Transactions  Test Cases Test Cases  Definition of Test Cases  Reference to Conformance Requirements  Test Setup  Actor Description  Configuration  Generation Context File  Test Data  Test Steps  Assertions  Expected Results List of Extracted Conformance Requirements and Test Case Mappings Example Test Messages  Driver Messages  Expected Results Messages HL7 V2Test Plan Title Associated Files  XML Conformance Profile  XML Table Library  XML Conditional Constraint Definitions  XML Static Constraint Definitions (formats, etc)  Context Value Specifications

13. 13 Original of HL7 V2 Assertions Standard Defined Conformance Requirements –Ingrain to standard; often special case –Built-in checks; design to turn on/off in authoring and validation tools Message Structure and Element Constraint Requirements –Conformance Message Profile or Standard Message Definition Implementation Guide Defined Conformance Requirements –Defined in constraint language; currently validation context file Dynamic Interchange Requirements –Acknowledgement Requirements –Transport Protocol Requirements Application Functional Behavior Requirements –Sequence of Test Cases –Data Content Requirements –Defined in validation context file

14. 14 Test Case Management Tool Instant Validation –Multiple Data Sets –How do we handle the complex data sets? –Right now we store in complex Excel Spreadsheets? –E.g., MU testing today Isolated System Validation –Scenario Testing –E.g., IHE Pre-connectathon testing today Peer-to-Peer System Testing –Interoperability Test Bed –Future

15. 15 TCAMT Detailed Design Conformance Profile (XML) HL7 Table Library (XML) Implementation Guide English Document Actor Builder Message Builder Interaction Builder List of Actors List of Messages List of Interactions User Driven Table Library Message Level Assertions User Driven Table Library Message Level Assertions Interaction Level Assertions Transaction Builder List of Transactions User Driven Table Library Message Level Assertions Interaction Level Assertions Transaction Level Assertions Assertion Generator Assertion Generator Assertion Generator Test Case Library Builder Test Case Builder Test Plan Builder Test Case Builder Test Step Builder Test plan List of Test Cases List of Test Steps User Driven Table Library Message Level Assertions Interaction Level Assertions Transaction Level Assertions Test Step Level Assertions Test Case Execution Script (XML) Table Library (XML) Validation Contexts (XML) Assertion Generator Message Generator Message Generation Context Message Contexts (XML) Test Suite (Test Cases) English Document Test Plan Table Library (HL7 + User’s) Table Library (HL7 + User’s) TC Library (Actor, MSG, Inter, Transac) TC Library (Actor, MSG, Inter, Transac) Test Assertion (TC, Transc, Inter, MSG) Test Assertion (TC, Transc, Inter, MSG) Message Generation (Message Context, Message example) Message Generation (Message Context, Message example) Test Suite (Test Plan Doc, Test Case Doc, Test Case Scripts) Test Suite (Test Plan Doc, Test Case Doc, Test Case Scripts) TCAMT Knowledge Base Library Authoring Library Reusing Test Case Authoring Library & TC Reusing Test Case Level Assertions

16. 16 Level of AssertionAssertion TypeTarget Data Value (Present, Value, RegExp) Single Message Correlation Conditional Data Value (Present, Value, RegExp) Single Message Correlation Conditional Multi Correlation Assertion Single Value Check Copy Data Value (Present, Value, RegExp) Single Message Correlation Conditional Multi Correlation Assertion Single Value Check Copy Assertions for TCAMT Message Level Assertion Interaction Level Assertion Test Step Level Assertion Transaction Level Assertion Test Case Level Assertion PCD01 DORDOC PCD01 DORDOC PCD01 ACK DOR (TA) DOC (SUT) DOR (TA) DOC (SUT) PCD01-001 ACK-001 PCD01-002 ACK-002 PCD01-001 ACK-001

17. 17 TCAMT Process ActorMessageInteractionTransactionTest StepTest Case Message Level Assertion Interaction Level Assertion Message Level Assertion Interaction Level Assertion Test Step Level Assertion Transaction Level Assertion Test Case Level Assertion Message Generation Context SUT Information Test Plan Information Library Test Case Use Case Scenario Use Case Definition Conformance Profile Table Library Supplement Validation Context

18. 18 Test Case Schema

19. 19 Test Case Archive DOR (TA) DOC (SUT) PCD01-001 ACK-001 PCD01-002 ACK-002 TCAMT Test Plan Description TCAMT Test Case Script XML TCAMT Message Generation Context PCD01-001 PCD01-002 TCAMT + imported Validation Context (Assertions) ACK-001 ACK-002 HL7 Conformance Profile PCD01 ACK HL7 Table Library PCD01 ACK

20. Test Case Authoring and Management Tool (TCAMT) - Screen shots National Institute of Standards and Technology April, 2012

21. 21 CREATE LIBRARY

22. 22 Home TCAMT Library Menu Welcome Message

23. 23 List of Actors Button to add a new Actor Table List of existing Actors Button to save/close Select Command to delete or duplicate

24. 24 Create Actor Button to create Actor Form to create Actor

25. 25 List of Messages Button to add a new Message Select Command to delete or duplicate Table List of existing Messages Button to save/close

26. 26 Create Message Form to create Message Button to create Actor

27. 27 List of Interactions Button to add a new Interaction Select Command to delete or duplicate Table List of existing Interaction Button to save/close

28. 28 Create Interaction Form to create Interaction Button to create Interaction

29. 29 List of Transactions Button to add a new Transaction Select Command to delete or duplicate Table List of existing Interaction Button to save/close

30. 30 Create Transaction Form to create Transaction Button to create Interaction Select Interaction for each line

31. 31 CREATE TEST PLAN & CASES

32. 32 Home Link for Test Plan & Cases Welcome Message

33. 33 List of Plans Button to add a new Test Plan Table List of existing Test Plans

34. 34 Create Test Plan Button to create Test Plan Form to create Test Plan

35. 35 List of Test Case Button to add a new Test Case Table List of existing Test Cases Details for Test Plan

36. 36 Create Test Case Button to create Test Case Form to create Test Case

37. 37 View Meta Data for Test Case Metadata for Test Case Link to Test Steps

38. 38 Define Test Steps Button to add a new Test Step Table List of existing Test Steps Select Actor Tested

39. 39 Export Test Case Script Select to export Test Case

40. 40 CREATE ASSERTIONS Message Level Assertion

41. 41 Link to Assertion Generator Link to Message Level Assertion Generator

42. 42 Message Level Assertion Generator Table Tree of Message Selected Tree Node (Yellow Highlighted) Button to manage Assertions Button to import assertions from existing Validation Context File Button to export assertions to a new Validation Context File

43. 43 Manage Assertion Draggable selected Nodes Dropable Form to create assertions List of existing assertions

44. 44 Create DataValue Assertion Meta Information Form for Assertion DataValue Form (PlainText/ RegExp/Presence) for Assertion Location of Target Node (Dropped Node) Add/remove check value

45. 45 Create Single Message Correlation Location of Target Node (First Dropped Node) Location of Comparison Node (Second Dropped Node) Meta Information Form for Assertion

46. 46 Create Conditional Assertion Meta Information Form for Assertion If Part Form (It is same as DataValue Interface.) Then Part Form (It is same as DataValue Interface.) Else Part Form (It is same as DataValue Interface.)

47. 47 CREATE ASSERTIONS Interaction Level Assertion

48. 48 Link to Assertion Generator Link to Interaction Level Assertion Generator

49. 49 Interaction Level Assertion Generator Table Tree of Interaction Button to manage Assertions Button to import assertions from existing Validation Context File Button to export assertions to a new Validation Context File Interaction Level Assertion Interface as same as Message Level interface

50. 50 CREATE ASSERTIONS Transaction Level Assertion

51. 51 Link to Assertion Generator Link to Transaction Level Assertion Generator

52. 52 Transaction Level Assertion Generator Table Tree of Transaction Selected Tree Node (Yellow Highlighted) Button to manage Assertions Button to import assertions from existing Validation Context File Button to export assertions to a new Validation Context File

53. 53 Create Correlation Assertion Draggable selected Nodes Comparison Node A (First Dropped Node) List of existing assertions Comparison Node B (Second Dropped Node) Select correlation mode

54. 54 CREATE ASSERTIONS Test Step Level Assertion

55. 55 Link to Assertion Generator Link to Test Step Level Assertion Generator

56. 56 Test Step Level Assertion Generator Table Tree of Test Step Button to manage Assertions Button to import assertions from existing Validation Context File Button to export assertions to a new Validation Context File Test Step Level Assertion Interface as same as Message & Interaction Level interface

57. 57 CREATE ASSERTIONS Test Case Level Assertion

58. 58 Link to Assertion Generator Link to Test Step Level Assertion Generator

59. 59 Test Case Level Assertion Generator Table Tree of whole Test Case Selected Tree Node (Yellow Highlighted) Button to manage Assertions Button to import assertions from existing Validation Context File Button to export assertions to a new Validation Context File Test Case Level Assertion Interface as same as Transaction Level interface