1. IHE International Meeting Gazelle Project Steve Moore, MIR Eric Poiseau, INRIA

3. Introduction IHE is expanding  New domains  New participating countries  More companies involved Need better testing process  Need for conformity testing  Need for interoperability

4. 4 Participation to the European Connect-a-thon

5. What we have learned Our Experience :  8 years of connect-a-thon  8 years of MESA tools  Connect-a-thon on 3 continents ! Need to distinguish between :  Conformance testing  Interoperability testing

6. Objectives of the project Provide IHE with a new set of test tools :  For more in depth testing during IHE connect- a-thon  For companies internal testing  For testing between companies  For Healthcare Enterprise QA/Acceptance tests

7. IHE International Meeting Conformance Testing

8. Conformance Testing (1/2) Is unit testing  Tests a single ‘part’ of a device Tests against well-specified requirements  For conformance to the requirements specified in the TF and the referenced standards  Usually limited to one requirement per test. Tests at a 'low' level  At the protocol (message/behaviour) level. Requires a test system (and executable test cases)  Can be expensive, tests performed under ideal conditions

9. Conformance Testing (2/2) High control and observability  Means we can explicitly test error behaviour  Can provoke and test non-normal (but legitimate) scenarios  Can be extended to include robustness tests Can be automated and tests are repeatable Conformance Testing is DEEP and NARROW  Thorough and accurate but limited in scope  Gives a high-level of confidence that key components of a device or system are working as they were specified and designed to do

10. Limitations of Conformance Testing Does not prove end-to-end functionality (interoperability) between communicating systems  Conformance tested implementations may still not interoperate This is often a specification problem rather than a testing problem! Need minimum requirements or profiles Does not test a complete system  Tests individual system components, not the whole A system is often greater than the sum of its parts! Does not test functionality  Does not test the user’s ‘perception’ of the system Standardised conformance tests do not include proprietary ‘aspects’  Though this may well be done by a manufacturer with own conformance tests for proprietary requirements

11. IHE International Meeting Interoperability Testing

12. Is system testing  Tests a complete device or a collection of devices Shows that (two) devices interoperate  within a limited scenario ! Tests at a ‘high’ level (as perceived by users)  Tests the ‘whole’, not the parts  Tests functionality Does not necessarily require a test system  Uses existing interfaces (standard/proprietary) Interoperability Testing is BROAD and SHALLOW  Less thorough but wide in scope  Gives a high-level of confidence that devices (or components in a system) will interoperate with other devices (components)

13. Limitations of Interoperability Testing Does not prove interoperability with other implementations with which no testing has been done  A may interoperate with B and B may interoperate with C. But it doesn’t necessarily follow that A will interoperate with C.  Combinatorial explosion Does not prove that a device is conformant  Interoperable devices may still interoperate even though they are non-conformant Cannot explicitly test error behaviour or unusual scenarios  Or other conditions that may need to be forced (lack of controllability)  Has limited coverage (does not fully exercise the device) Not usually automated and may not be repeatable

14. IHE International Meeting Conformance or Interoperability Testing ?

15. Conformance or Interoperability Both are Needed :  Complementary, not competitive  ETSI : « While it is not absolutely necessary to undertake both types of testing, the combined application of both techniques gives a greatly increased confidence in the tested product and its chances of interoperating with the other similar products ETSI : European Telecommunications Standards Institute

16. IHE International Meeting Gazelle Year 1 Plan

17. Year 1 target Prototype  evaluate the design  measure our estimates  Applied to selected profiles but designed to work for all (known ones) Radiology Scheduled Workflow (SWF) Laboratory Scheduled Workflow (LSWF) Patient Identity Cross-referencing (PIX) Audit Trail and Node Authentication (ATNA)

18. Year 1 development plan 1 st Quarter  HL7 & Dicom conformance tools (selected profiles)  Integration Statement Management  Application Logging Subsystem 2 nd Quarter  Web system for Interoperability and conformance Tests  HL7 & Dicom data sets 3 rd Quarter  Simulated Actors  User Management 4 th Quarter  Report and evaluation

19. Year 1 Resources IHE Europe  2 FTE  1 FTE Inria IHE North America  2 FTE (RSNA-MIR)  NIST (XDS) ?  IHE Canada / Infoway ? IHE Asia  IHE-Japan (Translation work, Spec review)

20. Consequences Mesa Tools  NO NEW DEVELOPMENT  LIMITED SUPPORT BY STEVE Kudu  NO NEW DEVELOPMENT  LIMITED SUPPORT BY ERIC But Connectathon Management Continues !

21. IHE International Meeting Questions ?