1 SAIF-Effects on Interoperability Reviews Baris Suzek Georgetown University Architecture/VCDE Joint Face-to-Face June,3, 2010 St. Louis, Missouri

22 Team Baris Suzek (Lead) Michael W. Riben Bob Freimuth Mukesh Sharma Lewis Frey Cui Tao Tony Pan Marty Humphrey Olga Tchuvatkina Natasha Sefcovic Riki Ohira Brian Davis Mike Keller

3 SAIF-Effects on Interoperability Review Goals Describe processes in caBIG for review that are taking place, should take place Describe the artifacts needed for review Describe key components; conformance statements, functional profiles, conformance profiles Describe the Knowledge, Skills and Abilities (KSA) of reviewers, and whether those KSA's exist in caBIG now Approach Development of draft outline for whitepaper in alignment with goals Weekly meetings/presentations Presentation of available resources/processes (e.g. NIST, NCI) Discussions around issues – sections of whitepaper Assignment of whitepaper sections to group members Links to Wiki:

4 SAIF and ECCF effects on the review SAIF/ECCF supports several reviews to support/assess interoperability: ECCF Specification Reviews Conformance Validation and Certification Compatibility Reviews These reviews may cover many viewpoints and/or abstraction levels, so, in fact, they contain many sub- reviews

5 Key Concepts Relevant to SAIF/ECCF Reviews - Conformance Conformance Statements Explicit testable representations of explicit assumptions made by the specification Can be at CIM/PIM/PSM levels for different viewpoints Can have hierarchical relationships to other conformance statements (e.g., increasing levels of specificity) Are also used to specify localizations

6 Key Concepts Relevant to SAIF/ECCF Reviews - Conformance Conformance Assertions Testable assertions linked pair-wise to specific Conformance Statement or Profile Verifiable as True or False at specific test points in an implementation

7 Key Concepts Relevant to SAIF/ECCF Reviews - Profiles Mechanisms used to constrain broader service capabilities Types: Functional Profiles: A group of related operations for a given specification (for computational viewpoint) that may be implemented independently from other profiles. Semantic Profiles: Subsets of the overall information content (static semantics) that supports the functional profiles (for informational viewpoint). Conformance Profiles: Combination of a set of functional and semantic profiles taken together to give a complete coherent set of capabilities.

8 Key Concepts Relevant to SAIF/ECCF Reviews – Semantic Profile

9 Key Concepts Relevant to SAIF/ECCF Reviews – Conformance Profile Functional and Semantic Profiles already defined in specification

10 Key Concepts Relevant to SAIF/ECCF Reviews – New Terms Compliance: Artifacts derived from other artifacts by traversal of successive levels of abstraction. Localization: Custom modifications or other alterations Conformance Statements/Profiles: Explicit testable representations of explicit assumptions. Conformance Assertions: Assertions against a conformance statement that can be verified as True or False. Compatibility : Relationship between two or more conformance statements involving two or more specification stacks.

11 ECCF Reviews ECCF Specification Reviews Conformance Validation and Certification Compatibility Reviews

12 ECCF Reviews – ECCF Specification Review Uses a review criteria list developed by Enterprise Service Specifications Team (ESST) Conducted by Composite Architecture Teams (CAT) on CIM, PIM, PSM level specifications where all RM-ODP viewpoints are represented Review goals relevant to interoperability include: To support interoperability between enterprise services that are conformant to the mature specification stacks (all necessary artifacts for viewpoints/abstraction levels are populated, no implicit assumptions) To provide a mechanism to assess interoperability between two or more services conforming to two or more specifications that can be within or across enterprises

13 ECCF Reviews – ECCF Specification Review Should: Check well-formedness/integrity of documents and/or artifacts Check traceability between abstraction levels Check consistency among viewpoints Check the validity of localizations and constraining of conformance statements/profiles/viewpoint artifacts Ensure explicitness and completeness of the specifications (e.g. all vocabulary standards used are listed for information viewpoint, all interactions are explicitly shown for computational viewpoint) Ensure enterprise service interactions/collaborations/dependencies reported in behavioral model are valid – e.g. checking compatibility of semantic/functional profiles between services

14 ECCF Reviews – ECCF Specification Review - Example Specification provide interactions/collaborations with other services. e.g. Adverse Event Service Specifications interactions: Subject Management Protocol Management Review should ask: Is the Subject or Protocol representation in AE compatible with other services? Is there interface specifications supporting “isProtocolValid”?

15 ECCF Reviews– Compatibility Reviews Conducted between one or more specification stacks that may come from different enterprises e.g. exchanging healthcare records between hospital services Review goal relevant to interoperability includes: To assess/review/demonstrate compatibility between specifications and consequently To assess the level (like parties shown in stairway to heaven figure) of interoperability that these two (or more) specifications can facilitate To identify requirements around transformations that need to happen between services conforming to these specifications Reuse of artifacts and tracking the reuse makes reviews easy: E.g. functional/semantic and/or conformance profiles reused by multiple services Needs a common language between enterprises (e.g. Information/Behavioral Framework), so currently not feasible

16 ECCF Reviews– Conformance Validation and Certification Conducted only for an instance/implementation Occurs after a system has been fully implemented Review goal relevant to interoperability includes: Validation of conformance of an implementation to a specification stack by testing the pair-wise conformance assertions it made against the conformance statements Validation of conformance of an implementation to conformance statements around interactions/collaborations facilitates the interoperability between enterprise services

17 ECCF Reviews - Relationship of Reviews Conformance Validation Adverse Events (.NET) Subject Man. (Java) Conformance Validation Interoperate Enterprise A Specification Review Conformance Validation Adverse Events (.NET) Interoperate ? Transformations? Compatibility Review Enterprise B

18 ECCF Reviews– Conformance Validation and Certification – Teams/Roles Service Developers Validation (Test) Team/Lab Conduct the conformance validation reviews Consists of ECCF viewpoint representatives Certification Authority Issues certificates to validated services Develop/maintain a validation/test policy Control Board Resolve technical questions or disputes related to validation process Adopted from NIST document on Conformance Tests:

19 ECCF Reviews– Conformance Validation and Certification - Process Submission of artifacts Conducting Validation Review/Tests Development of a test suite Deploying service to a test bed/sandbox environment Running conformance validation/tests Generation of validation/test report Certification Certifying the implementation based on the review Appeal (if needed)

20 ECCF Reviews– Conformance Validation and Certification- Test Suite Composed of three parts: Test Cases: A description of the purpose of the test with a reference to the conformance assertions in the specifications for which the test case is developed The pass/fail criteria or the expected outcome Test Documentation: A detailed description of how the testing is done to ensure reproducibility Test Tools. Programs, scripts, data, or instructions for manual actions required to perform the test “Ideally” allows fully-automated testing Unfortunately manual validation will be needed Adopted from NIST document on Conformance Tests:

21 ECCF Reviews– Conformance Validation and Certification - Artifacts CIM and PIM level specifications To validate the existence and completeness of ECCF service specification stack. To provide context for the review (e.g. business goals). PSM level specification To validate/test conformance assertions contextualized by the implementation Computable artifacts (not pictures) at PSM level To automate/semi-automate conformance validation by transforming conformance assertions to test cases A deployable service implementation with installation instructions

22 Example of one issue and resolution Issue: Not all participants are trained or familiar with SAIF/ECCF Limited time for rather complex questions and make recommendations Resolutions: Attempt to transfer knowledge through: Identification and distribution of relevant documentation provided by HL7, NIST, caBIG Training WG meeting presentations on calls summarizing resources, knowledge and existing processes

23 Conclusions SAIF/ECCF supports several reviews to support/assess interoperability: ECCF Specification Reviews Conformance Validation and Certification Compatibility Reviews The ECCF Specifications and their review are important to support enterprise and cross-enterprise service interoperability

24 Recommendations Training for ECCF reviews Organization of Review Submission Artifacts, supporting Collaborative documentation development (e.g. templates) Version control Centralized storage for all related artifacts Availability of a single latest version of each artifact Increased visibility a use/discovery of artifacts More computable conformance statements/assertions Not pictures Functional profiles Semantic profiles Not unstructured textual statements

25 Recommendations Testable conformance statements/assertions: Can you test “QueryXYZ will return results in 5 seconds” for all environments subjectively? Depends on load/hardware etc? Test/validation policy document What is being tested/validated? Who are the stakeholders/teams/roles? What are responsibilities of teams/roles? What are the certification cancellation/expiration policies? What are the privacy/confidentiality issues surrounding review? What is the result expected? How does the document look like and how it is going to affect decision making?

26 Recommendations Tools to automatically transform conformance assertions to test cases Tools to measure/audit “real-time” interoperability Logging/monitoring exceptions around interactions/exchanges Pilot conformance validation/certification process