Interoperability Testing

Work done so far WSDL subgroup Generated Web Service Description with aim for maximum interoperability between various SOAP stacks WSDL tested against tooling supplied by  JAX-RPC RI  AXIS .NET Basic and informal interop tests .NET AXIS

WSDL subgroup (cont ’ d) Released a subgroup report (Andre) Philosophy followed  e.g. choosing document/literal style Extension mechanism Found problems/concerns Raised bug reports against the stacks  AXIS begun work on all of them .NET reported fixes were in progress  JAX-RPC RI has not responded yet Fixes/workarounds reflected in the WSDL and interfaces

Interoperabilty WSDL subgroup work Allows various stacks to generate code from WSDL definitions Ensures interoperability on SOAP level Need for interoperability on WSRP protocol level cross-vendor testing Compliance testing ?

Cross-vendor testing Release open source project which provides sample implementation Allows vendors to test their Producers and Consumers against this implementation Does not guerantee compliance Vendors willing to make their Producers accessible to others for interop tests? Allows others to test their Consumers In turn vendors return feedback on their Producer implementations Requires similar (spec) level of implementation  Should be easy once version 1 is released Still no compliance testing in a formal sense

Cross-vendor Testing - Matrix Consumer /Producer Vendor1Vendor2Vendor3Vendor4 Vendor1 Vendor2 Vendor3 Vendor4

Compliance Testing Testing a particular implementation for compliance with the specification. Compliance testing is strictly “ black box ” testing

Compliance Testing - Tasks Develop a set of testable assertions which represent the specification Derive testable assertions from the specification Determine test cases implied by each assertion Develop a Compliance Test Suite available for public download so that vendors and customers can verify that their implementations are compliant with the WSRP 1.0 Version of the spec Design, develop, and test the tests for each assertion Create a test framework to run, evaluate, and report on the tests

Deriving Testable Assertions Significant Effort to recast the spec as assertions Requires committee involvement

Test Suite Framework A technical framework for running compliance tests. The framework shall be used to run, evaluate, and report test cases. Should be highly scriptable and configurable so test cases may be added easily This may be a significant effort by itself, depending on our ability to leverage existing frameworks

Testing Producers Develop a Consumer which executes test cases against a Producer and reports the results. Test cases are configured as XML input Reports are generated as XML output

Testing Consumers Develop a RI Producer conforms to the spec (I.e. passes all the compliance tests) produces reports on the behavior of the Consumer, if in error. For example, validating the state that was supposed to be kept and returned by the Consumer

Testing - Roadmap Form interop test subgroup Start interop tests February, 15th? Establish interoperability April, 15th Form compliance test subgroup Draft assertion suite March, 15th Finalize assertion suite May, 1st

Interop Mike Alan Richard Andre Nigel Subbu Gil(?)

Compliance Ross Dan Rich

Backup

Existing Examples Java Community Process Calls for developing a Technology Compatibility Kit (TCK) side by side with Reference Implementation Well developed process and tools W3C The W3C standardization process only produces specification documents (Recommendations, etc), and does not formally address compliance testing. Apache Each project defines its own testing practices and standards

WS-I Testing Working Group 2 Deliverables The Test Tools Development Working Group will produce the following deliverables:  1. WS-I Test Methodology White paper  2. WS-I Test Tools Specification  3. Two or more implementations of Test Tools and supporting documentation  4. Assertions and Test Conditions used as input to the Test Tools  5. WS-I Tool Configuration Template  6. WS-I Experience Report: Tool Development as a component of WS-I Profile Development 2 WS-I Testing Working Group Charter,

Scope of the Task 1 Derive testable assertions from the specification Determine test cases implied by each assertion Design, develop, and test the tests for each assertion Create a test framework to run, evaluate, and report on the tests 1 TCK Project Planning and Development Guide, Sun Microsystems, Inc

Existing Frameworks JUnit -- open source project BaRT – Batch generator for Regression Tests -- IBM open source tool used for testing JVMs. Sun ’ s Compatibility Test Toolkit – licensed, limited to JCP projects

Testing Optional Functionality Separate, or at least configurable, test suites for discrete levels of function The discrete levels of function currently mentioned are Simple and Sophisticated Finer distinctions may have to be made Perhaps the test suite can determine at runtime which level of function the Producer supports and be able to apply the correct tests.