Conformance Lynne S. Rosenthal Mary Brady National Institute of Standards and Technology

Outline Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies –XML, DOM, XSLT, –XSL FO, XML Schema Conformance Testing Certification Certificates Branding Conformance clause Test suitesTest tools Validation Procedures, reporting

Unscrambling the Terminology Conformance Specification Conformance Clause Conformance Testing test suite, test tools Validation process, reporting Certification certificate, brand

Terminology - Conformance CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2) –These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance

Terminology - Conformance Testing CONFORMANCE TESTING – a way to verify implementations of a specification to determine whether or not deviations from the specifications exist (through the use of test suites) –tests capture the technical requirements within the specification –testing is bound in scope by the specification –internal structure of product-under-test not accessible (black box testing) Necessary, but not sufficient, for interoperability Note: referred to as conformity assessment in ISO Guide 2

Testing Methodology Falsification Testing –find errors by means of experimentation –show presence of errors not their absence –prove non-conformance; can never prove conformance Tests Based on: –required functionality –logical errors by misunderstanding requirements –errors from boundary conditions and divergence –common programming errors OutputInput

Conformance Tests Requirements Test programs Specification Implementation

Terminology - Testing Process VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite –ensures that testing can be repeatable and reproducible –ensures that conclusions are consistent with facts presented in the evaluation CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.

Testing Ingredients Standard Conformance clause Conformance Testing Test assertions Test suite (test software, test scripts, test criteria) Validation Process - policy and procedures for testing Certification qualified bodies to do the testing and certification control board - advisory and arbiter

Benefits of Conformance Testing Increased probability that products are implemented correctly –Contains required functionality –Behaves as expected –Performs functions in a known manner Increased likelihood of portability and interoperability –Portability - move software or applications among different systems –Interoperability – ability of 2+ systems to exchange and use information

Benefits of Conformance Testing cont. For Developers of Specifications –Feedback loop For Software Developers –Help to improve quality - identify areas in which they conform or not conform For Buyers –Increases buyer’s confidence in a product and its ability to meet their needs –Substantiate seller’s claims –Not locked into purchasing from a single vendor

Components of a Test Suite Test Harness Test Cases Test Scenario Test Assertions, Axioms, Semantic Requirements Specification

OASIS XML Conformance XML Conformance Committee –Develop tests to improve the quality of XML processors –Provide a forum to discuss issues related to interoperability –Provide input to W3C regarding errors/ambiguities Deliverables –Test Suite containing over 2000 tests, Second Edition –Associated test report –Built using XML/XSLT

XML Test Suite Types of XML Tests –Binary Validating processors must check validity constraints. Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read. –Output Tests Requirements on XML processors to ensure that they pass the necessary information on to applications.

XML Test Matrix Non-validating Validating External Entities Ignored External Entities Read Valid DocumentsAccept Invalid DocumentsAccept Not-wf DocumentsReject WF Errors tied to External Entity Accept (varies)Reject Documents with Optional Errors (not specified)

XML Test Suite Conformance.xml XML Test Description FilesTest Case DTD Testcases.dtd Test Profile Test URI Unique ID Spec Section Type Entities Description NIST / OASIS SUN Microsystems James Clark FUJI Xerox IBM Test Report XSL Stylesheets IE5, HTML renditions Tests the Element Valid VC by including an undeclared child element. XML.COM Java application Tests XML processors Dave Brownell Outputs

XML Test Suite Results, v1 XML.COM Conformance Update – Dave Brownell Validating Parsers –Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V –Pass Rate 82% - 99% Non-validating Parsers –Tested all of the above plus XP 0.5 –Pass rate 86% - 99% Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett –Non-validating: DOM: 95%, SAX: 97% –Validating: 97%

NIST DOM Test Suite Test Harness – automatically runs available tests and presents user with a color-coded table of results; gives access to spec, assertions, and source code Test Scenario Load appropriate XML or HTML file, exercise DOM calls Test Assertions – Fundamental, Extended, HTML Interfaces DOM L1 Specification 800 ECMAScript fundamental, extended, HTML 200 Java – fundamental, extended Test Cases

DOM Test Suite Test Results: –Run tests against IE5, Mozilla (some fundamental interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98% –Implementations differ greatly in the way that parsers handle white-space and entity expansion issues – differences in DOM tree DOM Level 2 –Working with W3C to identify resources –Modified Java tests to be consistent with SUN contribution –Will update NIST tests to L2 status

XSLT Test Suite OASIS Technical Committee, chaired by G. Ken Holman Member contributions include Lotus, NIST, Sun Expands on XML Test Description file Dave Marston, XSLT/XPATH Conformance Wed, 4pm, Delaware Room Public Information Meeting, Wed, 8pm, Suite 8228

XSL FO Test Suite W3C WG to test Candidate Recommendation Develop test assertions for all basic formatting objects and properties Develop test DTD and associated harness Work with vendors to categorize available tests Research translating available CSS and XSLT tests Develop tests for additional spec components Expected March 2001

XML Schema Test Suite Test Control Test Types # schemas # instances Constraints Structure Limits Special info Schema Tests Test Generator

Summary Interesting URL’s – – – Questions???