Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 2 Overview lConformance Testing Methodology and Framework ISO IS-9646 Literature: “An Overview of OSI Conformance Testing”

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 3 Overview: This Lecture lProtocol conformance testing: why and what lStandardization lOverview Standard IS-9646: Conformance Testing Methodology and Framework nConformance: static and dynamic nOptions, protocol implementation conformance statement nTest generation: test purposes, abstract-, executable tests nTest methods, test architectures, nTest classification and -structuring nTest notation nTest implementation and -execution

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 4 Types of Testing

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 5 Protocol Conformance Testing distributed systems  exchange of information  rules for communication : protocol  standardized protocol specifications  conforming implementations  testing for protocol conformance

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 6 Protocol Conformance Testing International Standard ISO IS-9646 : Conformance Testing Methodology and Framework lHow to test protocol entity implementations for conformance with respect to their specifications lOriginally for OSI systems lNow more generally applied lDefines test language TTCN lStandard accepted by ITU-T, ETSI, ATM-Forum, ……

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 7 Protocol Conformance Testing lStarting point : a correct and valid (standardized) protocol specification lOriginal goal : standardization of test suites in TTCN nconsistency between specification and test suite ? lConformance testing  interoperability testing

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 8 The Conference Protocol

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 9 CPE UDP Layer CPE The Conference Protocol join leave send receive Conference Service

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 10 Conference Protocol Test Architecture CPE = IUT UT-PCO = C-SAP LT-PCO UDP Layer U-SAP LT-PCO Tester TorX BC A

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 11 (N)-ASPs (N)-service provider (N)-Protocol Entity (N)-PDUs (N-1)-ASPs (N-1)-service provider (N)-Protocol Entity protocol (N-1)-ASPs Protocol Conformance Testing vs. Interoperability Testing

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 12 Protocol Conformance Testing vs. Interoperability Testing lProtocol Conformance Testing: ncorrect protocol (design and validation) nconforming protocol entities (testing)  interoperability lInteroperability testing ntest directly whether each pair of protocol entities can work together (i.e. deliver (N)-service) lInternet: ninteroperability testing with (standard) reference entities

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 13 Standard Conformance Testing lTesting by: nmanufacturers nusers organizations ntelecom operators nindependent test laboratories lStandard IS-9646 for ncertification of implementations naccreditation of testing processes and test laboratories ninternational acceptance of test results

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 14 ISO IS lPart 1 :Introduction and general concepts lPart 2 :Abstract test suite specification lPart 3 :TTCN Tree and Tabular Combined Notation lPart 4 :Execution of tests lPart 5 :Requirements on test laboratories and administrative conformance assessment process

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 15 test generation test execution test implementation standard protocol specification standard conformance test suite verdict implementation process protocol implementation IUT process of conformance testing executable test suite

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 16 Conforming Implementation lSpecifications in natural language lWhat is a conforming implementation ? l“ … exhibits conformance if it complies with the conformance requirements of the applicable specification standard …” lSatisfaction of conformance requirements lConformance requirements : “the implementation shall do ….. “ “the implementation shall not do …..” lComplication : options

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 17 Conforming Implementation l(P)ICS - (Protocol) Implementation Conformance Statement : nlist of implemented options nsupplied by manufacturer l(P)ICS proforma nfill-in questionnaire with all possible options nattachment to protocol standard nto be completed by manufacturer to make PICS lRestrictions on combinations of implemented options ndocumented in standard in static conformance requirements

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 18 Example : Static Conformance Requirement lFeature A is optional lFeature B is optional lBut A may only be implemented if B is also implemented

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 19 Conforming Implementation lRequirements on dynamic behaviour: dynamic conformance requirements lMessages : nstructure ncoding lOrdering : nsequence of sent and receive messages nrelation between information content of different messages lExample: nWhen in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase.

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 20 Conforming Implementation lA conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the PICS lConformance testing: nchecking static conformance by reviewing nchecking dynamic conformance requirements by testing

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 21 Test Generation dynamic conformance requirement  test purpose(s)  generic test case  abstract test case  executable test case  test case execution

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 22 Test Generation lTest purpose : precise description (natural language) of what is to be tested in order to decide about satisfaction of a particular conformance requirement lExample: nWhen in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase. nCheck whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. (but implementation may also enter the abort phase!)

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 23 Test Generation lGeneric test case: operationalization of test purpose; description of actions to be performed to reach test purpose lExample: nCheck whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. nGo to state Enable, send a Data-Request, and observe a Data-PDU, and if not check that the implementation starts entering the abort phase

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 24 Test Generation lAbstract test case: take test method into account lTest method ( test architecture, test environment ) : abstract description of environment in which testing will be performed, in terms of PCO - Points of Control and Observation lExample: DoConnect-Request, check Connect-Response (now in state Enable), send Data-Request observe Data-PDU or observe Start-Abort

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 25 IUT LT UT (N)-ASPs (N)-PDUs PCO (N-1)-ASPs test coordination Local test method Test Methods

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 26 IUT LT UT (N)-ASPs (N)-PDUs PCO (N-1)-ASPs test coordination service provider test systemSUT Distributed test method Test Methods

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 27 Test Notation lAbstract test cases are written in standardized notation : TTCN : Tree and Tabular Combined Notation lSpecification language for Abstract Test Cases lBehaviour specified as tree of sequences of possible events lData specified in tables or in ASN.1 lISO IS-9646 Part 3 lWith:input ?, output ! (from tester perspective!), timers / time out, ordering of events, default behaviour, alternative behaviours, tree attachment (=subroutines), parallel component testers, verdicts: pass, fail, inconclusive

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 28 behaviour Test Case Dynamic Behaviour Test Case Name:Conn_Estab Group:transport/connection Purpose:Check connection establishment with remote initiative constraintsverdict +preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE pass fail inconclusive fail Example : TTCN

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 29 Classification of Tests lBasic interconnection tests lCapability tests lBehaviour tests lConformance resolution tests

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 30 Structuring of Test Suites Test suite Test case Test step Test event PCO ? XPCO ! A Test group

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 31 Test Implementation lTransform abstract test suite into executable test suite by ncompilation ninterpretation na test slave ….. lSelect test cases from abstract test suite based on PICS lIncorporate (P)IXIT - (Protocol) Implementation eXtra Information for Testing : naddress information ntimer information nspecific parameter values n….. lPIXIT proforma supplied by test laboratory

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 32 Test Execution lExecute / interpret executable test suite lCollect and analyse the results based on verdicts pass, fail, inconclusive of each test case execution lCollect all results of nstatic conformance review ndynamic testing into Protocol Conformance Test Report

© Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 33 implementation process static conformance review test execution analysis of results test selection test implementation standardized test methods test purposes generic test suite test notation verdict test report certification PIXIT proforma standard protocol specification PICS proforma dynamic conformance req’s static conformance req’s protocol implementation IUT PIXIT PICS (standardized) abstract test suite Basic interconn. capability behaviour executable test suite