Department of CIS University of Pennsylvania 1/31/2001 Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642

Department of CIS University of Pennsylvania 1/31/2001 Outline Introduction –Specifications, implementations –Conformance relations –Tests, testing architectures –Assumptions, etc. Formal protocol specifications Finite-state machine (FSM) testing Extended FSM (EFSM) testing Test generation for StateCharts

Department of CIS University of Pennsylvania 1/31/2001 Testing problem A specification gives a description of the system behavior. An implementation should behave according to the specification. Is this really true? c a b a b a b

Department of CIS University of Pennsylvania 1/31/2001 Black-box vs. White-box testing Black-box: unknown system structure –Testing against reference specification –Use the specification interface for test selection, coverage criteria, result analysis White-box: known system structure –Testing against reference specification –Use system structure for test selection, etc.

Department of CIS University of Pennsylvania 1/31/2001 Protocol testing architectures Tester Local architecture IUT IUT – implementation under test PCO PCO – point of control and observation

Department of CIS University of Pennsylvania 1/31/2001 Protocol testing architectures Upper Tester IUT PCO 1 PCO 2 underlying service layer Lower Tester Distributed architecture Coordinated architecture Remote architecture

Department of CIS University of Pennsylvania 1/31/2001 General testing architecture IUT PCO Tester

Department of CIS University of Pennsylvania 1/31/2001 Tests and test suites A test case defines: –A finite sequence of inputs –A finite sequence of expected outputs A test suite is a collection of tests to achieve a certain test coverage Testing process –test generation: construct a test suite –test application: execute each test –result evaluation: interpret outputs

Department of CIS University of Pennsylvania 1/31/2001 Test structure Test purpose –e.g., identify a state of the implementation Test preamble –lead the IUT into known state Test body –invoke behavior corresponding to the test purpose Checking of test results

Department of CIS University of Pennsylvania 1/31/2001 Modeling approach Testing criteria on a concrete system are hard to formulate implementation detailed specification ? conformance relation Abstract models make reasoning easier abstract model of specification abstract model of implementation assumptions and test hypothesis assumptions and test hypothesis

Department of CIS University of Pennsylvania 1/31/2001 Outline Introduction Formal protocol specifications –Formal specification languages Finite-state machine (FSM) testing Extended FSM (EFSM) testing Test generation for StateCharts

Department of CIS University of Pennsylvania 1/31/2001 Formal specification languages Extended FSM –SDL –Estelle –StateCharts Process Algebras –LOTOS Algebraic Specification –Z, VDM

Department of CIS University of Pennsylvania 1/31/2001 Outline Introduction Formal protocol specifications Finite-state machine (FSM) testing –FSM model –FSM fault models –FSM test selection Extended FSM (EFSM) testing Test generation for StateCharts

Department of CIS University of Pennsylvania 1/31/2001 FSM model A set of states An initial state Finite sets of input (X) and output (Y) events Transfer function x1x1 x2x2 x3x3 Output function /y 1 /y 2 /y 3

Department of CIS University of Pennsylvania 1/31/2001 FSM-based fault model Output faults Transfer faults Transfer faults with additional states Additional or missing transitions Control and data flow faults?

Department of CIS University of Pennsylvania 1/31/2001 Testing based on FSM models Testing methods based on control flow –Transition-tour (TT) method a single sequence of inputs to traverse all transitions simple but weak –Unique-input-output (UIO) method a test to identify each state in the specification –Distinguishing sequence (DS) method identify each state by the same test body provides full coverage, but may not have a DS

Department of CIS University of Pennsylvania 1/31/2001 UIO testing method Assumption: for each state, there is an input sequence that produces a unique output Each test identifies one state s1b/1 s2a/0a/0 b/1 s3b/1a/1 preamble body s1s1 s2s2 s3s3 b/1 b/0 a/1 a/0

Department of CIS University of Pennsylvania 1/31/2001 Outline Introduction Formal protocol specifications Finite-state machine (FSM) testing Extended FSM (EFSM) testing –EFSM model –EFSM fault models –EFSM test selection Test generation for StateCharts

Department of CIS University of Pennsylvania 1/31/2001 EFSM model A set of states An initial state Finite sets of input (X) and output (Y) events Transition relation x1x1 x2x2 x3x3 A finite set of variables (V) – Guards b1 b1  b2 b2  b3 b3  {output y 1 } {update v 2 } {output y 3 } – Update blocks

Department of CIS University of Pennsylvania 1/31/2001 Tests for EFSM models EFSM executions depend on input signals and data A test consists of: –test sequence –test data Executability problem: –Find data to execute the test sequence Undecidable, in general 

Department of CIS University of Pennsylvania 1/31/2001 Executability problem introduces an additional step in the testing process: –test generation: construct a test suite –test application: execute each test –result evaluation: interpret outputs Test validation –test validation: keep only executable tests –test application: execute each test –result evaluation: interpret outputs

Department of CIS University of Pennsylvania 1/31/2001 Control and data flow testing EFSM executions are data-dependent Control flow FSM testing methods are not adequate for EFSM models Data flow testing methods account for data dependencies

Department of CIS University of Pennsylvania 1/31/2001 Data flow: definitions and uses Data variables are –defined by inputs and updates (def) –used in updates (c-use) guards (p-use) Data-flow graph captures data dependencies

Department of CIS University of Pennsylvania 1/31/2001 Data flow graph Directed graph with nodes labeled with definitions and uses of variables v=0  {input u} u<0  {v:=u+1} p-use v def u p-use u c-use u def v

Department of CIS University of Pennsylvania 1/31/2001 Data-flow coverage criteria all-def –test suite covers each definition at least once all-use –cover each def-use association at least once all-du-paths –exercise all paths from each definition of a variable to every its use.

Department of CIS University of Pennsylvania 1/31/2001 all-use coverage Find at least one definition-free path for every def-use association def v use v no definitons of v

Department of CIS University of Pennsylvania 1/31/2001 Outline Introduction Formal protocol specifications Finite-state machine (FSM) testing Extended FSM (EFSM) testing Test generation for Statecharts

Department of CIS University of Pennsylvania 1/31/2001 EFSMs (FSMs + variables) + concurrency + hierarchy + communication + real-time Widely used for specifying real-time embedded HW/SW controllers Also used in most of object-oriented methodologies, e.g., UML Statecharts

Department of CIS University of Pennsylvania 1/31/2001 A coffee vending machine Statecharts = off power-on/ light-on; m:=0 power-off/ light-off Hierarchy + not empty inc/m:=0 dec[m=1]/m:=0 inc/m:=m+1 dec[m>1]/m:=m-1 money EFSMs + busyidle coffee on done/stop Concurrency + coffee[m>0]/start Real-time tm(coffee,[3,5])/stop Communication + coffee[m>0]/start;dec

Department of CIS University of Pennsylvania 1/31/2001 Test sequences for Statecharts The main purpose –What should implementations do? {power-on}/{light-on}, {inc}/{}, {coffee}/{start} –What should not implementations do? {power-on}/{light-on}, {coffee}/{} The main issue –How can we generate a finite and reasonable number of test sequences?

Department of CIS University of Pennsylvania 1/31/2001 State coverage off power-on/ light-on; m:=0 power-off/ light-off not empty inc/m:=0 dec[m=1]/m:=0 inc/m:=m+1 dec[m>1]/m:=m-1 money busyidle coffee on done/stop coffee[m>0]/start tm(coffee,[3,5])/stop coffee[m>0]/start;dec

Department of CIS University of Pennsylvania 1/31/2001 Transition coverage off power-on/ light-on; m:=0 power-off/ light-off not empty inc/m:=0 dec[m=1]/m:=0 inc/m:=m+1 dec[m>1]/m:=m-1 money busyidle coffee on done/stop coffee[m>0]/start tm(coffee,[3,5])/stop coffee[m>0]/start;dec

Department of CIS University of Pennsylvania 1/31/2001 Data flow oriented coverage off power-on/ light-on; m:=0 power-off/ light-off not empty inc/m:=0 dec[m=1]/m:=0 inc/m:=m+1 dec[m>1]/m:=m-1 money busyidle coffee on done/stop coffee[m>0]/start tm(coffee,[3,5])/stop coffee[m>0]/start;dec

Department of CIS University of Pennsylvania 1/31/2001 Test generation from Statecharts (I) Basic idea –Transforms Statecharts into EFSMs Advantage –Can reuse the existing methods and tools for EFSMs –Can handle infinite state space Disadvantage –Cannot determine the executability of test sequences

Department of CIS University of Pennsylvania 1/31/2001

Department of CIS University of Pennsylvania 1/31/2001 Test generation from Statecharts (II) Basic Idea –Test generation can be reduced into model checking –TG: for each state s, is there a path leading to s –MC: a temporal logic formula EFs express this property Advantage –Fully automatic –Generates only executable test sequences Disadvantages –Can handle only finite state space

Department of CIS University of Pennsylvania 1/31/2001 Another perspective Testing classes in object-oriented programs –Can we show an CLASS implementation conforms to a Statechart specification? –Variables of Statecharts: data members of classes –Events of Statecharts: member functions of classes Testing inheritance in object-oriented programs –Can we reuse the test sequences of a super-class for testing its subclasses ?

Department of CIS University of Pennsylvania 1/31/2001 A coffee-cocoa vending machine off power-on/ light-on; m:=0 power-off/ light-off busyidle cocoa on not empty inc/m:=0 dec[m=1]/m:=0 inc/m:=m+1 dec[m>1]/m:=m-1 money tm(coffee,[3,5])/stop cocoa[m>0]/start;dec busyidle coffee tm(coffee,[3,5])/stop coffee[m>0]/start;dec Vending Machine Coffee Vending Machine Cocoa Vending Machine Coffee- Cocoa Vending Machine