Constraint-Based Modeling of InterOperability Problems Using an Object-Oriented Approach Mohammed H. Sqalli Eugene C. Freuder University of New Hampshire -The first author is currently at the of Canada Ltd. -This material is based in part on work supported by the National Science Foundation.

IAAI-01: Sqalli & Freuder2August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder3August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder4August 7, 2001 InterOperability Testing Specifications (What should Happen) Observations (What really happens) Communication line InterOperability Testing Compare (Specifications, Observations) Device B Device A

IAAI-01: Sqalli & Freuder5August 7, 2001 Example from a Test Suite l Test Case ID: V4202H__004 l Test Purpose: Verify that the first Hello sent from both sides contains Remote node ID and Remote port ID set to zero (1Way). l Pre-requisite: Both SUTs (A, B) are in different lowest level peer groups (Out). l Verdict Criteria: The first Hello packet observed from each SUT will have the Remote node ID field and Remote port ID field set to zero (1WayOut).

IAAI-01: Sqalli & Freuder6August 7, 2001 Example - Test Description Switch ASwitch B 1WayOut

IAAI-01: Sqalli & Freuder7August 7, 2001 Example - CSP Model Node IDPort ID Remote Node ID Status Source 1WayOut(A) Variables Unary Binary Constraints 1WayOut(A) {Source, Time, Status, Node_ID,...} 1WayOut(B) {Source, Time, Status, Node_ID,...}... 1WayOut(A).Status = Mandatory 1WayOut(A).Remote_Node_ID != 0 1WayOut(A).Time < 1WayOut(B).Time 1WayOut(A).Peer_Group_ID != 1WayOut(B).Peer_Group_ID Remote Port ID Type Peer Group ID Time Hello Node IDPort ID Remote Node ID Status Source 1WayOut(B) 1 0 Remote Port ID Type Peer Group ID Time Hello < != 00

IAAI-01: Sqalli & Freuder8August 7, 2001 ADIOP- Test Suite Builder GUI Panel showing the CSP model of a test case Panel for declaring variables and domains Panel for declaring constraints

IAAI-01: Sqalli & Freuder9August 7, 2001 ADIOP - Test Suite Menu GUI

IAAI-01: Sqalli & Freuder10August 7, 2001 Protocol Specification Monitored Observations Result Decoded Observations Test Suites Test #1 Test #2 … CSP Models CSP #1 CSP #2 … CBR + Expert Result Report OK NoYes ? Store Next Step Debugging Modeling Diagnosis Decoding ADIOP – The big picture (Four Modules) Update

IAAI-01: Sqalli & Freuder11August 7, 2001 ADIOP - Modeling Protocol Specification Test Suites Test #1 Test #2 … CSP Models CSP #1 CSP #2 … Modeling

IAAI-01: Sqalli & Freuder12August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder13August 7, 2001 InterOperability Lab l Verify the interoperability and/or conformance of the computer communications products l Consortiums: ATM, FDDI, Wireless, … l Protocols tested: UNI, LANE, PNNI, … l Test Suites for IOP testing: LANE, PNNI, … l Testing Equipment: Snifters, Generators,... l Testing Problems: Data Analysis, Similar Problems,...

IAAI-01: Sqalli & Freuder14August 7, 2001 ATM and PNNI l Asynchronous Transfer Mode – supports all classes of traffic: voice, video, and data – provides quick and easy switching and multiplexing – connection oriented l Private Network Network Interface – provides dynamic routing: includes discovery of the topology of the network – supports QoS and hierarchical routing – scales to very large networks

IAAI-01: Sqalli & Freuder15August 7, 2001 InterOperability Testing Network MonitorDevice A Device B Protocol Specifications Compare to Report Observations

IAAI-01: Sqalli & Freuder16August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder17August 7, 2001 Motivations – The ADIOP System l Diagnose IOP problems in a timely manner l Analyze a large amount of data l Keep trace of previous problems and their solutions to be reused in future similar cases l Debug IOP tests

IAAI-01: Sqalli & Freuder18August 7, 2001 Solution - The ADIOP System l Automate the process of InterOperability Testing – CSP modeling of IOP tests – Diagnosing IOP problems quickly and efficiently using CSP models (including Explanation) l Debug the InterOperability tests

IAAI-01: Sqalli & Freuder19August 7, 2001 Contributions l Model building: Simple, declarative, transparent, GUI-based, and complete modeling language for stating and generating IOP tests. l OO provides a more natural, concise, scalable and reusable framework for model statement of IOP tests. l Combination of CSP and OO: metavariables and metavalues to represent packets and generate test cases as objects. l Modeling is generic across protocols: independent of the protocol being used. l Scalability: More protocols can be made available for modeling by adding the corresponding decoders. l Dynamic loading of protocols and packets being used. l Many-models architecture: each test case is represented by a CSP model. l Constraint expressiveness: unary and binary constraints.

IAAI-01: Sqalli & Freuder20August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder21August 7, 2001 Constraint Satisfaction Problems Values Constraints Problem Statement VariablesCSP Algorithm Solution X ZY Coloring Problem CSP Representation Search Inference X: red Y: blue Z: green Red green blue Red green blue Red green blue Z X Y Z X Y

IAAI-01: Sqalli & Freuder22August 7, 2001 One Model vs. Many Models l One Model (Protocol Specification) [Sqalli & Freuder 1996] – Only one model to use for testing, but too complex – Less inconsistencies – No redundant testing l Many Models (Test Suite) [Sqalli & Freuder 1998] – Easy and small models for specific tests l To state l To fix/update – Suitable for interoperability testing l Generate reports l Preferred by vendors/testers – Explanation of the cause of failure

IAAI-01: Sqalli & Freuder23August 7, 2001 CSP Modeling of InterOperability Testing Protocol Specification Monitored Observations Result Decoded Observations Test Suites Test #1 Test #2 … CSP Models CSP #1 CSP #2 … NB: The decoders have built in functionality that provides the information (protocols, packets, packet’s fields) to build the CSP models.

IAAI-01: Sqalli & Freuder24August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder25August 7, 2001 Modeling with Objects Packet... TimeSourceTypeNode ID Object (Metavariable/ Metavalue) Time Source Type Node ID getNodeID getTime getPacket CSP Metavariable: X = {x 1, x 2, …, x n }, where x i is a variable Metavalue: V = {v 1, v 2, …, v n }, where v i is a value Binary Metaconstraint: C xy = {c 1, c 2, …, c m }, where c k is a constraint between x i and y j

IAAI-01: Sqalli & Freuder26August 7, 2001 Class Hierarchy Packet TimeSourceInfo PNNI Routing Packet TimeSourcePNNI Routing HeaderInfo Hello Packet TimeSourceNode ID... PNNI Routing Header Packet Mpoa PnniRoutLane Hello Dbs

IAAI-01: Sqalli & Freuder27August 7, 2001 Advantages of Object-Orientation l Inheritance between objects allows for a hierarchical definition of packets which matches the way protocols are specified. l Encapsulation – Information hiding of the objects definition as the users do not need to know the details but only the functionality. – Modularity: Each object is a separate entity with its own functionality. l Scalability: Adding more protocols and packets does not affect the already existing system. l Reusability: The same objects are being used for decoding packets as well as for CSP model definition. l Natural: The CSP model obtained using objects is concise and expressive.

IAAI-01: Sqalli & Freuder28August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder29August 7, 2001 Modeling Interface (GUI) Protocols List: loaded from the class hierarchy of the Packet class Packet Types List: loaded from the class hierarchy of the protocol’s class Packet’s Parameters (CSP variables) List: loaded from the class definition of a packet.

IAAI-01: Sqalli & Freuder30August 7, 2001 Advantages of CSP Modeling using OOP l No need for the tester to know what each packet contains (built in the hierarchy). l No need to know the syntax of the modeling language (generated by the GUI) l Possibility to add more protocols just by adding the decoder for such protocol (protocol independent modeling). l Dynamic loading of information about protocols, packets, etc. (done at runtime).

IAAI-01: Sqalli & Freuder31August 7, 2001 Example from a Test Suite l Test Case ID: V4202H__004 l Test Purpose: Verify that the first Hello sent from both sides contains Remote node ID and Remote port ID set to zero (1Way). l Pre-requisite: Both SUTs (A, B) are in different lowest level peer groups (Out). l Verdict Criteria: The first Hello packet observed from each SUT will have the Remote node ID field and Remote port ID field set to zero (1WayOut).

IAAI-01: Sqalli & Freuder32August 7, 2001 CSP Model Node IDPort ID Remote Node ID Status Source 1WayOut(A) Variables Unary Binary Constraints 1WayOut(A) {Source, Time, Status, Node_ID,...} 1WayOut(B) {Source, Time, Status, Node_ID,...}... 1WayOut(A).Status = Mandatory 1WayOut(A).Remote_Node_ID != 0 1WayOut(A).Time < 1WayOut(B).Time 1WayOut(A).Peer_Group_ID != 1WayOut(B).Peer_Group_ID Remote Port ID Type Peer Group ID Time Hello Node IDPort ID Remote Node ID Status Source 1WayOut(B) 1 0 Remote Port ID Type Peer Group ID Time Hello < != 00 Metavariable

IAAI-01: Sqalli & Freuder33August 7, 2001 Test Suite Builder GUI Step 5: This generates the test case object Step 1: Choose the protocol Step 2: Start CSP Model Step 3: Define the packets Step 4: Define the constraints

IAAI-01: Sqalli & Freuder34August 7, 2001 Example of CSP Modeling - Modeling Language - $CSP $PROTOCOLPnniRout $PACKETOneWayOutAHello $PACKETOneWayOutBHello $UNARY_CONSTRAINTOneWayOutA.status D_Mandatory $UNARY_CONSTRAINTOneWayOutA.remote_port_id == 0 $UNARY_CONSTRAINTOneWayOutB.remote_port_id == 0 $UNARY_CONSTRAINTOneWayOutA.remote_node_id == 0 $UNARY_CONSTRAINTOneWayOutB.remote_node_id == 0 $BINARY_CONSTRAINTOneWayOutA.time < OneWayOutB.time $BINARY_CONSTRAINTOneWayOutA.source != OneWayOutB.source $BINARY_CONSTRAINTOneWayOutA.peer_group_id != OneWayOutB.peer_group_id $ENDCSP Metavariables Optional statemen t Unary Constraints Binary Constraints Protocol

IAAI-01: Sqalli & Freuder35August 7, 2001 Modeling Language Definition l $CSP l $ENDCSP l $PROTOCOL protocolTested l $PACKET packet_name packet_type l $DOMAIN domain value1 value2 … l $UNRAY_CONSTRAINT variable operation [domain|value] l $BINARY_CONSTRAINT variable1 operation variable2 l $CONSTRAINT variable1 variable2 f(variable1, variable2)

IAAI-01: Sqalli & Freuder36August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder37August 7, 2001 l This will generate an object in which fields are domains or metavariables, and methods are constraints. (This is the input to the Diagnosis module) Example of CSP Modeling - Test Cases as Objects - OneWayOutA OneWayOutB Hello OneWayOutA_Status OneWayOutB_Status OneWayOutA_Source_OneWayOutB_Source Mandatory Hello OneWayOutA_Type OneWayOutB_Type OneWayOutA_Time_OneWayOutB_Time

IAAI-01: Sqalli & Freuder38August 7, 2001 Test Suite Menu (GUI)

IAAI-01: Sqalli & Freuder39August 7, 2001 Test Suite Directory Hierarchy testsuite mpoa pnniroutlane V4301H_001 adiop adiopx V4301H_002V4301H_003 l Test cases are stored in the appropriate directory. l Menu of test cases is generated dynamically from this directory structure.

IAAI-01: Sqalli & Freuder40August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder41August 7, 2001 Protocol Specification Monitored Observations Result Decoded Observations Test Suites Test #1 Test #2 … CSP Models CSP #1 CSP #2 … CBR + Expert Result Report OK NoYes ? Store Next Step Debugging Modeling Diagnosis Decoding ADIOP – The big picture (Four Modules) Update

IAAI-01: Sqalli & Freuder42August 7, 2001 ADIOP (Automated Diagnosis of InterOperability Problems) Test Suite Builder Test Generation and Implementation Test Debugger Diagnoser Test Execution (Algorithms) Menus and Reports Decoder s BuilderDecoder Usage Decoder Search (hard explanation) Inference (better explanation) Automate Menus Creation Reports Generation CBR + User Interaction Correct/Update CSP Model New CSP Model Protocol Specification Frame Format (Defined language) Java file (Frame decoder) Java Class (Frame decoder) CSP Model Notes: Implemented To be implemented Not implemented Existent Documents Test Description from IOP test suite Natural Language Processing Graphical Representation CSP Model Java file Java Class (used by diagnoser) Packet PnniR MPOA HelloDBS Note: These classes are used by Decoder, Diagnoser, and Test Builder. Hierarchy of Classes

IAAI-01: Sqalli & Freuder43August 7, 2001 Application of CSP Modeling - ADIOP Diagnosis l The test cases built using the ADIOP’s modeling component are dynamically accessible through a menu. l Motivations of Diagnosis of IOP Problems: save time, reduce repetitive testing, store and reuse knowledge, automate reports generation. l Advantages of using CSP for modeling and diagnosis: take advantage of existing CSP algorithms. l Algorithms for Diagnosis: Search and Inference. l Generate an accurate and human-like explanation. l Evaluation and summary numbers: 50 (73% ) test cases out of 69 built produced a meaningful explanation.

IAAI-01: Sqalli & Freuder44August 7, 2001 Outline l Example l Domain of Application l Motivations and Contributions l CSP Modeling l Modeling with Objects l Modeling Interface and Language l Test Cases as Objects l Application & Evaluation l Summary

IAAI-01: Sqalli & Freuder45August 7, 2001 Summary l CSP Model building: Simple, declarative, transparent, GUI-based, and complete modeling language for stating and generating IOP tests. l OO provides a more natural, concise, scalable and reusable framework for model statement of IOP tests (+ Inheritance, Encapsulation) l Combination of CSP and OO: metavariables and metavalues to represent packets and generate test cases as objects. l Scalability: More protocols can be made available for modeling by adding the corresponding decoders. l Modeling Interface: dynamic loading of protocols and packets being used. l Modeling Language: – Generic across protocols: Independent of the protocol being used. – Constraint expressiveness: unary and binary constraints. l Many-models architecture: test suites are used instead of protocol specifications. Each test case is represented by a CSP model. l Application & Evaluation: Diagnosis and Explanation of IOP problems.

Constraint-Based Modeling of InterOperability Problems Using an Object-Oriented Approach Discussion Mohammed H. Sqalli Eugene C. Freuder University of New Hampshire