1. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 1 Fast Functional Test Generation Using an SDL Model Robert L. Probert Alan W. Williams School of Information Technology and Engineering University of Ottawa Ottawa Canada {bob,awilliam}@site.uottawa.ca www.site.uottawa.ca/~awilliam

2. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 2 Overview Goals of the project Project Steps Functional Testing Development of: requirements models test models abstract test cases Productivity Results Issues identified Conclusions and further work

3. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 3 Goals of the Project A local company had obtained equipment with a TTCN interface for automated testing of their telephone switching systems. Questions to answer: Can TTCN be used for functional testing? Are the tools for these methods “industrial strength”? How much effort does this process take? Can MSCs, SDL, be used to speed up the process of test generation, and decrease the time to market? For this project, Telelogic Tau (version 3.3) was used as the tool set.

4. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 4 Reduced Time-to-Market

5. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 5 Project Steps: Iterative and Incremental Process Develop MSC Use Cases Create/Revise SDL model Produce MSCs by executing SDL model, and compare with requirements MSCs Iterations for errors, added use cases Create TTCN abstract test cases by executing SDL model *

6. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 6 Functional versus Conformance Testing Focus of conformance testing: Exchange of protocol messages: both for sequence and for content. Goal is to verify system interoperability, by conforming to a standard. Focus of functional testing: User’s view of the system, with respect to actions and reactions. More indirect than conformance testing. Needed to ensure that customer requirements are met.

7. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 7 Comparison of Test Actions and Reactions Test actions: Functional testing: go off hook on a phone set. Conformance testing: send protocol message to the telephone switch as a result of going off hook. Dealing with a response to a test action : Conformance testing: check exact sequence and data format of messages that result from going off hook. Functional testing: check that a set of responses occurs within a suitable time interval: dial tone; lamps, displays updated. Persistent events: a dial tone comes on, and stays on.

8. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 8 Test Architecture Functional test view: test environment provides a set of phone emulators that do user- oriented actions according to a script. “Adaption layer” provides set of valid TTCN messages to user (e.g. off hook). TTCN message is converted to actual protocol message. auto test system user interface (TTCN) protocol interface switch under test line1line n... protocol “adaption layer” TTCN Test Cases

9. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 9 SDL - TTCN - Test Environment correspondence SDL message names are automatically extracted for use in the TTCN test cases. To match the test environment, the SDL model should use the names provided by the test environment “adaption layer” of the specific functionality to be tested. Signal a, b, c; SDL model ! a ? b ! c TTCN test suiteAdaption Layer a b c...

10. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 10 test tool command to detect switch response test tool command to cause switch action Mapping the MSC model to the test environment phone1user1phone2user2switch test tool phone command test tool phone response

11. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 11 Creating the Message Sequence Charts The MSC has only two actors: the environment (user) and the system. This models the interaction between the user(s) and the system. For testing, this models the exchange of messages between the test environment and the system under test. EnvironmentSystem userOffhook [1] userKey [1,2] userDialToneOn [1] userDialToneOff [1] userKeyTone [1,2] userKeyTone [1,2] userAlertingOn [2] userKey [1,2] userRingbackOn [1] userOffhook [2]

12. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 12 High-Yield Scenarios Yield a rich harvest of bugs in requirements or design UserRequirement + = Designer Tester 110V/220V ? 2-pin or 3-pin power plug ? Burnt fingers and electric shock when getting the toast. Does not accept bagels ? Toast only every 20 min ? Where’s the handler ? How to Control the “Toastness” ? Tester must “think like a user” and influence designer to do likewise

13. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 13 Creating the SDL test model Exceptions: will produce high-yield test cases Normal scenarios: will produce low-yield test cases

14. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 14 SDL Model considerations The model is not intended to replicate the entire system behaviour of the switch and phone sets. The model should reflect the test architecture. The model should include: the functionality to be tested. additional behaviour extending to points of control and observation, for test tool commands. enough internal behaviour to execute model. The model does not need to include: internal behaviour of the system that cannot be observed by test equipment, and is not part of the protocol.

15. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 15 Simulated Execution of the SDL model User can interact with the model, and view execution graphically. MSCs that record SDL model execution were produced. Execution MSCs were compared and verified to original requirements MSCs to check that the SDL model is correct. Coverage report was produced Created abstract test cases in Tree and Tabular Combined Notation (TTCN), generated directly from the SDL model. First iteration: generate complete test cases. Second iteration: (manually) identify test steps, and regenerate modular test cases.

16. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 16 Comparing Requirements and Execution MSCs States added Digit timer added Unexpired timer System structure added

17. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 17 Coverage-based Test Generation transient state inputs

18. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 18 TTCN test case

19. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 19 Finding a bug with the SDT validator Bug: ringing should have been turned off when call back activated Unspecified message reception: answering the “ringing” phone

20. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 20 Productivity Results People working on SDL model: One company employee with product knowledge, who had recently taken SDL and Tau tool training. Two graduate students, one with experience in telephony, SDL, and the Tau tool. One professor, who was consulted for reviews. Time to produce basic call model: Initial version took about a week of intensive work. After several restructurings, call model stabilized after four weeks. All work (SDL model with 5 features, initial test cases, verified feature MSCs) took 3 months.

21. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 21 Productivity Chart

22. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 22 Issues for functional testing: Event order Forced ordering of events on SDL transitions, TTCN test cases. Example: Update of screen, lamps on a telephone. Order is irrelevant for user. Puts unnecessary constraints on designers. Creates unnecessary alternatives in test case. Potential remedy: SDL “unordered” transitions. Allow TTCN to detect an interleaved set of responses: ! off hook ? screen | main lamp | line lamp | dial tone (state) (input) lamp on (state) screen update

23. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 23 Issues for functional testing: Durations Example situation: Test case generated with the intention of expiring the off hook timer: phone1 ! off hook phone1 ? dialToneOn phone1 ? dialToneOff phone1 ? announcementOn The duration of the off hook timer is specified in the SDL model. The duration does not get carried into the test case. TTCN does have timer facilities, so this is not a language deficiency. If the announcement appears too early from user’s view point, this test case will still pass.

24. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 24 Conclusions and Further Work Advantages of the process: User-oriented: testing is based on what the customer wants. Simulated execution and validation can find bugs earlier in the process. TTCN test cases are generated faster (15 minutes/test case) than with manual approach. Well-defined level of test coverage. Graphical documentation, in the form of MSCs and SDLs. A new project is underway that will: Verify the scalability of the approach. Bring design groups into the process. Continue to refine the metrics for productivity measurement.

25. UNIVERSITÉ D’OTTAWA UNIVERSITY OF OTTAWA IWTCS ‘99 - 25 Early Detection of Design Faults SDL Design Model MSCs HMSCs MSCs TTCN Autolink, TTCNlink *design regression tests Validator Navigator *High-Yield Iteration Design Information Designers UCMs, LOTOS traces, LOTOS semantics * MSC Editor