Rational Israel Users Conference Automated Model Driven Software Testing Rational Users Conference Tel Aviv June Alan Hartman IBM Israel – Haifa Research Laboratory

Rational Israel Users Conference Agenda AGEDIS Project Overview Architecture Methodology Interfaces Tools Future Plans

Rational Israel Users Conference AGEDIS Overview Automated model-based test Generation and Execution for DIStributed systems Methodology and tools for model-based testing Open interfaces Mixture of academic and industrial partners Three phase timetable of experiment and development November February 2004 Partially funded by the European Commission

Rational Israel Users Conference Consortium Partners IBM Haifa Research Lab Oxford University VERIMAG/IRISA Imbus France Telecom IBM UK Intrasoft International

Rational Israel Users Conference Methodology & Architecture UML based component testing Compliant with RUP High degree of automation No silver bullet

Rational Israel Users Conference Automated Testing Methodology

Rational Israel Users Conference Automated Testing Architecture GUI & Productivity Aids Model Generation Directives Execution Directives Compiler Intermediate Format Simulator Generator Abstract Test Suite Suite Execution Trace Analyzers Visualizer Editor Execution

Rational Israel Users Conference Benefits Starting from specification –Involves testers early in the development process –Teams testers with developers –Forces testability into product design Building behavioural model and test interface –Finds design and specification bugs - before code exists –The model is the test plan - and is easily maintained Automated test suite generation –Coverage is guaranteed - increases testing thoroughness –Matches coverage goals to testing budget –Zero test suite maintenance costs Automated test suite execution –Finds code and interface bugs –Includes a framework for the testing of distributed applications –Reduces test execution costs

Rational Israel Users Conference Interfaces UML Profile for AGEDIS Test Generation Directives Test Execution Directives IF Model Execution Interface Abstract Test Suite Suite Execution Trace UML TGD TED IF ATS SET

Rational Israel Users Conference User Modeling Interface The AGEDIS Modeling Language is a profile for UML 1.4: –UML Class diagrams - structure –UML Object diagrams - snapshots –UML State diagrams – behaviour & test purposes Annotated with an action language – IF

Rational Israel Users Conference Test Generation Directives Test Purposes – stereotyped system level state diagrams Coverage Directives – stereotyped system level state diagrams Coverage without tears – 4 levels – parameters to the test generation algorithm

Rational Israel Users Conference Test Execution Directives XML Schema –Model translation from model abstractions to SUT concrete implementations –Test architecture Host addresses Delays, timeouts Prologues, epilogues

Rational Israel Users Conference IF - Model Execution Interface IF Encoding of the model –Classes –Objects –State machines

Rational Israel Users Conference Test Suite and Trace Interface XML schema – for test execution and tracing Model description –classes : constants, types, control & observable signatures –a special class is defined for the tester –object identities Test Suite - set of test cases Test Trace – record of executed test cases

Rational Israel Users Conference Test Case and Test Trace Interaction graphs - between the tester and the SUT Associated verdicts (Pass, Fail, Inconclusive) Alternate paths for non-determinism in test cases Expected results in test suite Observed results and verdicts in trace

Rational Israel Users Conference Abstract Test Suite

Rational Israel Users Conference Tools User Interface Modeler & Model Compiler Model Simulator Test Generator Test Execution Engine Test Suite/Trace Viewer/Editor Feedback & Analysis Bug Reporter Report Generator UI & RG C S G FA VE EE

Rational Israel Users Conference GUI

ModelingTool & Compiler Modeling Tool & Compiler Objecteering UML modeling tool Tool profile to convert to XML General purpose XML to IF compiler –Written in Java, with XMI in mind as a future input format

Rational Israel Users Conference Model Simulator

Rational Israel Users Conference Test Generator Based on GOTCHA and TGV GOTCHA –uses Murphi specification language –explicit traversal of state space –extensive coverage criteria TGV –language independent simulator –focus on distributed applications –explicit test purposes as sequences of interactions

Rational Israel Users Conference Test Execution Engine –Input: Abstract Test Suite and Test Execution Directives –Output: Suite Execution Trace –Multiple platform (e.g. Linux, Windows) support –Multiple external interface (Java,C++,C) support –Test distribution + centralized logging –Interactive test case execution –GUI to view execution progress, inputs, and outputs

Rational Israel Users Conference Execution Engine Architecture N e t w o r k Test Suite Driver I/O Host Manager Process Manager 11 on each host1..N on each host... Object

Rational Israel Users Conference Feedback Tools Coverage analysis –Detect uncovered areas of the model in either test suite or test trace –Create test purposes to reach them –Invokes a functional coverage tool from Defect analysis –Clustering of defects –Feature extraction from clusters –Create test purposes to reproduce the bug

Rational Israel Users Conference AGEDIS Future Plans

Rational Israel Users Conference Future Plans Exploitation Activity –Available under license for non-commercial use –Available as a service Incorporation in wider Model Driven SE Effort –Integrate methodology with RUP –Migrate to Eclipse, XDE –Focus on business integration domain

Rational Israel Users Conference ModelWare New European Project Model driven software engineering Eclipse based tools for UML modeling, and the full MDA lifecycle –Model driven architecture –Platform independent and platform specific models –Model simulation –Model transformation

Rational Israel Users Conference Thanks to: A. Ramfos., S. Liapis, N. Giannelos, A. Hondouridakis, M. Sardis, K. Bechrakis, V. Akousi-Krivki K. Dussa-Zieger, J. Trost, B. Nossem, T. Linz, B. Mattern, J. Hofer, H. Raessler, T. Rossner, I. Craggs, I. Griffiths Y.-M. Quemener, D. Vincent, T. Heuillard, N. Moteau A.Bertolino, A. Wills, S. Reid K. Nagin, O. Edelstein, A. Kirshin, S. Olvovsky, M. Berg, L. Raskin, T. Shiran, C. Sacharen, M. Barshay, D. Neimer L. Mounier, M. Bozga, Y. Lakhnech T. Jeron, E. Demairy, V. Tschaen J. Davies, A. Cavarra, C. Crichton, J. Woodcock, M. Field