A Model Driven Tool for Automated System Level Testing of Middleware Turker Keskinpala Abhishek Dubey Steve Nordstrom Ted Bapty Sandeep Neema Institute for Software Integrated Systems Vanderbilt University Nashville, TN USA

STV'06 Workshop March 30, 2006 Outline Motivation and Challenges Model Driven Approach Tool Structure Test Series Definition Modeling Language Example Test Series Definition Conclusions and Ongoing Work

STV'06 Workshop March 30, 2006 Motivation and Challenges Many middleware framework configurations are generally represented by XML Configuring functional operation of components Component deployment in the cluster Frameworks are complicated and need to be tested Application topologies Parameter variations Need for many test configurations

STV'06 Workshop March 30, 2006 Motivation and Challenges Hand writing test configurations Cumbersome Error prone Inefficient Big configuration space of control and deployment parameters of applications Response to changes in hardware resources or other criteria Difficult to scale up manually written XML configurations Usual outcome: Under tested software

STV'06 Workshop March 30, 2006 Goals Testing large sets of system configurations Simple powerful specifications Tool to take specifications to instances Configure series of tests from the specifications Help finding corner cases Potential functional errors Potential performance problems Unexpected system interactions

STV'06 Workshop March 30, 2006 Model Driven Approach Domain Specific Modeling Use domain specific models to encapsulate relevant parts of the system properties Provide multiple view points Decouple types, resources, dataflow and deployment aspects Generative modeling Vary range of behavioral and structural system parameters System generation Automatically generate configurations from higher level models

STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models

STV'06 Workshop March 30, 2006 Test Series Definition Define tests using a high level modeling language called Test Series Definition Modeling Language (TSDML) Define application types Form a template test case from those types Define dataflow association between applications by connection rules Define deployment associations for applications Define the variance on the system size and structure Define variance on the application parameters Define available resources Automatically create series of test configuration models by replicating and adapting the template test case

STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models Generator (1-*) Test Configuration Models

STV'06 Workshop March 30, 2006 Test Configuration Models Form the configuration layer Each test configuration model capture: Specific dataflow scenario Specific deployment scenario Configuration Schema Represents single system representation to be tested Automatically generate single configuration file per configuration model Test driver scripts can also be generated

STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models Generator (1-*) Test Configuration Models XML Configs Test Scripts Generator (1) Application Runtime ( System Under Test ) Test Execution & Validation Engine Applications

STV'06 Workshop March 30, 2006 System Under Test XDAQ Framework High performance distributed middleware framework developed at CERN Will be used for Compact Muon Selenoid (CMS) experiment Run-time support for clustered applications Predefined functions to configure and control the applications and their parameters XML is used for configuration of XDAQ and its applications

STV'06 Workshop March 30, 2006 Test Series Definition Modeling Language (TSDML) Parameterized model Varying system structure (size) Iterators: e.g. i,j,k = {start, step, stop} Replicators: e.g. r1=n*i, r2=m*j Connectors: Boolean function of iterator, and replicator If src=(i,r1) and dst=(j,r2) C=f(src,dst) C=2*src==3*dst i r1 App1 j App2 r2 Connector

STV'06 Workshop March 30, 2006 Iterator-Replicator Example Assume i = 1:1:2, j= 1:1:4, r1 = 2, r2 = 4 r2 i r1 App1 j App2 Test Caseij#App1#App

STV'06 Workshop March 30, 2006 Connector Example i = 1:1:2, j= 1:1:4, r1 = 2, r2 = 3 CR = src=floor((dst-1)/(j*r2/i*r1)) + 1 i r1 App1 i App2 r2 Test Caseij#App1#App Connector

STV'06 Workshop March 30, 2006 Connector Example (ctd…) App1 App2 App1 App2 (1) (2) (1) dstdst srcsrc Rule: src=floor((dst-1)/(j*r2/i*r1)) + 1 (2) (3) (4) 1=floor(0/2) + 1 => TRUE1=floor(1/2) + 1 => TRUE1=floor(2/2) + 1 => FALSE1=floor(3/2) + 1 => FALSE

STV'06 Workshop March 30, 2006 Test Series Definition Modeling Language (TSDML) Varying application parameters with parameter generator Has a function attribute Function of iterator Function attribute may be: Internal function e.g. result = src * 256 where src = i Matlab script e.g. lut = [ ]; result = lut[src]; Internal lookup table

STV'06 Workshop March 30, 2006 Example TSDML

STV'06 Workshop March 30, 2006 Example TSDML 2*i 1:1:8 1:1:1 2*i 16*i 1*i

STV'06 Workshop March 30, 2006 Conclusion Domain specific modeling for capturing relevant system parts Vary system structure and application parameters from high level model Generate series of test configurations from a single high level model Generate the configuration representation Mitigate the complexity of creating many system test configurations

STV'06 Workshop March 30, 2006 Ongoing Work Add another layer on top of TSDML System model Application behavior Target framework Analyze the models Assess important parts of the system to exercise and test

STV'06 Workshop March 30, 2006 Thank you!

A Model Driven Tool for Automated System Level Testing of Middleware Turker Keskinpala Abhishek Dubey Steve Nordstrom Ted Bapty Sandeep Neema Institute for Software Integrated Systems Vanderbilt University Nashville, TN USA