1. A Model Driven Tool for Automated System Level Testing of Middleware Turker Keskinpala Abhishek Dubey Steve Nordstrom Ted Bapty Sandeep Neema Institute for Software Integrated Systems Vanderbilt University Nashville, TN USA

2. STV'06 Workshop March 30, 2006 Outline Motivation and Challenges Model Driven Approach Tool Structure Test Series Definition Modeling Language Example Test Series Definition Conclusions and Ongoing Work

3. STV'06 Workshop March 30, 2006 Motivation and Challenges Many middleware framework configurations are generally represented by XML Configuring functional operation of components Component deployment in the cluster Frameworks are complicated and need to be tested Application topologies Parameter variations Need for many test configurations

4. STV'06 Workshop March 30, 2006 Motivation and Challenges Hand writing test configurations Cumbersome Error prone Inefficient Big configuration space of control and deployment parameters of applications Response to changes in hardware resources or other criteria Difficult to scale up manually written XML configurations Usual outcome: Under tested software

5. STV'06 Workshop March 30, 2006 Goals Testing large sets of system configurations Simple powerful specifications Tool to take specifications to instances Configure series of tests from the specifications Help finding corner cases Potential functional errors Potential performance problems Unexpected system interactions

6. STV'06 Workshop March 30, 2006 Model Driven Approach Domain Specific Modeling Use domain specific models to encapsulate relevant parts of the system properties Provide multiple view points Decouple types, resources, dataflow and deployment aspects Generative modeling Vary range of behavioral and structural system parameters System generation Automatically generate configurations from higher level models

7. STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models

8. STV'06 Workshop March 30, 2006 Test Series Definition Define tests using a high level modeling language called Test Series Definition Modeling Language (TSDML) Define application types Form a template test case from those types Define dataflow association between applications by connection rules Define deployment associations for applications Define the variance on the system size and structure Define variance on the application parameters Define available resources Automatically create series of test configuration models by replicating and adapting the template test case

9. STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models Generator (1-*) Test Configuration Models

10. STV'06 Workshop March 30, 2006 Test Configuration Models Form the configuration layer Each test configuration model capture: Specific dataflow scenario Specific deployment scenario Configuration Schema Represents single system representation to be tested Automatically generate single configuration file per configuration model Test driver scripts can also be generated

11. STV'06 Workshop March 30, 2006 Tool Structure Test Series Definition Models Generator (1-*) Test Configuration Models XML Configs Test Scripts Generator (1) Application Runtime ( System Under Test ) Test Execution & Validation Engine Applications

12. STV'06 Workshop March 30, 2006 System Under Test XDAQ Framework High performance distributed middleware framework developed at CERN Will be used for Compact Muon Selenoid (CMS) experiment Run-time support for clustered applications Predefined functions to configure and control the applications and their parameters XML is used for configuration of XDAQ and its applications

13. STV'06 Workshop March 30, 2006 Test Series Definition Modeling Language (TSDML) Parameterized model Varying system structure (size) Iterators: e.g. i,j,k = {start, step, stop} Replicators: e.g. r1=n*i, r2=m*j Connectors: Boolean function of iterator, and replicator If src=(i,r1) and dst=(j,r2) C=f(src,dst) C=2*src==3*dst i r1 App1 j App2 r2 Connector

14. STV'06 Workshop March 30, 2006 Iterator-Replicator Example Assume i = 1:1:2, j= 1:1:4, r1 = 2, r2 = 4 r2 i r1 App1 j App2 Test Caseij#App1#App2 11124 21228 313212 414216 52144 62248 723412 824416

15. STV'06 Workshop March 30, 2006 Connector Example i = 1:1:2, j= 1:1:4, r1 = 2, r2 = 3 CR = src=floor((dst-1)/(j*r2/i*r1)) + 1 i r1 App1 i App2 r2 Test Caseij#App1#App2 11124 21228 313212 414216 52144 62248 723412 824416 Connector

16. STV'06 Workshop March 30, 2006 Connector Example (ctd…) App1 App2 App1 App2 (1) (2) (1) dstdst srcsrc Rule: src=floor((dst-1)/(j*r2/i*r1)) + 1 (2) (3) (4) 1=floor(0/2) + 1 => TRUE1=floor(1/2) + 1 => TRUE1=floor(2/2) + 1 => FALSE1=floor(3/2) + 1 => FALSE

17. STV'06 Workshop March 30, 2006 Test Series Definition Modeling Language (TSDML) Varying application parameters with parameter generator Has a function attribute Function of iterator Function attribute may be: Internal function e.g. result = src * 256 where src = i Matlab script e.g. lut = [10 20 55 88 400 800 1025 4490 7342]; result = lut[src]; Internal lookup table

18. STV'06 Workshop March 30, 2006 Example TSDML

19. STV'06 Workshop March 30, 2006 Example TSDML 2*i 1:1:8 1:1:1 2*i 16*i 1*i

20. STV'06 Workshop March 30, 2006 Conclusion Domain specific modeling for capturing relevant system parts Vary system structure and application parameters from high level model Generate series of test configurations from a single high level model Generate the configuration representation Mitigate the complexity of creating many system test configurations

21. STV'06 Workshop March 30, 2006 Ongoing Work Add another layer on top of TSDML System model Application behavior Target framework Analyze the models Assess important parts of the system to exercise and test

22. STV'06 Workshop March 30, 2006 Thank you!

23. A Model Driven Tool for Automated System Level Testing of Middleware Turker Keskinpala Abhishek Dubey Steve Nordstrom Ted Bapty Sandeep Neema Institute for Software Integrated Systems Vanderbilt University Nashville, TN USA