1. JRA2 – TEST MANAGEMENT TOOLS
WP6
JRA2 – TEST MANAGEMENT TOOLS
ETICS2 first Review
Eva Takacs
4D SOFT
Bruxelles, 3 April 2009

2. Contents Goals Case studies Major Achievements Challenges
Metrics and Statistics
Conclusions
WPX WP name
Bruxelles, 3 April

3. Goals
WPX WP name
Bruxelles, 3 April 3 3

4. Major goals of ETICS test services
ETICS aims to provide a distributed, complex build/test framework
where the functionalities of the framework and the testing tools are treated as separate entities
and which follows the actual trend in software quality assurance of providing “scriptless tools that can be used by nondevelopers” (Gartner Group)
and which promotes testing by offering really user friendly, hands-on services
contributes to standards
WPX WP name
Bruxelles, 3 April

5. Test services in ETICS1
Having these principles in mind in ETICS1 distributed testing environment consisted of four major elements:
Distributed multinode testing mechanism
Test and Metrics Plugin framework – as part of the ETICS client (command line interface)
Testing tools and Tools for Metrics collection as plugins of the Test and Metrics Plugin framework
Metronome Build&Test system – as multi-platform execution framework
WPX WP name
Bruxelles, 3 April

6. Testing and validating with ETICS
ETICS testing functionality is provided by plugins, which are Python wrappers around external test tools.
These testing tools are used by ETICS to analyse particular aspects of the code or execute test suites.
Example of plugins are: Sloccount, PyUnit, Junit, Pycoverage, JEmma, Findbugs, CCCC, CKJM, IPv6, WSI
ETICS CLI contains the Test and Metrics Plugin Framework, a well-defined interface, that allows the ETICS Client to activate the plugins during an etics-build process
Policies called profiles (for example: junit, wsi) activates the plugins during the build
WPX WP name
Bruxelles, 3 April

7. Plugin implementation
Plugins can be implemented by user by developing a Python wrapper around the test tool following the rules defined in the interface provided by Test And Metrics Plugin Framework
JRA2 extention to plugin implementation
Plugin Generator Service – by providing templates and guidance semi-automates the implementation.
Fulfils the objective of
separating the test tools from the ETICS framework
providing „scriptless” tools
user friendly, easy to use services
WPX WP name
Bruxelles, 3 April

8. Plugin deployment, usage
Plugins could be made available by the ETICS system administrator
Mandatory plugins will be downloaded in the ETICS CLI
Optional plugins could be downloaded by users on demand in their local environment
Individual plugin implementations are available in the respective local environment
JRA2 extention to plugin deployment, usage
Test Tool Registration System – web application for general purpose plugin registration, management, serves as an Information System on test tools/plugins
Fulfils the objective of
user friendly testing services, high level front-end application for the plugin management
WPX WP name
Bruxelles, 3 April

9. Plugins and Metrics JRA2 extension to metrics format
The plugins typically collect information and publish data in the form of Metrics
Metrics are published with each build/test reports and are also stored individually in the ETICS Repository
The collected metrics values can be used further to analyse a project status or behaviour (trend analysis) and for the certification (A-QCM)
JRA2 extension to metrics format
Standardised xsd formats for the summary and detail metrics representation
Fulfils the objective of
contribution to standards, improved testing services by providing better reports
WPX WP name
Bruxelles, 3 April

10. Multinode test mechanism
The ability to setup scenarios where multiple services are automatically deployed in multiple nodes. The user has to syncronise the process and to control the order of operations using the information service (etics-get, etics-set) provided by ETICS Command Line client.
JRA2 extention to multinode mechanism
Workflow Designer – high level front-end web application able to orchestrate complex, multinode test scenarious
Fulfils the objectives of
Providing „scriptless” tools for test design
User friendly, high level test services for distributed systems
WPX WP name
Bruxelles, 3 April

11. Main objectives of the JRA2 activity Summary
Develop a registration system by which the ETICS users can register their tools to be integrated into the ETICS testing infrastructure
Provide an advanced interface (workflow designer) for the users to construct their complex multinode deployments scenarios and testing process in a convenient way
Develop the necessary test interface standards for proper collaboration of testing tools and the proper collaboration with the job submission systems and also presentation of the results in standard formats
Provide the mean for users to visualise their test results
WPX WP name
Bruxelles, 3 April

12. Case studies
WPX WP name
Bruxelles, 3 April 12 12

13. Case study 1 of the ETICS Test System CKJM’s plugins lifecycle
Description of the case study
In order to perform the certification process inside NA2 work package the Evaluation Model of the A-QCM (Automated – Quality Certification Model) requires calculation of Structural Testing/Static Analysis metrics.
Among them, Chidamber and Kemerer object-oriented metrics fall in this category.
WPX WP name
Bruxelles, 3 April

14. Case study 1 of the ETICS Test System CKJM’s plugins lifecycle
Metrics proposed by Chidamber and Kemerer are:
WMC: Weighted methods per class
DIT: Depth of Inheritance Tree
NOC: Number of Children
CBO: Coupling between object classes
RFC: Response for a Class
LCOM: Lack of cohesion in methods
Ca: Afferent couplings
NPM: Number of public methods
The ckjm open source tool calculates these metrics for Java. (
Bruxelles, 3 April

15. Case study 1 of the ETICS Test System CKJM’s plugins lifecycle
In order to have the required metrics available for A-QCM we have to
Register ckjm as a plugin in Test Tool Registration system.
As a result of this operation the JCkjmPlugin is available for ETICS users
Create the project configuration structure for the program under analysis or chosing an existing one according to requirements.
Setting the profiles for the required ETICS project element
Execute the build for the required configuration.
As a result of the build the metrics will be available in the ETICS repository for further analysis for the A-QCM service.
WPX WP name
Bruxelles, 3 April

16. Case study 2 - Interoperability Scenario Multinode test design
Case study description
The case study is interoperability in job submission among different middleware implementations
The test involve ARC, gLite and UNICORE and tests job submission with any of the clients towards any of the servers.
The packages are taken from their official distribution repositories
The interoperability tests and the respective ARC, UNICORE and gLite clients are expected to be in a same node and
These clients call the external servers/services maintained by their system administrators
WPX WP name
Bruxelles, 3 April

17. Case study 2 - Interoperability Scenario Multinode test design
Moving Interoperability Scenario to multinode
The goal of this use case is moving the interoperability test to multinode to be able to deploy a complete test infrastructure without having to rely on external services.
In the existing one node scenario where the clients and the tests are in one node, the UNICORE server has to be available in Juelich, the ARC server in Kosice and the CREAM server in Padova.
WPX WP name
Bruxelles, 3 April

18. Case study 2 - Interoperability Scenario Multinode test design
Description of the ETICS process - steps
Create the project configuration structure
Design the workflow of the deployment process
Create an abstract Workflow
Create a concrete multinode workflow
Browse the project configuration data
Set the flow connectors (data that communicates between nodes, for example, hostname)
Set the phases of the deployment
Add the necessary deployment scripts for different phases
Generate the output xml file – it serves as input for the job submission service
WPX WP name
Bruxelles, 3 April

19. Major Achievements
WPX WP name
Bruxelles, 3 April 19 19

20. ETICS Test System
ETICS Test System is the new portlet containing the frontend applications for testing services and is accessed via a new tab of the current ETICS main portal.
The main components (portlets and the associating web services of this new tab) of the ETICS Test System are the following:
Test Tool Registration System – portlet and the underlying services for supporting test tool registration adding/removing test tools, plugins, licensed products to and from the ETICS framework
Plugin Generation Service - service for plugin template generation
Workflow Designer - workflow tool and the associated services for complex test design

21. ETICS Test System Architecture
WPX WP name
Bruxelles, 3 April

22. Test Tool Registration System
The motivation that lies behind the test tools registration is to augment the current Test and Metrics Plugin Framework
by applying a generic registration/installation procedure for testing tool providers, supporting the plugin implementation by providing wizards and templates for the users, on demand download of the plugins
by providing information on the availability of plugins
and to enhance the variety of testing tools inside ETICS framework.
The registration can be conducted by means of various operations depending on the functionality of test tool, licence, property issues.
As a result of the registration operation, the testing tools are deployed on well defined locations and they are available for ETICS testing processes.
WPX WP name
Bruxelles, 3 April

23. Plugin Generation Service
The aim of this service is to augment the current plugins implementation practice by generating a plugin template.
Plugin Generator Service creates a plugin (python wrapper) template around the registered tools in the Test Registration System during the registration process.
The generated template is the ‘standard’ implementation of a ‘standard test tool’. This template can be enhanced by the users’ according to their needs.
WPX WP name
Bruxelles, 3 April

24. Example case study1: CKJM Registration Process Creates JCkjmPlugin
WPX WP name
Bruxelles, 3 April

25. Example case study 1- CKJM registration process Uploading the JCkjmPlugin
WPX WP name
Bruxelles, 3 April

26. Example case study 1- CKJM registration process JCkjmPlugin in production state - available for users
WPX WP name
Bruxelles, 3 April

27. Standard interfaces
ETICS Test System provides standard formats for the reports’ format of static and dynamic analyser tools integrated as plugins.
The proposed formats are based on the format provided by the NIST SAMATE (Software Assurance Metrics And Tool Evaluation) project for the static analysis tools.
Why we suggest using standard interfaces for report generation?
to speed adoption of the tools by objectively demonstrating their use on real software
the value of a tool depends on how effectively developers can use the tgenerated reports to review and address issues
improved readability of the generated reports could result in more identified bugs, errors
having standardised reports leads to standardised processes
WPX WP name
Bruxelles, 3 April

28. Workflow Designer
Two scenarios to define complex tests have been identified in ETICS:
multinode test design, when the required services are deployed on different mashines
(the message passing between different phases of service deployment on different nodes is crucial)
test process design, when the focus is on the orchestration of different testing tools
(the dependencies and the necessary exchange of reports are crucial among registered and deployed test tools/plugins)
WPX WP name
Bruxelles, 3 April

29. Main features of Workflow Designer
Supported ETICS workflows
Multinode deployment tests
Complex test process design
Design concepts
Abstract and Concrete Workflows
Workflow repository
Workflow status management
Implementation is based on standard and widespread technologies
JSR168 compliant portlets for compatibility
WS-PGRADE/GridSphere framework for plugable system
WebServices/SOA concept for robust and scalable architecture
XML based workflow description for interfacing
WPX WP name
Bruxelles, 3 April

30. Architecture of Workflow Designer
WPX WP name
Bruxelles, 3 April

31. Workflow Designer - Test process
Goal: Orchestration of registered plugins
Support for
Cyclic execution of plugins and workflows
Conditional execution
Provides interface
to repository of registered plugins
to workflow interpreter
WPX WP name
Bruxelles, 3 April

32. Workflow Designer - Multinode deployment test
Goal: Orchestration of deployment phases of service sets
Support for
Definition of deployment phases for configurations
Data dependencies based on key-pairs
Provides interface
to configuration information
to workflow interpreter
WPX WP name
Bruxelles, 3 April

33. Example case study 2 Interoperability Workflow
WPX WP name
Bruxelles, 3 April

34. Example case study 2 -Output of the Workflow design Input for job submission service
<service name="string" module="string">   <version name="string" configuration="string">       <platformMapping platform="string">           <phase name="string" script="string">               <logEntry type="pre/post"/>               <pair key="string"                     type="requires/provides"                     scope="phase/service/node/global"                     mandatory="boolean"                     resolutionTime="Runtime/Design"                     valueResolution="string"                     defaultValue="string"/>               <runsAfter phase="string"/>           </phase>       </platformMapping>   </version> </service>
WPX WP name
Bruxelles, 3 April

35. Achievements of JRA2 Summary
The main achievement of JRA2 in ETICS2 is the extention of the existing distributed testing mechanism with advanced functionalities
by supplying a Test Tool Registration and Plugin Generation Service mechanism allowing ETICS users to register their tools equipped with the necessary Information System
by supplying standard report formats
by supplying a high level Workflow Designer able to orchestrate distributed deployment and testing scenarious with the goal of reducing the complexity of designing and implementing these complex tests for applications
WPX WP name
Bruxelles, 3 April

36. Metrics and Statistics
WPX WP name
Bruxelles, 3 April 36 36

37. Metrics and Statistics
Deliverables
DJRA2.1 – Use-cases Specification
DJRA2.2 – Test interface standards
DJRA2.3 – Test tool registration system (prototype)
DJRA2.4 – User interface and Test Execution Generator (prototype)
Statistics as defined on the project QA Plan
???? To be added
WPX WP name
Bruxelles, 3 April

38. Challenges
WPX WP name
Bruxelles, 3 April 38 38

39. Challenges Technical challenges Organisational challenges
Applying high-level workflows for testing scenarious
Integration of different challenging technologies, GWT, GridSphere, web services
Organisational challenges
New people in ETICS2 – time to allocate for understanding concepts
WP leader out of work for 4 month
????to be added
WPX WP name
Bruxelles, 3 April

40. Conclusions
WPX WP name
Bruxelles, 3 April 40 40

41. Conclusions Three or four bullets summarising the major points
WPX WP name
Bruxelles, 3 April