JRA2 – TEST MANAGEMENT TOOLS WP6 JRA2 – TEST MANAGEMENT TOOLS ETICS2 first Review Eva Takacs 4D SOFT Bruxelles, 3 April 2009

Contents Goals Case studies Major Achievements Challenges Metrics and Statistics Conclusions WPX WP name Bruxelles, 3 April

Goals WPX WP name Bruxelles, 3 April 3 3

Major goals of ETICS test services ETICS aims to provide a distributed, complex build/test framework where the functionalities of the framework and the testing tools are treated as separate entities and which follows the actual trend in software quality assurance of providing “scriptless tools that can be used by nondevelopers” (Gartner Group) and which promotes testing by offering really user friendly, hands-on services contributes to standards WPX WP name Bruxelles, 3 April

Test services in ETICS1 Having these principles in mind in ETICS1 distributed testing environment consisted of four major elements: Distributed multinode testing mechanism Test and Metrics Plugin framework – as part of the ETICS client (command line interface) Testing tools and Tools for Metrics collection as plugins of the Test and Metrics Plugin framework Metronome Build&Test system – as multi-platform execution framework WPX WP name Bruxelles, 3 April

Testing and validating with ETICS ETICS testing functionality is provided by plugins, which are Python wrappers around external test tools. These testing tools are used by ETICS to analyse particular aspects of the code or execute test suites. Example of plugins are: Sloccount, PyUnit, Junit, Pycoverage, JEmma, Findbugs, CCCC, CKJM, IPv6, WSI ETICS CLI contains the Test and Metrics Plugin Framework, a well-defined interface, that allows the ETICS Client to activate the plugins during an etics-build process Policies called profiles (for example: junit, wsi) activates the plugins during the build WPX WP name Bruxelles, 3 April

Plugin implementation Plugins can be implemented by user by developing a Python wrapper around the test tool following the rules defined in the interface provided by Test And Metrics Plugin Framework JRA2 extention to plugin implementation Plugin Generator Service – by providing templates and guidance semi-automates the implementation. Fulfils the objective of separating the test tools from the ETICS framework providing „scriptless” tools user friendly, easy to use services WPX WP name Bruxelles, 3 April

Plugin deployment, usage Plugins could be made available by the ETICS system administrator Mandatory plugins will be downloaded in the ETICS CLI Optional plugins could be downloaded by users on demand in their local environment Individual plugin implementations are available in the respective local environment JRA2 extention to plugin deployment, usage Test Tool Registration System – web application for general purpose plugin registration, management, serves as an Information System on test tools/plugins Fulfils the objective of user friendly testing services, high level front-end application for the plugin management WPX WP name Bruxelles, 3 April

Plugins and Metrics JRA2 extension to metrics format The plugins typically collect information and publish data in the form of Metrics Metrics are published with each build/test reports and are also stored individually in the ETICS Repository The collected metrics values can be used further to analyse a project status or behaviour (trend analysis) and for the certification (A-QCM) JRA2 extension to metrics format Standardised xsd formats for the summary and detail metrics representation Fulfils the objective of contribution to standards, improved testing services by providing better reports WPX WP name Bruxelles, 3 April

Multinode test mechanism The ability to setup scenarios where multiple services are automatically deployed in multiple nodes. The user has to syncronise the process and to control the order of operations using the information service (etics-get, etics-set) provided by ETICS Command Line client. JRA2 extention to multinode mechanism Workflow Designer – high level front-end web application able to orchestrate complex, multinode test scenarious Fulfils the objectives of Providing „scriptless” tools for test design User friendly, high level test services for distributed systems WPX WP name Bruxelles, 3 April

Main objectives of the JRA2 activity Summary Develop a registration system by which the ETICS users can register their tools to be integrated into the ETICS testing infrastructure Provide an advanced interface (workflow designer) for the users to construct their complex multinode deployments scenarios and testing process in a convenient way Develop the necessary test interface standards for proper collaboration of testing tools and the proper collaboration with the job submission systems and also presentation of the results in standard formats Provide the mean for users to visualise their test results WPX WP name Bruxelles, 3 April

Case studies WPX WP name Bruxelles, 3 April 12 12

Case study 1 of the ETICS Test System CKJM’s plugins lifecycle Description of the case study In order to perform the certification process inside NA2 work package the Evaluation Model of the A-QCM (Automated – Quality Certification Model) requires calculation of Structural Testing/Static Analysis metrics. Among them, Chidamber and Kemerer object-oriented metrics fall in this category. WPX WP name Bruxelles, 3 April

Case study 1 of the ETICS Test System CKJM’s plugins lifecycle Metrics proposed by Chidamber and Kemerer are: WMC: Weighted methods per class DIT: Depth of Inheritance Tree NOC: Number of Children CBO: Coupling between object classes RFC: Response for a Class LCOM: Lack of cohesion in methods Ca: Afferent couplings NPM: Number of public methods The ckjm open source tool calculates these metrics for Java. (http://www.spinellis.gr/sw/ckjm/) Bruxelles, 3 April

Case study 1 of the ETICS Test System CKJM’s plugins lifecycle In order to have the required metrics available for A-QCM we have to Register ckjm as a plugin in Test Tool Registration system. As a result of this operation the JCkjmPlugin is available for ETICS users Create the project configuration structure for the program under analysis or chosing an existing one according to requirements. Setting the profiles for the required ETICS project element Execute the build for the required configuration. As a result of the build the metrics will be available in the ETICS repository for further analysis for the A-QCM service. WPX WP name Bruxelles, 3 April

Case study 2 - Interoperability Scenario Multinode test design Case study description The case study is interoperability in job submission among different middleware implementations The test involve ARC, gLite and UNICORE and tests job submission with any of the clients towards any of the servers. The packages are taken from their official distribution repositories The interoperability tests and the respective ARC, UNICORE and gLite clients are expected to be in a same node and These clients call the external servers/services maintained by their system administrators WPX WP name Bruxelles, 3 April

Case study 2 - Interoperability Scenario Multinode test design Moving Interoperability Scenario to multinode The goal of this use case is moving the interoperability test to multinode to be able to deploy a complete test infrastructure without having to rely on external services. In the existing one node scenario where the clients and the tests are in one node, the UNICORE server has to be available in Juelich, the ARC server in Kosice and the CREAM server in Padova. WPX WP name Bruxelles, 3 April

Case study 2 - Interoperability Scenario Multinode test design Description of the ETICS process - steps Create the project configuration structure Design the workflow of the deployment process Create an abstract Workflow Create a concrete multinode workflow Browse the project configuration data Set the flow connectors (data that communicates between nodes, for example, hostname) Set the phases of the deployment Add the necessary deployment scripts for different phases Generate the output xml file – it serves as input for the job submission service WPX WP name Bruxelles, 3 April

Major Achievements WPX WP name Bruxelles, 3 April 19 19

ETICS Test System ETICS Test System is the new portlet containing the frontend applications for testing services and is accessed via a new tab of the current ETICS main portal. The main components (portlets and the associating web services of this new tab) of the ETICS Test System are the following: Test Tool Registration System – portlet and the underlying services for supporting test tool registration adding/removing test tools, plugins, licensed products to and from the ETICS framework Plugin Generation Service - service for plugin template generation Workflow Designer - workflow tool and the associated services for complex test design

ETICS Test System Architecture WPX WP name Bruxelles, 3 April

Test Tool Registration System The motivation that lies behind the test tools registration is to augment the current Test and Metrics Plugin Framework by applying a generic registration/installation procedure for testing tool providers, supporting the plugin implementation by providing wizards and templates for the users, on demand download of the plugins by providing information on the availability of plugins and to enhance the variety of testing tools inside ETICS framework. The registration can be conducted by means of various operations depending on the functionality of test tool, licence, property issues. As a result of the registration operation, the testing tools are deployed on well defined locations and they are available for ETICS testing processes. WPX WP name Bruxelles, 3 April

Plugin Generation Service The aim of this service is to augment the current plugins implementation practice by generating a plugin template. Plugin Generator Service creates a plugin (python wrapper) template around the registered tools in the Test Registration System during the registration process. The generated template is the ‘standard’ implementation of a ‘standard test tool’. This template can be enhanced by the users’ according to their needs. WPX WP name Bruxelles, 3 April

Example case study1: CKJM Registration Process Creates JCkjmPlugin WPX WP name Bruxelles, 3 April

Example case study 1- CKJM registration process Uploading the JCkjmPlugin WPX WP name Bruxelles, 3 April

Example case study 1- CKJM registration process JCkjmPlugin in production state - available for users WPX WP name Bruxelles, 3 April

Standard interfaces ETICS Test System provides standard formats for the reports’ format of static and dynamic analyser tools integrated as plugins. The proposed formats are based on the format provided by the NIST SAMATE (Software Assurance Metrics And Tool Evaluation) project for the static analysis tools. Why we suggest using standard interfaces for report generation? to speed adoption of the tools by objectively demonstrating their use on real software the value of a tool depends on how effectively developers can use the tgenerated reports to review and address issues improved readability of the generated reports could result in more identified bugs, errors having standardised reports leads to standardised processes WPX WP name Bruxelles, 3 April

Workflow Designer Two scenarios to define complex tests have been identified in ETICS: multinode test design, when the required services are deployed on different mashines (the message passing between different phases of service deployment on different nodes is crucial) test process design, when the focus is on the orchestration of different testing tools (the dependencies and the necessary exchange of reports are crucial among registered and deployed test tools/plugins) WPX WP name Bruxelles, 3 April

Main features of Workflow Designer Supported ETICS workflows Multinode deployment tests Complex test process design Design concepts Abstract and Concrete Workflows Workflow repository Workflow status management Implementation is based on standard and widespread technologies JSR168 compliant portlets for compatibility WS-PGRADE/GridSphere framework for plugable system WebServices/SOA concept for robust and scalable architecture XML based workflow description for interfacing WPX WP name Bruxelles, 3 April

Architecture of Workflow Designer WPX WP name Bruxelles, 3 April

Workflow Designer - Test process Goal: Orchestration of registered plugins Support for Cyclic execution of plugins and workflows Conditional execution Provides interface to repository of registered plugins to workflow interpreter WPX WP name Bruxelles, 3 April

Workflow Designer - Multinode deployment test Goal: Orchestration of deployment phases of service sets Support for Definition of deployment phases for configurations Data dependencies based on key-pairs Provides interface to configuration information to workflow interpreter WPX WP name Bruxelles, 3 April

Example case study 2 Interoperability Workflow WPX WP name Bruxelles, 3 April

Example case study 2 -Output of the Workflow design Input for job submission service <service name="string" module="string">   <version name="string" configuration="string">       <platformMapping platform="string">           <phase name="string" script="string">               <logEntry type="pre/post"/>               <pair key="string"                     type="requires/provides"                     scope="phase/service/node/global"                     mandatory="boolean"                     resolutionTime="Runtime/Design"                     valueResolution="string"                     defaultValue="string"/>               <runsAfter phase="string"/>           </phase>       </platformMapping>   </version> </service> WPX WP name Bruxelles, 3 April

Achievements of JRA2 Summary The main achievement of JRA2 in ETICS2 is the extention of the existing distributed testing mechanism with advanced functionalities by supplying a Test Tool Registration and Plugin Generation Service mechanism allowing ETICS users to register their tools equipped with the necessary Information System by supplying standard report formats by supplying a high level Workflow Designer able to orchestrate distributed deployment and testing scenarious with the goal of reducing the complexity of designing and implementing these complex tests for applications WPX WP name Bruxelles, 3 April

Metrics and Statistics WPX WP name Bruxelles, 3 April 36 36

Metrics and Statistics Deliverables DJRA2.1 – Use-cases Specification DJRA2.2 – Test interface standards DJRA2.3 – Test tool registration system (prototype) DJRA2.4 – User interface and Test Execution Generator (prototype) Statistics as defined on the project QA Plan ???? To be added WPX WP name Bruxelles, 3 April

Challenges WPX WP name Bruxelles, 3 April 38 38

Challenges Technical challenges Organisational challenges Applying high-level workflows for testing scenarious Integration of different challenging technologies, GWT, GridSphere, web services Organisational challenges New people in ETICS2 – time to allocate for understanding concepts WP leader out of work for 4 month ????to be added WPX WP name Bruxelles, 3 April

Conclusions WPX WP name Bruxelles, 3 April 40 40

Conclusions Three or four bullets summarising the major points WPX WP name Bruxelles, 3 April