1. www.eu-etics.org INFSOM-RI-026753 WP 4 : Testing Tools and Methodologies Status Report ETICS Review – 15 February 2008 Éva Takács (4D SOFT)

2. INFSOM-RI-026753 Content WP4 in numbers Goals Challenges Distributed testing Achievments Case study Future plans Conclusion

3. INFSOM-RI-026753 WP in numbers 4D Soft: PM – 24 ? UW: PM –18 To be completed following the template

4. INFSOM-RI-026753 Staffing and Resources/ 4D Soft ????? István ForgácsIstván Forgács Tasks:Tasks: – Coordination Etics, Diligent Eva Takács – full time EticsEva Takács – full time EticsTasks: – Implementation of plugins – Implementation and testing of installation modules – Diligent system testing – modelling in ETICS Andor Dirner – 30% Etics, 60% Diligent, 10% system administrationAndor Dirner – 30% Etics, 60% Diligent, 10% system administration Tasks:Tasks: – Implementation and testing of installation modules – Diligent testing 4

5. INFSOM-RI-026753 Deliverables Status ETICS-D4.1-Requirements for unit, functional and regression testing – M6ETICS-D4.1-Requirements for unit, functional and regression testing – M6 ETICS-D4.2-Distributed testing – M12ETICS-D4.2-Distributed testing – M12 ETICS-D4.3-Distributed testing – draft M18ETICS-D4.3-Distributed testing – draft M18 ETICS-D4.3-Distributed testing – final M22ETICS-D4.3-Distributed testing – final M22 5 ETICS 3rd All-Hands Meeting - University of Wisconsin, Madison - 23-25/05/2007

6. INFSOM-RI-026753 Goals The major goal of WP4 is to realize distributed testing by: Identification and analysis of test metrics Identification and selection of software testing tools and metric calculation utilities Adopting of the selected testing tools and metric calculation utilities to the ETICS build and test system (Test and Metrics Plugin Framework) by plugin implementation Implementation of distributed testing mechanisms

7. INFSOM-RI-026753 Challenges For the WP4 as technical work – –Realising distributed testing – a quite new aspect in software engineering – –Customising existing ‘traditional’ testing technics for cutting-edge new technologies – –Identifying the most suitable open source test tools that fit te best in the ETICS system and could help the users with testing facilities after performing builds and before releasing software packages For 4D Soft as company – –Organisational challenge – being a WP leader as an SME in a project having mostly large academic institutions – –Technical challenge - working with open source operation systems/tools widely adopted by academic institutions in contrast to the industry working mostly with commercially available software

8. INFSOM-RI-026753 Distributed testing in ETICS The aim of ETICS is to provide a distributed build/test framework where the functionalities of the framework and the testing tools are clearly separated. Having these principles in mind in ETICS distributed testing environment consists of four major elements: Test and Metrics Plugin framework – as part of the ETICS client (command line interface) Testing tools and Tools for Metrics collection as plugins of the Test and Metrics Plugin framework NMI/Metronome Build&Test system – as execution framework co-scheduling mechanism – feature that allows automated multi-site deployment and test execution

9. INFSOM-RI-026753 Test and Metrics Plugin Framework In order to host the wide range of testing tools (client-side), without breaking the ETICS’s core functionality, a plugin-framework was developed The goal of the plugin framework can be summarised as follows: Provide an extendible framework for common actions to be performed during build and test executionProvide an extendible framework for common actions to be performed during build and test execution Provide natural separation between specific build and test execution and analysis and metrics collectionProvide natural separation between specific build and test execution and analysis and metrics collection Provide open interface for users to register their own plugins (comercially available tools) to be executed during build and testProvide open interface for users to register their own plugins (comercially available tools) to be executed during build and test Provide raw data for build and test and repository services especially for tracking trends, having statisticsProvide raw data for build and test and repository services especially for tracking trends, having statistics

10. INFSOM-RI-026753 Why using Plugin Framework ? ETICS ships with some built-in plugins but the rational behind using „Eclipse style” plug-in architecture is that most of present and/or future ETICS users will come with some tests and testing tools already in place and of course they would like to use them when working with ETICS Users can plug comercially available testing tools Commercial providers can extend the sytem with their products

11. INFSOM-RI-026753 Examples of metrics provided Static analysis – –Single line of code count (SLOC) – –Cyclomatic complexity – –Depth of inheritance – –IPv6 compliance – –Check style – –Nr of possible bugs Dynamic analysis – –Code coverage – –Memory leaks Test execution – –Java: JUnit – –C++: CPPUNIT (CPPTEST) – –Python: PyUnit – –Shell: shutil – –Script/Executable

12. INFSOM-RI-026753 Implemented Test and Metrics plugins PyUnitPlugin.py (pyton unit tests)PyUnitPlugin.py (pyton unit tests) PyCoveragePlugin.py (pyton coverage)PyCoveragePlugin.py (pyton coverage) PyComplexityPlugin.pyPyComplexityPlugin.py SLOCCountPluginSLOCCountPlugin IPv6PluginIPv6Plugin WSInteroperabilityPluginWSInteroperabilityPlugin JUnitPlugin.py (junit)JUnitPlugin.py (junit) JUnitreportsPlugin.pyJUnitreportsPlugin.py JUnitemmaPlugin.py (emma coverage for junit)JUnitemmaPlugin.py (emma coverage for junit) JCoberturaPlugin.py (cobertura coverage for junit)JCoberturaPlugin.py (cobertura coverage for junit) 12

13. INFSOM-RI-026753 Implemented Test and Metrics plugins JCnnPlugin.py (javaccn)JCnnPlugin.py (javaccn) JDependPlugin.py (jdepend – dependency analyser)JDependPlugin.py (jdepend – dependency analyser) JPmdPlugin.py (pmd – source code quality)JPmdPlugin.py (pmd – source code quality) JFindbugsPlugin.py (findbugs – bug pattern analysis)JFindbugsPlugin.py (findbugs – bug pattern analysis) JRat.py (java profiler)JRat.py (java profiler) CFlawfinderPlugin.py (flawfinder - static analyser for C/C++)CFlawfinderPlugin.py (flawfinder - static analyser for C/C++) CPyPHPRatsPlugin.py (Rats – static analyser for C/C++, Perl, python, php)CPyPHPRatsPlugin.py (Rats – static analyser for C/C++, Perl, python, php) CValgrindPlugin.py (Valgrind – C profiler)CValgrindPlugin.py (Valgrind – C profiler) 13

14. INFSOM-RI-026753 Metrics and their tool support in ETICS MetricsTypeProgramming languages/ technologies ToolEtics Plugin complexitystaticJava/PythonJavancss/ x.pyJCcnPlugin/ PyComplexityPlugin.py design qualitystaticJavaJDependJDependPlugin nr of security bugsstaticC/C++/Python/Perl/ PHPFlawfinder, RATS CFlawfinderPlugin/ CPyPhpRatsPlugin nr of warningsstaticJavaPMD, FindbugsJPmdPlugin, JFindbugsPlugin lines of codestaticallSLOCCountPlugin coveragedynamicJava/PythonEmma/ Cobertura/ coverage.py JUnitemmaPlugin unit test/unit test reportsdynamicJava/ PythonJUnit/PyUnitJUnitPlugin/ JUnitreportsPlugin.py/ PyUnitPlugin.py interoperabilitystaticIPv6/web serviceIPv6Plugin/ WSInteroperabilityPlugin profilerdynamicC/C++/JavaJrat, ValgrindJRatPlugin/ CValgrindPlugin

15. INFSOM-RI-026753 Installation modules Additionally to plugins so-called installation modules have been implemented The aim of installation modules is to assist users in their installation and testing procedure by providing some general purpose installation modules, for example, tomcat, mysql or wscore. The installation modules are components with appropriate test commands to install, start and stop the services on which other test components can set dependencies in order to get those services running before the test is executed.

16. INFSOM-RI-026753 Major Tasks Performed/Achievments Distributed testing facilities are in place Particularly, – –distributed testing environment is in place and co-schedulig mechanism is implemented – –metrics and testing tools have been identified – –Test and Metrics Plugins Framework as part of ETICS client is in place – –approximately 15 test and metrics plugins have been implemented – –general purpose deployment scripts to assist testing have been implemented

17. INFSOM-RI-026753 Case study - Diligent system testing Remote regression testing using the Diligent system testing environment as a private resource – –grid7.4dsoft.hu/grid5.4dsoft.hu are attached to the etics pool Role of regression tests:Role of regression tests: – Initially, to assure the testers that the test environment system is testable. Successfull running of regression tests means for testers that they can start working on the implementing new test cases. – Later on, the initial set of test cases have been augmented with the new test cases implemented by testers.

18. INFSOM-RI-026753 Case study - Diligent system testing Applying Java static analysis plugins for core Diligent services Applied plugins: JFindbugsPlugin, JDependPlugin JPmdPlugin, JCcnPlugin The principal aim here has been to model a system testing environment in ETICS for Diligent components where the java static analysis plugins can run easily in an automatic way on a periodic manner.

19. INFSOM-RI-026753 Future plan As the test execution mechanism assuaring test process reproducibility is in place in ETICSAs the test execution mechanism assuaring test process reproducibility is in place in ETICS the future work in ETICS2 is to implement a modelling environment for creating advance test cases/ test workflowsto implement a modelling environment for creating advance test cases/ test workflows 19

20. INFSOM-RI-026753 Conclusion 20