1. Module 01 ETICS Overview ETICS Online Tutorials
Welcome to the ETICS Online Tutorials
Module 1: ETICS Overview

2. The Problem Distributed Computing, Distributed Development
Distributed computing almost always requires distributed development.
The development teams are often geographically spread and have different technical backgrounds and skills, use different operating systems and programming languages.
The services deployed in the distributed infrastructures have to talk to each other using agreed interfaces. They have to adhere to existing standards and comply with software development processes.
And tests have to done and passed along the entire process
How does ETICS helps?
ETICS Online tutorials - Module 01

3. The Solution: The ETICS System
Continuous software build, test and QA verification system
Set of core tools and services
Easily extensible with additional plugins
Scheduled (cron) or on-demand build and test jobs
User computers or remote distributed infrastructures
Built-in connectors to distribute build and test jobs on different types of infrastructures from standard job management systems to the grid
Open repository of configuration metadata, packages and build, test and QA reports
Support for standard package management systems like YUM
It’s multi-platform and independent from any specific language, build or test tool
ETICS is a continuous build, test and quality assurance validation system
It comes with an extensive set of built-in features, tools and services and can be easily extended by adding plugins
It can be used to execute scheduled jobs using cron tasks or on-demand user jobs. The jobs can be executed directly on a user computer or submitted to remote distributed infrastructures
ETICS comes with built-in connectors for standard job management systems or infrastructures, like PBS, LSF, the grid or clouds and more connectors can be added as needed
ETICS has an open repository of configuration information, software packages and build/test reports and support for well-know package management systems like YUM
ETICS is multi-platforms and independent from any specific programming language, source code management system or build and test tool
In a word ETICS is a complete software development management system that allows developers, integrators, testers and projects managers to take full control of their projects
ETICS Online tutorials - Module 01

4. Architecture
A typical ETICS architecture is composed of six major components
The Web Portal is the main graphical user interface. It used to define and manage project configurations, submit and monitor remote builds and tests, browse the packages and reports in repository and much more
The command-line client provides very similar functionality as the web portal, but it is mainly used to automate the build and test processes, execute scheduled builds and non-interactive jobs. It is also used to execute builds and tests on local user computers.
The Configuration service handles all configuration tasks allowing users to create and manage software components, configurations, dependencies, etc
The Repository service provides access to the Repository digital library to download and upload software packages, reports, quality assurance metrics, etc
The Execution Engines provide access to the remote management systems or infrastructure. Each engine is designed for a specific system and more engines can be added to let ETICS submit to different types of infrastructures
The Worker Nodes are the systems where remote builds and tests are executes. ETICS can run in physical nodes or virtual machines and has support to instantiate dynamically and automatically guest virtual machines as necessary.
ETICS Online tutorials - Module 01

5. The Configuration Web Client
The Configuration Web Client is the main access point to the ETICS Configuration database
The main ETICS installation is currently hosted at CERN and can be access simply by typing the URL
The system can be easily installed by the end-users to manage private projects
The application can be used by guest users, but most operations require a user to register in the system. Registration is based on standard x509 certificates. Users can have one or more role (developer, integrator, tester, etc) on almost any object in the database, allowing fine-grained authorization control
The application gives a complete overview of simple and complex projects, allowing to have full control on the project structure and configuration. Information about the source control system used, the build and test commands, the dependencies, etc are easily accessible and manageable
ETICS Online tutorials - Module 01

6. Command-line Client
The Command-line client provides a set of commands to perform many configuration, build and test operations
Using the commands and the provided Python APIs, the client can be integrated in more complex processes and workflows
The client is also the typical tool used by developers to reproduce on local computers the build and test tasks defined in the main ETICS database. The operations are executed in isolated spaces called Workspaces, ensuring as much as possible the reproducibility and portability of the execution environment on many different platforms.
ETICS Online tutorials - Module 01

7. Local and Remote Operations
Build and test jobs can be executed on the user computer
Or sent to remote distributed infrastructures
ETICS has connectors for different types of infrastructures (like the grid or clouds) and different job management system (like PBS, LSF or Condor)
The jobs can be scheduled using standard cron jobs and the ETICS command-line client
All necessary dependencies are handled by ETICS
Developers can build or test locally their software on a single platform and then submit the same build or test to be executed in parallel on multiple platforms
Build and test jobs can be executed on local user computers during the development and initial test phases and then submitted to remote worker nodes.
The ETICS Client ensures as much as possible that a job runs in the same way on local and remote compouters. It automatically handles the dependencies downloading and installing all required packages and libraries
Normally build jobs can be run as standard user without root privileges. The ETICS Client installs the packages in private repository without the need to change the machine configuration. This allows working on several projects and configurations at the same time, even with conflicting dependencies.
Once a developers has executed a build or test on his or her local machine, the same build or test can be submitted to the remote infrastructures to be validated in parallel on as many platforms as necessary. Detailed results are stored in the ETICS Repository for analisys
ETICS Online tutorials - Module 01

8. The Repository
ETICS provides an online repository where all software packages and build, test and QA reports are stored
The Repository generates dynamic YUM repository at every build to automate the execution of deployment, integration and functional tests
It can be browsed and queried using a web application with a REST interface, a command-line client or a mountable file system (AFS-based)
ETICS includes an online digital library called the Repository, where all packages created or used during builds and tests can be stored
The Repository also stores the reports of each build and test and the metrics generated during their execution by the various plugins
The Repository generates automatically APT and YUM-based repositories for every executed build, which can be used to automate the execution of deployment, integration and functional tests
The Repository content can be browsed using the Repository Web Application or a REST interface, which supports xpath and SQL queries. A mountable, AFS-based file system is also provided
ETICS Online tutorials - Module 01

9. The Repository Web Application
The Repository web application can be used to browse all available packages and reports.
Packages and reports are stored in a structure similar to the project structure used in the Configuration Web Application to make it easy to navigate complex projects
Packages are stored by project, component, version and platform. For each build or test submitted for remote execution, the Repository shows a summary of the results with direct links to the corresponding packages and reports.
For each package a property page can be easily displayed showing all details of a package for any of the supported formats
ETICS Online tutorials - Module 01

10. Extensibility and Plugins
ETICS comes preconfigured with extensive build and test features
It can also be extended by adding additional plugins
The plugins are provided by ETICS and by its user community and can be easily installed from the online repository
ETICS comes with extensive build and test feature and it can be extended even more with plugins
Plugins are thin wrappers around other build and test tools. The wrapper contain the instructions to execute a tool and allow the ETICS client to activate it when and if needed according to user defined policies or profiles
Many plugins are available from the ETICS developers and many more can be created by the ETICS user community using the simple specifications provided
ETICS Online tutorials - Module 01

11. Programming languages/ technologies
Examples of Plugins
Metrics
Type
Programming languages/ technologies
Tool
ETICS Plugin
Complexity
static
Java
Python
Javancss
JCcnPlugin
PyComplexityPlugin.py
Design quality
Jdepend
JDependPlugin
N of potential bugs
C/C++
Perl
PHP
Flawfinder,
RATS
PMD
Findbugs
CFlawfinderPlugin
CPyPhpRatsPlugin
JPmdPlugin
JFindbugsPlugin
dynamic
Valgrind
CValgrindPlugin
Lines of code
All
SLOCCount
SLOCCountPlugin
Coverage
Emma
Cobertura
JUnitemmaPlugin
JCoberturaPlugin
Unit tests success rate
JUnit
PyUnit
JUnitPlugin
JUnitreportsPlugin.py PyUnitPlugin.py
Compliance with standards
IPv6
WSI
IPv6Plugin WSInteroperabilityPlugin
Profiling
Jrat
JRatPlugin
Plugins are already available for many different types of static and dynamic tests and for several programming languages
ETICS provides also best practice guidelines and workflows for the execution of the plugins and the collection of quality assurance metrics
The A-QCM or Automated Quality Certification Model is a practical QA validation method designed and implemented by ETICS that can be used to measure and monitor several quality aspects of software projects
ETICS Online tutorials - Module 01

12. Distributed Testing
One of most powerful features of ETICS is the “Distributed Testing” or “Parallel Testing”
This features allows to design complex tests composed of several interacting nodes and to deploy automatically complete test environments
This feature is currently being developed and will be available to the ETICS user community in the Summer 2009
Finally some words about the ETICS ‘Distributed Testing’ feature, also known as ‘Parallel Testing’.
This features allows designing and executing with ETICS complex test workflows involving multiple nodes, services, clients, test scripts, etc
ETICS analysis the test provided by the user and allocates automatically the required number of worker nodes, deploys and configures the various client and services using the information stored in its Configuration database and then executes the tests.
Synchronization among the different components of the workflow is done using two simple instrumentation APIs, the ETICS ‘getters’ and ‘setters’ that allows the components to exchange information and orchestrate the correct flow of execution
This feature is currently under development. Prototypes have been designed and implemented in collaboration with some ETICS users. The final version will be available during Summer 2009
ETICS Online tutorials - Module 01

13. 02 – The ETICS Portal and Web Applications
Next Module
02 – The ETICS Portal and Web Applications
Now that you have become familiar with the general concepts and feature of ETICS, we are ready to use it starting with a more detailed overview of the ETICS Portal and its Web Applications
ETICS Online tutorials - Module 01

14. http://www.eticsproject.eu Thanks!
More information about ETICS can be found of the web site at
ETICS Online tutorials - Module 01