1. The ETICS Build and Test Service
ETICS Project, CERN

2. Contents Introduction The System Conclusions ETICS Features
Service architecture
Data model
Execution engine
Reporting and Repository
Administration
Conclusions
ETICS System Induction Training - FZJ April 2008

3. Introduction
ETICS System Induction Training - FZJ April 2008

4. Training Content ETICS System Overview Building and testing with ETICS
this presentation
Building and testing with ETICS
How to use the ETICS System to build and test software either locally or remotely
ETICS Metadata Editing
How to create, modify, clone, remove configuration objects either locally or in the central database
The ETICS Web Applications
How to use the ETICS Portal, the Configuration application and the Repository application to manage builds and tests and analyse reports and metrics
ETICS Multinode Testing
An overview of the ETICS multinode testing capabilities
ETICS System Induction Training - FZJ April 2008

5. The System
ETICS System Induction Training - FZJ April 2008

6. ETICS Features Out-of-the-box automatic build/test system
Multiplatform support for development and testing
Powerful and flexible automatic dependency management (internal and external dependencies)
Trigger coding convention checks, unit test, documentation tools and collect and publish the results
Possible to build everything from source or use pre-built binary packages
Designed to support several Version Control Systems
Possible to register artefacts on the repository
Publishes run-time configuration information such as environment variables, runtime dependencies, etc …
Produce different package formats (rpms, tar, msi, debs)
Produces rich build and test reports
Ability to run custom plugins during build and test procedure
Register the build/tests artefacts on the repository
ETICS System Induction Training - FZJ April 2008

7. ETICS Portal (myETICS, Build/Test, Repository, Administration)
Architecture
ETICS Portal (myETICS, Build/Test, Repository, Administration)
Web Service
Repository Service
Via browser
Report DB
Build/Test
Artefacts
Project DB
Metronome Execution Engine
Clients
Metronome Client
Wrapper
WNs
ETICS Infrastructure
ETICS System Induction Training - FZJ April 2008

8. Service Architecture ETICS Service build as an N-tier architecture
All elements of ETICS are configured, built and tested with ETICS
ETICS Service is Open Source: Apache 2.0 license
ETICS System Induction Training - FZJ April 2008

9. Data model - 1
Project: container for components and subsystems
Subsystem: container for components
Component: a module configuration can also be made of sub-module’s configurations
Module: generic way to refer to project, subsystem, component
Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms) P S
Conf C
Configuration: different versions of the modules. A configuration could be identified by ags, dependencies, platforms.
ETICS System Induction Training - FZJ April 2008

10. Data model - 2 Plat P S Conf C
Platform: represents an specific combination of architecture, operating system and C/C++ compiler
E.g. slc4_ia32_gcc346
Allows users to specify platform specific metadata
VCS commands (e.g. CVS)
Build commands
Test commands
Properties
Environment variables
Dependencies
Plat P S
Conf C
Platforms can be attached to configurations. Each platform can contain specific buildCommand, vcsCommand, properties, ...
ETICS System Induction Training - FZJ April 2008

11. ETICS Web Application ETICS Infrastructure Web Application Web Service
Via browser
Report DB
Project DB
Build/Test
Artefacts
NMI Execution Engine
NMIJob execution engine, develped at Wisconsin
Clients 2 (gui and command line)
Etics is:
Client: allows you to build locally
. WebApplication
. WebService
. The submission infrastructure
. Repository
Different repositories
Clients
NMI Client
Wrapper
WNs
ETICS Infrastructure
ETICS System Induction Training - FZJ April 2008

12. Web Applications ETICS is composed of the following Web Applications:
The ETICS Portal
The Configuration application
The Repository applications
The Administration application
The applications are integrated under a single Portal
ETICS System Induction Training - FZJ April 2008

13. The ETICS Portal https://etics.cern.ch/eticsPortal Short live overview
ETICS System Induction Training - FZJ April 2008

14. Command-line Client ETICS Infrastructure Web Application Web Service
Via browser
Report DB
Project DB
Build/Test
Artefacts
NMI Execution Engine
Clients
NMI Client
Wrapper
WNs
ETICS Infrastructure
ETICS System Induction Training - FZJ April 2008

15. Command-line Client – (1)
Build and Test Command-Line client (CLI) allows users to
Build and test locally
Browse metadata
Edit metadata
Submit build and test jobs
CLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisation
Interfaces with the ETICS Build and Test Web Service
ETICS System Induction Training - FZJ April 2008

16. Command-line Client Local build – (2)
Download project metadata: etics-get-project <project-name>
Action: download project metadata
Checkout and/or download software:
etics-checkout –c <config-name> <module-name> Action: download binary packages or fetch code from source
Build the software: etics-build <module-name> Action : build and unit test code hieratically, and generate packages and reports
Test the software: etics-test <module-name> Action : test code hieratically, and generate packages and reports
ETICS System Induction Training - FZJ April 2008

17. Command-line Client Editing - (3)
Command-line Client can also be used for editing: etics-module [<options>] [<module-name>] etics-configuration [<options>] [<module-name>
Can be used for scripting or interactive sessions
Provides the same features as the Web Application
ETICS System Induction Training - FZJ April 2008

18. Command-line Client Client Plugin-Framework – (4)
Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment
Users can select, from a rich set, which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible
Promote inclusion of contributions from ETICS user community (more in this in dissemination presentation)
Examples of already available plugins:
IPv6 compliance tests
xUnit tests automatic execution
Multi-language line of code count (SLOC)
ETICS System Induction Training - FZJ April 2008

19. Configuration Web Service
Web Application
Web Service
Via browser
Report DB
Project DB
Build/Test
Artefacts
NMI Execution Engine
NMIJob execution engine, develped at Wisconsin
Clients 2 (gui and command line)
Etics is:
Client: allows you to build locally
. WebApplication
. WebService
. The submission infrastructure
. Repository
Different repositories
Clients
NMI Client
Wrapper
WNs
ETICS Infrastructure
ETICS System Induction Training - FZJ April 2008

20. WS - (1)
The Configuration Web Service (WS) provides single secure interface to the ETICS backend system
Metadata (persisted in database)
Execution engine
Security (role based)
Heart of the ETICS System, it performs complex queries on behalf of the clients (WA and CLI)
ETICS System Induction Training - FZJ April 2008

21. WS – Security – (2)
All accesses to the WS must be performed using a secure port
Authentication is performed using digital certificates (x.509)
Authorisation is role-based
Clients connecting without certificate are mapped to Guest, with minimal privileges (most likely just read access)
Service can be setup to leverage certification authority management efforts such as EUGridPMA
ETICS System Induction Training - FZJ April 2008

22. Execution Engine ETICS Infrastructure Web Application Web Service
Via browser
Report DB
Project DB
Build/Test
Artefacts
NMI Execution Engine
NMIJob execution engine, develped at Wisconsin
Clients 2 (gui and command line)
Etics is:
Client: allows you to build locally
. WebApplication
. WebService
. The submission infrastructure
. Repository
Different repositories
Clients
NMI Client
Wrapper
WNs
ETICS Infrastructure
ETICS System Induction Training - FZJ April 2008

23. Metronome - Execution engine
Metronome is a job execution engine internally used by ETICS to manage remote build and test jobs on different platforms
Metronome is a layer built on the top of Condor
Condor is a middleware distribution that “provides mechanisms and policies supporting High Throughput Computing (HTC) on large collections of distributed computing resources”
Both Metronome and Condor are developed and supported by the University of Wisconsin in Madison, USA
ETICS System Induction Training - FZJ April 2008

24. Metronome - Web interface
ETICS System Induction Training - FZJ April 2008

25. Metronome- Additional features
The following Metronome features are available through the ETICS interface
Private resources: projects can provide specific resources for exclusive usage
Freeze: lock the node for further analysis in case of failure
Root-enabled environment: build or test jobs can request to run as root
Users can register private machines if special setup and/or specific licenses are required
ETICS System Induction Training - FZJ April 2008

26. Metronome - ETICS sites
Server at CERN etics.cern.ch
SLC3, SLC4, RHES, Debian, FC (32- and 64-bits), WinXP, Mac OSX
Server at Univ. of Wisconsin, Madison, USA
HP HPUX, IBM AIX, OSF1, SGI Irix, Sun Solaris, Linux (several flavours)
Server at INFN, Bologna, Italy
CentOS, SLC3, SLC4 (32- and 64-bits)
ETICS System Induction Training - FZJ April 2008

27. Reporting and Repository
Full execution reports are generated when running builds and tests
The reports include the information generated by the plugins
All build and test reports and the packages generated by the running the ETICS System remotely are stored in the Repository
There are two areas in the Repository:
The Registered area: stores reports and packages produced by locked configurations, cannot be modified
The Volatile area: stores reports and package for any type of configuration, limited history, can be modified (overwritten)
ETICS System Induction Training - FZJ April 2008

28. Getting started
ETICS System Induction Training - FZJ April 2008

29. ETICS registration https://etics.cern.ch:8443/eticsAdmin/
/public/registration/requestRegistration.jsp
ETICS System Induction Training - FZJ April 2008

30. Conclusions For more information: etics-discuss@cern.ch
Some of these projects early adopters, different profiles
ETICS System Induction Training - FZJ April 2008

31. Thanks http://www.eu-etics.org
ETICS System Induction Training - FZJ April 2008