The ETICS Build and Test Service ETICS Project, CERN

Contents Introduction The System Conclusions ETICS Features Service architecture Data model Execution engine Reporting and Repository Administration Conclusions ETICS System Induction Training - FZJ - 15-16 April 2008

Introduction ETICS System Induction Training - FZJ - 15-16 April 2008

Training Content ETICS System Overview Building and testing with ETICS this presentation Building and testing with ETICS How to use the ETICS System to build and test software either locally or remotely ETICS Metadata Editing How to create, modify, clone, remove configuration objects either locally or in the central database The ETICS Web Applications How to use the ETICS Portal, the Configuration application and the Repository application to manage builds and tests and analyse reports and metrics ETICS Multinode Testing An overview of the ETICS multinode testing capabilities ETICS System Induction Training - FZJ - 15-16 April 2008

The System ETICS System Induction Training - FZJ - 15-16 April 2008

ETICS Features Out-of-the-box automatic build/test system Multiplatform support for development and testing Powerful and flexible automatic dependency management (internal and external dependencies) Trigger coding convention checks, unit test, documentation tools and collect and publish the results Possible to build everything from source or use pre-built binary packages Designed to support several Version Control Systems Possible to register artefacts on the repository Publishes run-time configuration information such as environment variables, runtime dependencies, etc … Produce different package formats (rpms, tar, msi, debs) Produces rich build and test reports Ability to run custom plugins during build and test procedure Register the build/tests artefacts on the repository ETICS System Induction Training - FZJ - 15-16 April 2008

ETICS Portal (myETICS, Build/Test, Repository, Administration) Architecture ETICS Portal (myETICS, Build/Test, Repository, Administration) Web Service Repository Service Via browser Report DB Build/Test Artefacts Project DB Metronome Execution Engine Clients Metronome Client Wrapper WNs ETICS Infrastructure ETICS System Induction Training - FZJ - 15-16 April 2008

Service Architecture ETICS Service build as an N-tier architecture All elements of ETICS are configured, built and tested with ETICS ETICS Service is Open Source: Apache 2.0 license ETICS System Induction Training - FZJ - 15-16 April 2008

Data model - 1 Project: container for components and subsystems Subsystem: container for components Component: a module configuration can also be made of sub-module’s configurations Module: generic way to refer to project, subsystem, component Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms) P S Conf C Configuration: different versions of the modules. A configuration could be identified by ags, dependencies, platforms. ETICS System Induction Training - FZJ - 15-16 April 2008

Data model - 2 Plat P S Conf C Platform: represents an specific combination of architecture, operating system and C/C++ compiler E.g. slc4_ia32_gcc346 Allows users to specify platform specific metadata VCS commands (e.g. CVS) Build commands Test commands Properties Environment variables Dependencies Plat P S Conf C Platforms can be attached to configurations. Each platform can contain specific buildCommand, vcsCommand, properties, ... ETICS System Induction Training - FZJ - 15-16 April 2008

ETICS Web Application ETICS Infrastructure Web Application Web Service Via browser Report DB Project DB Build/Test Artefacts NMI Execution Engine NMIJob execution engine, develped at Wisconsin Clients 2 (gui and command line) Etics is: Client: allows you to build locally . WebApplication . WebService . The submission infrastructure . Repository Different repositories Clients NMI Client Wrapper WNs ETICS Infrastructure ETICS System Induction Training - FZJ - 15-16 April 2008

Web Applications ETICS is composed of the following Web Applications: The ETICS Portal The Configuration application The Repository applications The Administration application The applications are integrated under a single Portal https://etics.cern.ch/eticsPortal ETICS System Induction Training - FZJ - 15-16 April 2008

The ETICS Portal https://etics.cern.ch/eticsPortal Short live overview ETICS System Induction Training - FZJ - 15-16 April 2008

Command-line Client ETICS Infrastructure Web Application Web Service Via browser Report DB Project DB Build/Test Artefacts NMI Execution Engine Clients NMI Client Wrapper WNs ETICS Infrastructure ETICS System Induction Training - FZJ - 15-16 April 2008

Command-line Client – (1) Build and Test Command-Line client (CLI) allows users to Build and test locally Browse metadata Edit metadata Submit build and test jobs CLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisation Interfaces with the ETICS Build and Test Web Service ETICS System Induction Training - FZJ - 15-16 April 2008

Command-line Client Local build – (2) Download project metadata: etics-get-project <project-name> Action: download project metadata Checkout and/or download software: etics-checkout –c <config-name> <module-name> Action: download binary packages or fetch code from source Build the software: etics-build <module-name> Action : build and unit test code hieratically, and generate packages and reports Test the software: etics-test <module-name> Action : test code hieratically, and generate packages and reports ETICS System Induction Training - FZJ - 15-16 April 2008

Command-line Client Editing - (3) Command-line Client can also be used for editing: etics-module [<options>] [<module-name>] etics-configuration [<options>] [<module-name> Can be used for scripting or interactive sessions Provides the same features as the Web Application ETICS System Induction Training - FZJ - 15-16 April 2008

Command-line Client Client Plugin-Framework – (4) Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment Users can select, from a rich set, which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible Promote inclusion of contributions from ETICS user community (more in this in dissemination presentation) Examples of already available plugins: IPv6 compliance tests xUnit tests automatic execution Multi-language line of code count (SLOC) ETICS System Induction Training - FZJ - 15-16 April 2008

Configuration Web Service Web Application Web Service Via browser Report DB Project DB Build/Test Artefacts NMI Execution Engine NMIJob execution engine, develped at Wisconsin Clients 2 (gui and command line) Etics is: Client: allows you to build locally . WebApplication . WebService . The submission infrastructure . Repository Different repositories Clients NMI Client Wrapper WNs ETICS Infrastructure ETICS System Induction Training - FZJ - 15-16 April 2008

WS - (1) The Configuration Web Service (WS) provides single secure interface to the ETICS backend system Metadata (persisted in database) Execution engine Security (role based) Heart of the ETICS System, it performs complex queries on behalf of the clients (WA and CLI) ETICS System Induction Training - FZJ - 15-16 April 2008

WS – Security – (2) All accesses to the WS must be performed using a secure port Authentication is performed using digital certificates (x.509) Authorisation is role-based Clients connecting without certificate are mapped to Guest, with minimal privileges (most likely just read access) Service can be setup to leverage certification authority management efforts such as EUGridPMA ETICS System Induction Training - FZJ - 15-16 April 2008

Execution Engine ETICS Infrastructure Web Application Web Service Via browser Report DB Project DB Build/Test Artefacts NMI Execution Engine NMIJob execution engine, develped at Wisconsin Clients 2 (gui and command line) Etics is: Client: allows you to build locally . WebApplication . WebService . The submission infrastructure . Repository Different repositories Clients NMI Client Wrapper WNs ETICS Infrastructure ETICS System Induction Training - FZJ - 15-16 April 2008

Metronome - Execution engine Metronome is a job execution engine internally used by ETICS to manage remote build and test jobs on different platforms Metronome is a layer built on the top of Condor Condor is a middleware distribution that “provides mechanisms and policies supporting High Throughput Computing (HTC) on large collections of distributed computing resources” Both Metronome and Condor are developed and supported by the University of Wisconsin in Madison, USA ETICS System Induction Training - FZJ - 15-16 April 2008

Metronome - Web interface http://etics.cern.ch/nmi ETICS System Induction Training - FZJ - 15-16 April 2008

Metronome- Additional features The following Metronome features are available through the ETICS interface Private resources: projects can provide specific resources for exclusive usage Freeze: lock the node for further analysis in case of failure Root-enabled environment: build or test jobs can request to run as root Users can register private machines if special setup and/or specific licenses are required ETICS System Induction Training - FZJ - 15-16 April 2008

Metronome - ETICS sites Server at CERN etics.cern.ch SLC3, SLC4, RHES, Debian, FC (32- and 64-bits), WinXP, Mac OSX Server at Univ. of Wisconsin, Madison, USA HP HPUX, IBM AIX, OSF1, SGI Irix, Sun Solaris, Linux (several flavours) Server at INFN, Bologna, Italy CentOS, SLC3, SLC4 (32- and 64-bits) ETICS System Induction Training - FZJ - 15-16 April 2008

Reporting and Repository Full execution reports are generated when running builds and tests The reports include the information generated by the plugins All build and test reports and the packages generated by the running the ETICS System remotely are stored in the Repository There are two areas in the Repository: The Registered area: stores reports and packages produced by locked configurations, cannot be modified The Volatile area: stores reports and package for any type of configuration, limited history, can be modified (overwritten) ETICS System Induction Training - FZJ - 15-16 April 2008

Getting started ETICS System Induction Training - FZJ - 15-16 April 2008

ETICS registration https://etics.cern.ch:8443/eticsAdmin/ /public/registration/requestRegistration.jsp ETICS System Induction Training - FZJ - 15-16 April 2008

Conclusions For more information: etics-discuss@cern.ch etics-support@cern.ch Some of these projects early adopters, different profiles ETICS System Induction Training - FZJ - 15-16 April 2008

Thanks http://www.eu-etics.org ETICS System Induction Training - FZJ - 15-16 April 2008