INFSOM-RI ETICS: E-infrastructure for Testing, Integration and Configuration of Software Alberto Di Meglio Project Manager

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 2 The ETICS Partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT The Condor batch system, distributed testing tools, service infrastructure, NMI

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 3 ETICS Organization Project Manager (Alberto Di Meglio) WP1 Project administrative and technical management Alberto Di Meglio WP2 Infrastructure and service management Marian Zurek Matteo Selmi WP3 Software configuration tools and methodologies Elisabetta Ronchieri Enrico Ferro, Matteo Selmi, Guillermo Diez WP4 Testing tools and methodologies WP5 Dissemination, certification and web tools Marc-Elian Begin PMB ETICS Experts Board

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 4 Systems status ETICS is born especially from the experience of gLiteETICS is born especially from the experience of gLite The goal is to provide efficient automated tools for building and testingThe goal is to provide efficient automated tools for building and testing It also must be as simple as possible to useIt also must be as simple as possible to use Currently the gLite build system allows to:Currently the gLite build system allows to: –Build the entire gLite project managing automatically internal and external dependencies –Produce packages like source and binary tarballs and binary RPMS for RHEL3/SLC3 –Trigger coding conventions, unit test, documentation tools and collect and publish the results (more or less automatically) But we know there are issues and missing functionality:But we know there are issues and missing functionality: –Dependency mechanism is powerful, but not very flexible and it’s difficult to find information –More package types are needed –Much better support for multiplatform development is needed –It must be possible to build everything from source –It must produce run-time configuration information from build information –Coupling with CVS and Ant is too tight –Better support for the release process must be provided (tagging, release notes, quick fix tracking, etc) –… And the current gLite system doesn’t provide any automated testing functionalityAnd the current gLite system doesn’t provide any automated testing functionality

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 5 Service Overview Build/Test Artefacts Web Application Report DB Project DB NMI Scheduler Clients Web Service NMI Client Wrapper Via browser Command- Line tools WNs ETICS Infrastructure Continuous Builds

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 6 ETICS Web Interface Web Application layoutWeb Application layout

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 7 NMI Interface Preliminary integration of the client with NMIPreliminary integration of the client with NMI

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 8 Continuous builds interface

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 9 Command line tools ClientClient

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 10 Databases and repositories Build/Test Artifacts Report DB Project DB Contains all project and configuration information, currently implemented as a MySQL DB Contains all project and configuration information, currently implemented as a MySQL DB Contains the output of builds and test runs. Implemented as a MySQL database Contains the output of builds and test runs. Implemented as a MySQL database Contains the artifacts produced by build and test runs (packages, text, files, complete distributions, etc). Implemented as a file system Contains the artifacts produced by build and test runs (packages, text, files, complete distributions, etc). Implemented as a file system

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 11 User profiles Developers: can do local builds of project, subsystems and components.Developers: can do local builds of project, subsystems and components. Integrators: can create and run integration configurationsIntegrators: can create and run integration configurations Testers: can register and submit tests attached to configurationsTesters: can register and submit tests attached to configurations Release Managers: can create and publish configurations, create release notes and other documentationRelease Managers: can create and publish configurations, create release notes and other documentation Project Managers: can create and manage projects, assign permissions, set policies, etcProject Managers: can create and manage projects, assign permissions, set policies, etc

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 12 What will change for you? Developers Impact: low.Impact: low. –A few new commands for creating workspaces and building components –Syntax is very close to familiar CVS commands –The general principle is the same as the gLite build system with workspaces containing everything that is needed for a build, the stage area, the repository area, etc. Functionality changes: highFunctionality changes: high –The smallest unit of build is a component. Dependencies can be set at the component level. No need to download full subsystems anymore, only what’s really needed (if the build configuration is correctly defined). –Three build modes are supported: –Source only, where everything (including external dependencies) is built from sources –Binary only, where everything except the modules in the currently built configuration are taken from the repository –Mixed, where everything is taken from the repository if it exists, otherwise is built from source –General functionality is improved (customization, platform detection, speed, etc) –Automatic early validation on multiple platforms comes for free without any additional effort by submitting builds to the ETICS service Timescale: From April 2006Timescale: From April 2006

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 13 What will change for you? Integrators Impact: mediumImpact: medium –The definition of external dependencies is now kept in the ETICS DB and the info in the Ant XML file is not used by ETICS. –CVS tags (both for subsystems and components) are still required in the same way as now, but they are not taken from any Ant XML file. Instead they are used to define CONFIGURATIONS in the ETICS database. –Some equivalent and improved tools for tagging, branching, etc have to be provided with support for ETICS configurations management Functionality changes: highFunctionality changes: high –Much easier to navigate the relationships, make modifications, clone existing configurations, submit configuration to be built on multiple platforms –Better categorization of dependencies: build-time, run-time or both, per platform –Automatic creation of run-time configuration information (environment variables) –The Continuous Integration tool has been redeveloped from scratch in python and it’s faster and doesn’t have the CruiseControl memory issues. –More package types are created. The first release of the ETICS packages will be able to produce: source and binary tarballs source, binary and debug RPMS debs MSIs Timescale: From April 2006Timescale: From April 2006

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 14 What will change for you? Testers Impact: highImpact: high –It may require a different way of working. Tests can be run by testers on multiple platforms and different conditions and test results analysed asynchronously –Tests may have to follow some conventions to fully exploit the ETICS functionality (but it’s not mandatory) Functional changes: highFunctional changes: high –Once a test is available, ETICS allows to register it and submit it for execution to any of the available platforms with custom scheduling –All results are archived in the ETICS database and it is possible to do functional, regression, stress and performance tests across different configurations, builds and releases –A number of common test output schemas are natively supported. Some conventions may have to be defined/agreed by projects to get full functionality Timescale: May-June 2006Timescale: May-June 2006

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 15 What will change for you? Release Managers/Project Managers Impact: lowImpact: low –No particular effort is requested from RM/PM, except to take the time to see what benefits they get from using the new tools Functionality changes: highFunctionality changes: high –ETICS supports several projects with access control and roles –Once a configuration has been successfully built and tested, it can be frozen and published as a release/RC. –Reporting tools will allow to create release notes and other QA reports directly from the ETICS web interface. Timescale: Summer 2006Timescale: Summer 2006

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 16 Project Timeline Jan 06Jun 06 Dec 06 Kick-off I Review All-hands meeting (Budapest) All-hands meeting (Geneva) April 2006 gLite 3.0 is built on ETICS May 2006 gLite WMS and VOMS tests run on ETICS More contributions are welcome from other interested parties September 2006 gLite 3.2 is fully built and tested on ETICS QA and Project Management tools are available

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 17 Conclusions ETICS provides a multi-platform environment for building and testing middleware and applicationsETICS provides a multi-platform environment for building and testing middleware and applications ETICS focus is on automation: of builds, of tests, of reporting, etc. The goal is to simplify life when managing complex software management tasksETICS focus is on automation: of builds, of tests, of reporting, etc. The goal is to simplify life when managing complex software management tasks We want it to be as simple as possible to use. For this we need your feedback:We want it to be as simple as possible to use. For this we need your feedback:

INFSOM-RI EGEE JRA1 All Hands Meeting- 23 March CERN 18 Thanks