1. www.eu-etics.org INFSOM-RI-026753 The ETICS Build and Test Service ETICS Project, CERN

2. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 2 Contents IntroductionIntroduction Use CasesUse Cases ServiceService –ETICS Features –Service architecture –Data model –NMI execution engine –Reporting and repository Conclusions and future workConclusions and future work

3. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 3 Introduction

4. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 4 ETICS in a nutshell ETICS stands for e-Infrastructure for Testing, Integration and Configuration of SoftwareETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software Project started on January 1 st, 2006 and has a duration of two yearsProject started on January 1 st, 2006 and has a duration of two years Goal is to provide efficient automated tools for building and testingGoal is to provide efficient automated tools for building and testing ETICS is born from the experience of developing, integrating and testing grid and distributed softwareETICS is born from the experience of developing, integrating and testing grid and distributed software

5. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 5 Usecases

6. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 6 Basic build usecase Context: A user develops, for example, a Grid service (as a single component)Context: A user develops, for example, a Grid service (as a single component) This service relies on existing software (dependencies)This service relies on existing software (dependencies) User performs following actionsUser performs following actions –For each required dependencies –Find documentation –Identify correct version to use (release, maintenance, beta, etc) –Identify if a binary package is adequate –In case building is required (very often the case), find build instructions –Build code –Install and configure code (using available documentation) –Verify that the code is built, installed and configured correctly –Find developer contact information in case of problems –Inter-dependencies must be understood and version clashes resolved –Write a script to build all dependencies –Write a script to build the current component (Grid service), including fetching code, build and test instructions/command –Store the component in a repository for others to use This above process has to be executed, in full or in part, for each targeted platformsThis above process has to be executed, in full or in part, for each targeted platforms For each platform, a machine has to be setup and maintainedFor each platform, a machine has to be setup and maintained If regular builds and tests are required, a scheduled task must be setup for each required platformIf regular builds and tests are required, a scheduled task must be setup for each required platform Each version of the software must be advertised to potential user communitiesEach version of the software must be advertised to potential user communities

7. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 7 What does this usecase tells us? The above usecase can take between a few hours to a few weeks, depending on the number of dependenciesThe above usecase can take between a few hours to a few weeks, depending on the number of dependencies When complexity grows, so does the time required to get startedWhen complexity grows, so does the time required to get started Different software teams will define the required information differently, which takes longer for the developers to integrate their workDifferent software teams will define the required information differently, which takes longer for the developers to integrate their work Unless well controlled, builds and tests might not be reproducibleUnless well controlled, builds and tests might not be reproducible Significant cost in maintaining different machines for build and testSignificant cost in maintaining different machines for build and test

8. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 8 Same Usecase using ETICS Context: A user develops, for example, a Grid service (as a single component)Context: A user develops, for example, a Grid service (as a single component) This service relies on existing software (dependencies)This service relies on existing software (dependencies) Using ETICS, user:Using ETICS, user: –Enters the commands to checkout, build and test the component –Selects the dependencies this component requires (from the rich catalog of existing components in ETICS) –Builds and tests the component locally –Submit build and test for parallel execution on all required platforms –All build and test artefacts are registered in the ETICS repository for public access

9. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 9 Service

10. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 10 ETICS Features Out-of-the-box automatic build/test systemOut-of-the-box automatic build/test system Multiplatform support for development and testingMultiplatform support for development and testing Powerful and flexible automatic dependency management (internal and external dependencies)Powerful and flexible automatic dependency management (internal and external dependencies) Trigger coding convention checks, unit test, documentation tools and collect and publish the resultsTrigger coding convention checks, unit test, documentation tools and collect and publish the results Possible to build everything from source or use pre-built binary packagesPossible to build everything from source or use pre-built binary packages Designed to support several Version Control SystemsDesigned to support several Version Control Systems Possible to register artefacts on the repositoryPossible to register artefacts on the repository Publishes run-time configuration information such as environment variables, runtime dependencies, etc …Publishes run-time configuration information such as environment variables, runtime dependencies, etc … Produce different package formats (rpms, tar, msi, debs)Produce different package formats (rpms, tar, msi, debs) Produces rich build and test reportsProduces rich build and test reports Ability to run custom plugins during build and test procedureAbility to run custom plugins during build and test procedure

11. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 11 The ETICS partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT The Condor batch system, distributed testing tools, service infrastructure, NMI

12. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 12 Service architecture Build/Test Artefacts Web Application Report DB Project DB NMI Execution Engine Clients Web Service NMI Client Wrapper Via browser WNs ETICS Infrastructure

13. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 13 Service Architecture ETICS Service build as an N-tier architectureETICS Service build as an N-tier architecture All elements of ETICS are configured, built and tested with ETICSAll elements of ETICS are configured, built and tested with ETICS ETICS Service is Open Source: Apache 2.0 licenseETICS Service is Open Source: Apache 2.0 license

14. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 14 Data model - 1 Project: container for components and subsystemsProject: container for components and subsystems Subsystem: container for componentsSubsystem: container for components Component: a module configuration can also be made of sub-module’s configurationsComponent: a module configuration can also be made of sub-module’s configurations Module: generic way to refer to project, subsystem, componentModule: generic way to refer to project, subsystem, component Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms)Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms) P S C Conf

15. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 15 Data model - 2 Platform : represents an specific combination of architecture, operating system and C/C++ compilerPlatform : represents an specific combination of architecture, operating system and C/C++ compiler –E.g. slc3_ia32_gcc323 Allows users to specify platform specific metadataAllows users to specify platform specific metadata –VCS commands (e.g. CVS) –Build commands –Test commands –Properties –Environment variables –Dependencies Conf Plat P S C

16. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 16 Getting started

17. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 17 ETICS registration https://etics.cern.ch:8443/eticsAdmin/ /public/registration/requestRegistration.jsp

18. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 18 ETICS Web Application Build/Test Artefacts Web Application Report DB Project DB NMI Execution Engine Clients Web Service NMI Client Wrapper Via browser WNs ETICS Infrastructure

19. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 19 Web Applications ETICS composed of following Web Applications:ETICS composed of following Web Applications: –Build and Test (WA) –Repository –Administration The applications are being integrated under a single user experienceThe applications are being integrated under a single user experience

20. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 20 WA – Description (1) Build and Test Web Application (WA) allows users toBuild and Test Web Application (WA) allows users to –Browse metadata –Edit metadata –Submit build and test jobs –Compatible with Firefox browser State-full and secure applicationState-full and secure application WA written in Java, jsp, javascript and JDuck (Engineering client/server web framework)WA written in Java, jsp, javascript and JDuck (Engineering client/server web framework) Interfaces with the ETICS Build and Test Web ServiceInterfaces with the ETICS Build and Test Web Service

21. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 21 WA - Description (2) Web Application is the simplest way to understand ETICSWeb Application is the simplest way to understand ETICS We will now go through a few simple steps to explain how ETICS works using the Build and Test Web ApplicationWe will now go through a few simple steps to explain how ETICS works using the Build and Test Web Application Note: Much more details will be provided during the demoNote: Much more details will be provided during the demo

22. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 22 WA – Project Metadata (3) All ETICS registered projects are available from a central entry point: https://etics.cern.ch:8443/eticsAll ETICS registered projects are available from a central entry point: https://etics.cern.ch:8443/eticshttps://etics.cern.ch:8443/etics Project details are shown upon selection in a drop- down listProject details are shown upon selection in a drop- down list Project Details

23. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 23 WA - Project Metadata (4) Browse Panel Configuration List View/Edit Panel

24. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 24 WA- Configuration Metadata (5) ETICS defines configuration similar to version in CVSETICS defines configuration similar to version in CVS Configurations can be defined differently for different platformsConfigurations can be defined differently for different platforms Configurations include what to do to checkout, build and test software modules:Configurations include what to do to checkout, build and test software modules: –Version control system (VCS) - e.g. CVS, SVN - commands –Build commands –Test commands –Properties and environment –Dependencies Properties and environment variables are inherited and propagatedProperties and environment variables are inherited and propagated Each command includes clear set targets. e.g. build: clear, init, checkstyle, doc, compile, package, publishEach command includes clear set targets. e.g. build: clear, init, checkstyle, doc, compile, package, publish

25. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 25 WA - Configuration Metadata (6) Here are a few examplesHere are a few examples Version Control System Commands:Version Control System Commands: –Checkout: cvs –d ${vcsroot} co ${moduleName} Build Commands:Build Commands: –Init: mkdir –p etc; cp ${security.location}/etc –Configure:./configure –Compile: ant –f build.xml Test Commands:Test Commands: –Init:./scripts/install-script ${target-hostname} –Test:./scripts/test-script ${reportDir}

26. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 26 WA - Remote build (7)

27. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 27 ETICS Reports

28. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 28 ETICS Repository – (1)

29. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 29 ETICS Repository – (2) TreesTrees –AJAX Technology –Easy browsing trees –Organised by Project Detailed description of filesDetailed description of files –Project, Package, Version and Platform –Type, Size and Last Modified –Download Link http://etics-test.cern.ch:8080/repositoryBrowser/

30. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 30 ETICS Administration Tool – (1)

31. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 31 ETICS Administration Tool – (2)

32. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 32 Command-line Client Build/Test Artefacts Web Application Report DB Project DB NMI Execution Engine Clients Web Service NMI Client Wrapper Via browser WNs ETICS Infrastructure

33. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 33 Command-line Client – (1) Build and Test Command-Line client (CLI) allows users toBuild and Test Command-Line client (CLI) allows users to –Build and test locally –Browse metadata –Edit metadata –Submit build and test jobs State-full and secure applicationState-full and secure application CLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisationCLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisation Interfaces with the ETICS Build and Test Web ServiceInterfaces with the ETICS Build and Test Web Service

34. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 34 Command-line Client Local build – (2) Download project metadata: etics-get-project Download project metadata: etics-get-project Action: download project metadata Checkout and/or download software:Checkout and/or download software: etics-checkout –c Action: download binary packages or fetch code from source Build the software: etics-build Action : build and unit test code hieratically, and generate packages and reportsBuild the software: etics-build Action : build and unit test code hieratically, and generate packages and reports Test the software: etics-test Action : test code hieratically, and generate packages and reportsTest the software: etics-test Action : test code hieratically, and generate packages and reports

35. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 35 Command-line Client Editing - (3) Command-line Client can also be used for editing: etics-module [ ] [ ] etics-configuration [ ] [ Command-line Client can also be used for editing: etics-module [ ] [ ] etics-configuration [ ] [ Can be used for scripting or interactive sessionsCan be used for scripting or interactive sessions Provides the same features as the Web ApplicationProvides the same features as the Web Application

36. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 36 Command-line Client Client Plugin-Framework – (4) Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investmentFramework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment Users can select, from a rich set, which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visibleUsers can select, from a rich set, which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible Promote inclusion of contributions from ETICS user community (more in this in dissemination presentation)Promote inclusion of contributions from ETICS user community (more in this in dissemination presentation) Examples of already available plugins:Examples of already available plugins: –IPv6 compliance tests –xUnit tests automatic execution –Multi-language line of code count (SLOC)

37. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 37 Build and Test Web Service Build/Test Artefacts Web Application Report DB Project DB NMI Execution Engine Clients Web Service NMI Client Wrapper Via browser WNs ETICS Infrastructure

38. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 38 WS - (1) Build and Test Web Service (WS) provides single secure interface to the ETICS backend systemBuild and Test Web Service (WS) provides single secure interface to the ETICS backend system –Metadata (persisted in database) –Execution engine –Security (role based) Heart of the ETICS System, it performs complex queries on behalf of the clients (WA and CLI)Heart of the ETICS System, it performs complex queries on behalf of the clients (WA and CLI)

39. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 39 WS – Security – (2) All accesses to the WS must be performed using a secure portAll accesses to the WS must be performed using a secure port Authentication is performed using digital certificates (x.509)Authentication is performed using digital certificates (x.509) Authorisation is role-basedAuthorisation is role-based Clients connecting without certificate are mapped to Guest, with minimal privileges (most likely just read access)Clients connecting without certificate are mapped to Guest, with minimal privileges (most likely just read access) Service can be setup to leverage certification authority management efforts such as EUGridPMAService can be setup to leverage certification authority management efforts such as EUGridPMA

40. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 40 Execution Engine Build/Test Artefacts Web Application Report DB Project DB NMI Execution Engine Clients Web Service NMI Client Wrapper Via browser WNs ETICS Infrastructure

41. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 41 NMI - Execution engine – (1) NMI is an execution engine internally used by ETICS to manage remote build and test jobs on different platformsNMI is an execution engine internally used by ETICS to manage remote build and test jobs on different platforms NMI is a layer built on the top of CondorNMI is a layer built on the top of Condor Condor is a middleware distribution that “provides mechanisms and policies supporting High Throughput Computing (HTC) on large collections of distributed computing resources”Condor is a middleware distribution that “provides mechanisms and policies supporting High Throughput Computing (HTC) on large collections of distributed computing resources” Both NMI and Condor are developed and supported by our partner, the University of Wisconsin in Madison, USABoth NMI and Condor are developed and supported by our partner, the University of Wisconsin in Madison, USA

42. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 42 http://etics.cern.ch/nmi NMI - Web interface – (2)

43. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 43 NMI- Additional features – (3) Following features were added to NMI/Condor to better support the ETICS Execution EngineFollowing features were added to NMI/Condor to better support the ETICS Execution Engine –Private resources: projects can provide specific resources for exclusive usage –Freeze: lock the node for further analysis in case of failure –Root-enabled environment: build or test jobs can request to run as root Users can register private machines if special setup and/or specific licenses are requiredUsers can register private machines if special setup and/or specific licenses are required

44. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 44 NMI- Using the ETICS client – (4) For users to trust the system, local and remote builds have to be executed in as close a condition as possibleFor users to trust the system, local and remote builds have to be executed in as close a condition as possible Once a build or test job has landed on a Condor worker-node (WN), the ETICS client takes overOnce a build or test job has landed on a Condor worker-node (WN), the ETICS client takes over The exact same sequence is executed on the WN, as if on a local machineThe exact same sequence is executed on the WN, as if on a local machine –Setup –Get project –Checkout –Build/Test The difference is that the WN is controlled and maintained by the ETICS experts and guaranteed to be a clean standard installationThe difference is that the WN is controlled and maintained by the ETICS experts and guaranteed to be a clean standard installation This approach guarantees that over time, builds and tests will be reproducible, a important tenet in software quality standardsThis approach guarantees that over time, builds and tests will be reproducible, a important tenet in software quality standards

45. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 45 NMI - ETICS sites – (5) Server at CERN etics.cern.chServer at CERN etics.cern.ch –SLC3, SLC4, RHES, Debian, FC (32- and 64-bits), WinXP, Mac OSX Server at Univ. of Wisconsin, Madison, USAServer at Univ. of Wisconsin, Madison, USA –HP HPUX, IBM AIX, OSF1, SGI Irix, Sun Solaris, Linux (several flavours) Server at INFN, Bologna, ItalyServer at INFN, Bologna, Italy –CentOS, SLC3, SLC4 (32- and 64-bits)

46. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 46 ETICS Reports – (1)

47. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 47 ETICS Reports – (2)

48. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 48 ETICS Reports – (3)

49. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 49 ETICS Reports – (4) TreesTrees –Easy browsing trees –Organised by Project or by Build Date Detailed reportsDetailed reports –Package Summary –Build Summary –Build Configuration –Build Logs http://etics.cern.ch:8080/reportBrowser/

50. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 50 ETICS Repository – (1)

51. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 51 Conclusions ETICS defined, integrated and delivered a novel Configuration, Build and Test ServiceETICS defined, integrated and delivered a novel Configuration, Build and Test Service The ETICS service was used in an incremental fashion to build and test the ETICS service, providing rapid feedback to the teamThe ETICS service was used in an incremental fashion to build and test the ETICS service, providing rapid feedback to the team Currently 20 projects registered and 74 usersCurrently 20 projects registered and 74 users Always in contact with early adoptersAlways in contact with early adopters –Feedback really useful Future functionality for next year will includeFuture functionality for next year will include –Handle complex testing scenarios (automatic deployment and test) –Virtual machines If you need further information do not hesitate to contact usIf you need further information do not hesitate to contact us etics-discuss@cern.ch etics-support@cern.ch

52. INFSOM-RI-026753 EGEE Bioinformatics meeting. UPV, Valencia, Spain 6-7 March 2007 52 Thanks http://www.eu-etics.org