1. www.eu-etics.org INFSOM-RI-026753 Training: The ETICS Service The ETICS Build Process Alberto Di Meglio ETICS Project

2. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 2 Building Software with ETICS ETICS can build your code and create distribution packagesETICS can build your code and create distribution packages It provides great flexibility and allows implementing different types of scenarios, from almost completely automated to almost completely user-definedIt provides great flexibility and allows implementing different types of scenarios, from almost completely automated to almost completely user-defined However the system relies on the information you register as a user and on a good understanding of the ETICS data modelHowever the system relies on the information you register as a user and on a good understanding of the ETICS data model It consists essentially of three main steps:It consists essentially of three main steps: CHECKOUT  BUILD  PACKAGING  PUBLISHING

3. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 3 Workspace Preparation

4. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 4 Client installation The ETICS client can be installed with two simple commands:The ETICS client can be installed with two simple commands: wget http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py [-O etics-client-setup] http://eticssoft.web.cern.ch/eticssoft/repository/etics-client-setup.py python etics-client-setup[.py] [-r x.y.z-r] [--volatile= ] [--force] export ETICS_HOME= export ETICS_HOME= export PATH=$ETICS_HOME/bin:$PATH

5. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 5 Workspace creation A workspace is a private area where a build/test can be executedA workspace is a private area where a build/test can be executed You can have as many workspaces as you wantYou can have as many workspaces as you want They can share the local repository cacheThey can share the local repository cache mkdir ws cd ws etics-workspace-setup

6. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 6 Get a project To work on a project, you need first to download its definitionTo work on a project, you need first to download its definition etics-get-project etics-get-project To list the existing project use a list commandTo list the existing project use a list commandetics-list-project

7. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 7 CHECKOUT

8. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 8 The etics-checkout command Before you can build and test a configuration, you need to check it outBefore you can build and test a configuration, you need to check it out The ETICS command to check out code or binary packages:The ETICS command to check out code or binary packages: etics-checkout [options] [-c configuration] [module] There are many possible options, the most used are usually: etics-checkout [options] [-c configuration] [module] There are many possible options, the most used are usually:--project-config=<configuration>--frombinary/--fromsource--ignorelocking--continueonerror

9. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 9 BUILD

10. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 10 The etics-build command The ETICS command to perform builds is called etics- build. The syntax of the command is:The ETICS command to perform builds is called etics- build. The syntax of the command is: etics-build [options] [module] It requires that the configurations to build have been previously checked out with the etics-checkout command etics-build [options] [module] It requires that the configurations to build have been previously checked out with the etics-checkout command

11. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 11 Command options (1) OptionDescription -h, --helpShow the usage instructions -c, --config Define a specific configuration to be used instead of the default one, where default is the first configuration found in the store. This is normally not required unless multiple or non- default configurations of the selected module have been checked out -p, --property This option allows passing properties to the build process. If the property is already defined, its value is overridden by the value specified in the command-line. To pass multiple properties, use multiple –p options -e, --env This option allows passing environment variables to the build process. If the variable is already defined, its value is overridden by the value specified in the command-line. To pass multiple variables, use multiple –e options -t, --target Execute the build stopping at the specified target, If not specified the publish target is executed. Allowed targets are: clean, init, checkstyle, compile, test, doc, packager, publish, install --platform Overwrite the local platform (useful for testing or when the local platform is not a valid ETICS platform, but it’s compatible with one).

12. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 12 Command options (2) OptionDescription --nobuildDo not perform the build, just print the sequence --continueonerrorDo not stop building if an error is found --nodepsOnly build the currently specified module (do not build children and dependencies) --forceForce the build of unmodified modules --verbosePrint verbose messages --versionReturn the current client version number.

13. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 13 Basic build scenario The standard command to build a configuration is:The standard command to build a configuration is: etics-build etics-build This command builds the current configuration of the specified module.This command builds the current configuration of the specified module. When run by itself, the command builds the configuration of the current project:When run by itself, the command builds the configuration of the current project: etics-build  etics-build etics-build  etics-build

14. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 14 Build Targets Command nameDescriptionMandatory cleanCommand to clean initCommand to perform initialization operations before compiling No checkstyleCommand to perform code checking operations (coding conventions) No compileCommand to compile codeNo testCommand to perform static tests like unit tests or coverage tests docCommand to generate documentationNo packagingCommand to generate distribution packagesNo publishCommand to publish build artifacts to standard distribution formats No installCommand to install the packageNo (but no packaging can be done if this is not defined)

15. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 15 How Targets are Evaluated The etics-build command executes the target in the following order:The etics-build command executes the target in the following order: init  configure  checkstyle  compile  test  doc  packaging  prepublish  publish  postpublish The cycle stops after the execution of the target specified with the –t option or after postpublish if –t is not used.The cycle stops after the execution of the target specified with the –t option or after postpublish if –t is not used. The clean and install target are executed separately using the –t option.The clean and install target are executed separately using the –t option. If a target command is not defined, it is normally simply skipped (but see later for packaging and publish).If a target command is not defined, it is normally simply skipped (but see later for packaging and publish).

16. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 16 The Stage Target ETICS defines an internal target called ‘stage’ETICS defines an internal target called ‘stage’ This target uses the module ‘install’ target to install the module in a common stage areaThis target uses the module ‘install’ target to install the module in a common stage area The stage area can be used by other modules to locate interface files and libraries during the build processThe stage area can be used by other modules to locate interface files and libraries during the build process The ‘stage’ location is available to your build commands using the built-in ${stageDir} property that can be passed as a parameter or used to define environment variables. Ex if mod2 relies on MOD1_HOME, just attach the following env var to mod1, so that it will be available to mod2The ‘stage’ location is available to your build commands using the built-in ${stageDir} property that can be passed as a parameter or used to define environment variables. Ex if mod2 relies on MOD1_HOME, just attach the following env var to mod1, so that it will be available to mod2 MOD1_HOME = ${stageDir}

17. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 17 PACKAGING

18. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 18 The ETICS Packaging System The ETICS Client comes with a built-in packaging system that allows to build distribution packages on all supported platforms in different formats (tarballs, RPMS, debs, MSIs, etc)The ETICS Client comes with a built-in packaging system that allows to build distribution packages on all supported platforms in different formats (tarballs, RPMS, debs, MSIs, etc) In order to use the ETICS Packager the install target has to be defined, since it is used internally to identify what the content of the package should be.In order to use the ETICS Packager the install target has to be defined, since it is used internally to identify what the content of the package should be. In order to activate the ETICS Packager, leave the packaging target undefinedIn order to activate the ETICS Packager, leave the packaging target undefined

19. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 19 When You Should Use the EPS You have typical packaging needsYou have typical packaging needs You have platform-dependent packagesYou have platform-dependent packages Once the metadata is defined it can be used to produce different kinds of packages of various platformsOnce the metadata is defined it can be used to produce different kinds of packages of various platforms It allows to enforce consistent conventions for all packages across a projectIt allows to enforce consistent conventions for all packages across a project

20. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 20 When You Should Not Use the EPS You have very specific packaging needsYou have very specific packaging needs Your modules create more than one packageYour modules create more than one package Multi-platform compatibility is not an issueMulti-platform compatibility is not an issue There is also an intermediate solutionThere is also an intermediate solution –Define your own spec file or rules file –Let ETICS use your definitions to build the packages

21. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 21 Naming conventions Binary tarballs: -x.y.x-r.tar.gzBinary tarballs: -x.y.x-r.tar.gz Source tarballs: -x.y.x-r.src.tar.gzSource tarballs: -x.y.x-r.src.tar.gz Binary RPMS: -x.y.x-r...rpmBinary RPMS: -x.y.x-r...rpm Binary RPMS: -x.y.x-r.src.rpmBinary RPMS: -x.y.x-r.src.rpm Distribution is a standard OS distribution name like slc3, rhel4, fc4Distribution is a standard OS distribution name like slc3, rhel4, fc4 Architecture is a cpu architecture name like i386, i686, x86_64, noarch, etc.Architecture is a cpu architecture name like i386, i686, x86_64, noarch, etc.

22. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 22 Packaging Options OptionDescription --createsourceCreate source tarball and RPMS in addition to binary ones (this slows down the build due to the extra compilation steps --createdebugCreate debug RPMS in addition to binary ones (this slows down the build due to the extra compilation steps [1] [1] --nodistnameDo not append the distribution name to the RPM revision (it has no effect if --userspec is used) --usetimestampAppend a timestamp to the RPM revision number (it has no effect if --userspec is used) --versioneddepsUse version information when setting package dependencies in RPMS (ex: etics-client >= 1.0.0). Cannot be used together with –strictversioneddeps --strictversioneddepsUse strict version information when setting package dependencies in RPMS (ex: etics-client = 1.0.0). Cannot be used together with –versioneddeps --userspecUse a user-defined spec file. The spec file must be placed in /project. If this option is used all other packaging options are ignored [1] [1] Not supported in the current release

23. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 23 Packaging Properties Property nameDefault value if not specifiedTypeDescription package.prefix/opt/module-namestringThe default installation prefix for the generated RPMS package.buildarchLocal platform/distribution specifier (slc3, rhel4, etc) stringIt can be used to override the default specifier. It is necessary to set it to ‘noarch’ to generate noarch RPMS package.providesAutodetected if autoreqprov is set to yes (default) stringThe list of provides entries for the RPM spec file (overrides the autodetected list) package.requiresAutodetected if autoreqprov is set to yes (default) stringThe list of requires entries for the RPM spec file (overrides the autodetected list) package.obsoletesstringThe list of obsoletes entries for the RPM spec file package.conflictsstringThe list of conflicts entries for the RPM spec file

24. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 24 Packaging Properties Property nameDefault value if not specifiedTypeDescription package.autoreqprovyesstringCan be ‘yes’ or ‘no’. If it is set to yes, rpm will try to automatically detect requires and provides from the packaged files and libs, otherwise will only use specified entries Additional properties are available to set.pre/.post scripts and other various spec files optionsAdditional properties are available to set.pre/.post scripts and other various spec files options

25. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 25 How to Tell the Packager What to Package The packager relies on two elements from your metadataThe packager relies on two elements from your metadata –The install target –The installation prefix There are various conventions about how to pass this parameter depending on which installation method is used (autotools, ant, python distutils, etc)There are various conventions about how to pass this parameter depending on which installation method is used (autotools, ant, python distutils, etc) ETICS defines the following generic conventionETICS defines the following generic convention –whatever method is used to pass the prefix, the value of the prefix must be set in the commands as ${prefix}. The ETICS Packager will then resolve this value to an internal location of its own –Note that the same method is used to stage the files in the stage area

26. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 26 Installation Instructions Examples AutotoolsAutotools –init =./configure - -prefix=${prefix} –compile = make –install = make install python distutilspython distutils –compile = python setup.py build –install = python setup.py - -prefix=${prefix} Custom install scriptCustom install script –install =./my_installation_script - -install_location=${prefix}

27. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 27 PUBLISHING

28. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 28 The ETICS Publisher The ETICS Client comes with pre-defined publishing conventions that allow harvesting build artifacts (packages, logs, etc) from the various modules into a common location. The ETICS Publisher uses the following conventions:The ETICS Client comes with pre-defined publishing conventions that allow harvesting build artifacts (packages, logs, etc) from the various modules into a common location. The ETICS Publisher uses the following conventions: –All checkout and build logs are stored in the reports directory of the workspace. The main log file is called: build-status.xml and contains summary checkout and build information for each module in the current build run. Detailed logs of the checkout and build operations of each modules are saved in files of the format: [CHECKOUT | BUILD]- -.log –All packages are stored in the directory dist

29. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 29 Packages Locations The packages in dist are organized in a tree that mirrors the structure of the ETICS software repository: / / / / The packages in dist are organized in a tree that mirrors the structure of the ETICS software repository: / / / / The packages are taken by default from the following locations in each module root:The packages are taken by default from the following locations in each module root: –(S)RPMS: /RPMS –(src.)tar.gz: /tgz

30. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 30 Packaging Properties used by the Publisher Property nameDefault value if not specifiedTypeDescription package.tgzLocation${location}/tgzstringThe default location where generated tarballs are stored in each module at the end of a build and where the Publisher looks for them package.SRPMSLocation${location}/RPMSstringThe default location where generated source RPMS are stored in each module at the end of a build and where the Publisher looks for them package.RPMSLocation${location}/RPMSstringThe default location where generated binary RPMS are stored in each module at the end of a build and where the Publisher looks for them

31. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 31 The Remote Build Service Build/Test Artefacts Web Application Report DB Project DB Metronome Engine Clients Web Service Metronome Client Wrapper Via browser WNs ETICS Infrastructure Repository Service

32. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 32 Submitting Remote Builds Builds are submitted to the ETICS Service using the command etics-submit build|test [options] [module]Builds are submitted to the ETICS Service using the command etics-submit build|test [options] [module] The options can be categorized in two groups:The options can be categorized in two groups: –Checkout options: options passed to the remote checkout command –Build/test options: options passed to the remote build/test command

33. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 33 Remote Build Options Property nameDefault value if not specified TypeDescription --platformsCurrent local platformComma separated list of platform strings (do not use blanks or quote the entire string) The list of platforms where to build --projectCurrent project if definedDefine the name of the project to get for the build --project-configDefine a specific project configuration for the metadata --configDefine a specific configuration for the requested module. If this is missing the HEAD configuration is used

34. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 34 Remote Build Options Property nameDefault value if not specified TypeDescription --fromsourceWhen possible, check out source code instead of downloading binaries. This implies that the packages will not be built. --frombinaryWhen possible, download binaries instead of checking out source code. This implies that packages will be built (this is the default if not option is specified). --romsourceonlyCheck out source code only. This means that all the configurations, including dependencies have to be available in source code form or the operation will fail. --frombinaryonlyCheckout binaries only. This means that all the configurations, including dependencies have to be available in binary form, except the modules specified to the command.

35. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 35 Remote Build Options Property nameDefault value if not specified TypeDescription --registerAsk to register the artifacts in the permanent repository (only applies to locked configurations). --register-volatile= Ask to register the artifacts in a volatile repository. If not specified, the artifacts goes automatically in the default volatile repository In particular note the following requirement:In particular note the following requirement:--register used to upload build artifacts into the repository after the build

36. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 36 ETICS Production Setup etics.cern.chetics.cern.ch –Official ETICS submission node - production host –2250+ jobs (as of 22 Sept 2006) etics-01.cnaf.infn.itetics-01.cnaf.infn.it –200+ (as of 22 Sept 2006) “Grand Central” at University of Wisconsin“Grand Central” at University of Wisconsin –Hundred Thousands jobs used by several projects Future job migration between testbeds to cover the largest possible platform spectrum and increase resource availabilityFuture job migration between testbeds to cover the largest possible platform spectrum and increase resource availability

37. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 37 Currently Supported platforms 64 bits64 bits – slc3_ia64_gcc323 (ia64_slc_3) – slc3_x86_64_gcc323 (x86_64_slc_3) – slc4_x86_64_gcc346 (x86_64_slc_4) – macos10_ppc_gcc33 (ppc_macos_10.4) 32 bits32 bits – win32_vc71 (x86_winnt_5.1) – slc3_ia32_gcc323 (x86_slc_3) – slc4_ia32_gcc346 (x86_slc_4) – sl5_ia32_gcc411 (x86_sl_5) – fc3_ia32_gcc345 (x86_fc_3) – fc4_ia32_gcc345 (x86_fc_4) – rhel3_ia32_gcc323 (x86_rhes_3) – rhel4_ia32_gcc346 (x86_rhes_4) – deb4_ia32_gcc412 – centos4_ia32_gcc346 (x86_cent_4.3) – etc

38. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 38

39. INFSOM-RI-026753 ETICS 2 Induction Training - FZJ - 15-16 April 2008 39