INFSOM-RI Training: The ETICS Service The ETICS Build Process Alberto Di Meglio ETICS Project

INFSOM-RI ETICS 2 Induction Training - FZJ April Building Software with ETICS ETICS can build your code and create distribution packagesETICS can build your code and create distribution packages It provides great flexibility and allows implementing different types of scenarios, from almost completely automated to almost completely user-definedIt provides great flexibility and allows implementing different types of scenarios, from almost completely automated to almost completely user-defined However the system relies on the information you register as a user and on a good understanding of the ETICS data modelHowever the system relies on the information you register as a user and on a good understanding of the ETICS data model It consists essentially of three main steps:It consists essentially of three main steps: CHECKOUT  BUILD  PACKAGING  PUBLISHING

INFSOM-RI ETICS 2 Induction Training - FZJ April Workspace Preparation

INFSOM-RI ETICS 2 Induction Training - FZJ April Client installation The ETICS client can be installed with two simple commands:The ETICS client can be installed with two simple commands: wget [-O etics-client-setup] python etics-client-setup[.py] [-r x.y.z-r] [--volatile= ] [--force] export ETICS_HOME= export ETICS_HOME= export PATH=$ETICS_HOME/bin:$PATH

INFSOM-RI ETICS 2 Induction Training - FZJ April Workspace creation A workspace is a private area where a build/test can be executedA workspace is a private area where a build/test can be executed You can have as many workspaces as you wantYou can have as many workspaces as you want They can share the local repository cacheThey can share the local repository cache mkdir ws cd ws etics-workspace-setup

INFSOM-RI ETICS 2 Induction Training - FZJ April Get a project To work on a project, you need first to download its definitionTo work on a project, you need first to download its definition etics-get-project etics-get-project To list the existing project use a list commandTo list the existing project use a list commandetics-list-project

INFSOM-RI ETICS 2 Induction Training - FZJ April CHECKOUT

INFSOM-RI ETICS 2 Induction Training - FZJ April The etics-checkout command Before you can build and test a configuration, you need to check it outBefore you can build and test a configuration, you need to check it out The ETICS command to check out code or binary packages:The ETICS command to check out code or binary packages: etics-checkout [options] [-c configuration] [module] There are many possible options, the most used are usually: etics-checkout [options] [-c configuration] [module] There are many possible options, the most used are usually:--project-config=<configuration>--frombinary/--fromsource--ignorelocking--continueonerror

INFSOM-RI ETICS 2 Induction Training - FZJ April BUILD

INFSOM-RI ETICS 2 Induction Training - FZJ April The etics-build command The ETICS command to perform builds is called etics- build. The syntax of the command is:The ETICS command to perform builds is called etics- build. The syntax of the command is: etics-build [options] [module] It requires that the configurations to build have been previously checked out with the etics-checkout command etics-build [options] [module] It requires that the configurations to build have been previously checked out with the etics-checkout command

INFSOM-RI ETICS 2 Induction Training - FZJ April Command options (1) OptionDescription -h, --helpShow the usage instructions -c, --config Define a specific configuration to be used instead of the default one, where default is the first configuration found in the store. This is normally not required unless multiple or non- default configurations of the selected module have been checked out -p, --property This option allows passing properties to the build process. If the property is already defined, its value is overridden by the value specified in the command-line. To pass multiple properties, use multiple –p options -e, --env This option allows passing environment variables to the build process. If the variable is already defined, its value is overridden by the value specified in the command-line. To pass multiple variables, use multiple –e options -t, --target Execute the build stopping at the specified target, If not specified the publish target is executed. Allowed targets are: clean, init, checkstyle, compile, test, doc, packager, publish, install --platform Overwrite the local platform (useful for testing or when the local platform is not a valid ETICS platform, but it’s compatible with one).

INFSOM-RI ETICS 2 Induction Training - FZJ April Command options (2) OptionDescription --nobuildDo not perform the build, just print the sequence --continueonerrorDo not stop building if an error is found --nodepsOnly build the currently specified module (do not build children and dependencies) --forceForce the build of unmodified modules --verbosePrint verbose messages --versionReturn the current client version number.

INFSOM-RI ETICS 2 Induction Training - FZJ April Basic build scenario The standard command to build a configuration is:The standard command to build a configuration is: etics-build etics-build This command builds the current configuration of the specified module.This command builds the current configuration of the specified module. When run by itself, the command builds the configuration of the current project:When run by itself, the command builds the configuration of the current project: etics-build  etics-build etics-build  etics-build

INFSOM-RI ETICS 2 Induction Training - FZJ April Build Targets Command nameDescriptionMandatory cleanCommand to clean initCommand to perform initialization operations before compiling No checkstyleCommand to perform code checking operations (coding conventions) No compileCommand to compile codeNo testCommand to perform static tests like unit tests or coverage tests docCommand to generate documentationNo packagingCommand to generate distribution packagesNo publishCommand to publish build artifacts to standard distribution formats No installCommand to install the packageNo (but no packaging can be done if this is not defined)

INFSOM-RI ETICS 2 Induction Training - FZJ April How Targets are Evaluated The etics-build command executes the target in the following order:The etics-build command executes the target in the following order: init  configure  checkstyle  compile  test  doc  packaging  prepublish  publish  postpublish The cycle stops after the execution of the target specified with the –t option or after postpublish if –t is not used.The cycle stops after the execution of the target specified with the –t option or after postpublish if –t is not used. The clean and install target are executed separately using the –t option.The clean and install target are executed separately using the –t option. If a target command is not defined, it is normally simply skipped (but see later for packaging and publish).If a target command is not defined, it is normally simply skipped (but see later for packaging and publish).

INFSOM-RI ETICS 2 Induction Training - FZJ April The Stage Target ETICS defines an internal target called ‘stage’ETICS defines an internal target called ‘stage’ This target uses the module ‘install’ target to install the module in a common stage areaThis target uses the module ‘install’ target to install the module in a common stage area The stage area can be used by other modules to locate interface files and libraries during the build processThe stage area can be used by other modules to locate interface files and libraries during the build process The ‘stage’ location is available to your build commands using the built-in ${stageDir} property that can be passed as a parameter or used to define environment variables. Ex if mod2 relies on MOD1_HOME, just attach the following env var to mod1, so that it will be available to mod2The ‘stage’ location is available to your build commands using the built-in ${stageDir} property that can be passed as a parameter or used to define environment variables. Ex if mod2 relies on MOD1_HOME, just attach the following env var to mod1, so that it will be available to mod2 MOD1_HOME = ${stageDir}

INFSOM-RI ETICS 2 Induction Training - FZJ April PACKAGING

INFSOM-RI ETICS 2 Induction Training - FZJ April The ETICS Packaging System The ETICS Client comes with a built-in packaging system that allows to build distribution packages on all supported platforms in different formats (tarballs, RPMS, debs, MSIs, etc)The ETICS Client comes with a built-in packaging system that allows to build distribution packages on all supported platforms in different formats (tarballs, RPMS, debs, MSIs, etc) In order to use the ETICS Packager the install target has to be defined, since it is used internally to identify what the content of the package should be.In order to use the ETICS Packager the install target has to be defined, since it is used internally to identify what the content of the package should be. In order to activate the ETICS Packager, leave the packaging target undefinedIn order to activate the ETICS Packager, leave the packaging target undefined

INFSOM-RI ETICS 2 Induction Training - FZJ April When You Should Use the EPS You have typical packaging needsYou have typical packaging needs You have platform-dependent packagesYou have platform-dependent packages Once the metadata is defined it can be used to produce different kinds of packages of various platformsOnce the metadata is defined it can be used to produce different kinds of packages of various platforms It allows to enforce consistent conventions for all packages across a projectIt allows to enforce consistent conventions for all packages across a project

INFSOM-RI ETICS 2 Induction Training - FZJ April When You Should Not Use the EPS You have very specific packaging needsYou have very specific packaging needs Your modules create more than one packageYour modules create more than one package Multi-platform compatibility is not an issueMulti-platform compatibility is not an issue There is also an intermediate solutionThere is also an intermediate solution –Define your own spec file or rules file –Let ETICS use your definitions to build the packages

INFSOM-RI ETICS 2 Induction Training - FZJ April Naming conventions Binary tarballs: -x.y.x-r.tar.gzBinary tarballs: -x.y.x-r.tar.gz Source tarballs: -x.y.x-r.src.tar.gzSource tarballs: -x.y.x-r.src.tar.gz Binary RPMS: -x.y.x-r...rpmBinary RPMS: -x.y.x-r...rpm Binary RPMS: -x.y.x-r.src.rpmBinary RPMS: -x.y.x-r.src.rpm Distribution is a standard OS distribution name like slc3, rhel4, fc4Distribution is a standard OS distribution name like slc3, rhel4, fc4 Architecture is a cpu architecture name like i386, i686, x86_64, noarch, etc.Architecture is a cpu architecture name like i386, i686, x86_64, noarch, etc.

INFSOM-RI ETICS 2 Induction Training - FZJ April Packaging Options OptionDescription --createsourceCreate source tarball and RPMS in addition to binary ones (this slows down the build due to the extra compilation steps --createdebugCreate debug RPMS in addition to binary ones (this slows down the build due to the extra compilation steps [1] [1] --nodistnameDo not append the distribution name to the RPM revision (it has no effect if --userspec is used) --usetimestampAppend a timestamp to the RPM revision number (it has no effect if --userspec is used) --versioneddepsUse version information when setting package dependencies in RPMS (ex: etics-client >= 1.0.0). Cannot be used together with –strictversioneddeps --strictversioneddepsUse strict version information when setting package dependencies in RPMS (ex: etics-client = 1.0.0). Cannot be used together with –versioneddeps --userspecUse a user-defined spec file. The spec file must be placed in /project. If this option is used all other packaging options are ignored [1] [1] Not supported in the current release

INFSOM-RI ETICS 2 Induction Training - FZJ April Packaging Properties Property nameDefault value if not specifiedTypeDescription package.prefix/opt/module-namestringThe default installation prefix for the generated RPMS package.buildarchLocal platform/distribution specifier (slc3, rhel4, etc) stringIt can be used to override the default specifier. It is necessary to set it to ‘noarch’ to generate noarch RPMS package.providesAutodetected if autoreqprov is set to yes (default) stringThe list of provides entries for the RPM spec file (overrides the autodetected list) package.requiresAutodetected if autoreqprov is set to yes (default) stringThe list of requires entries for the RPM spec file (overrides the autodetected list) package.obsoletesstringThe list of obsoletes entries for the RPM spec file package.conflictsstringThe list of conflicts entries for the RPM spec file

INFSOM-RI ETICS 2 Induction Training - FZJ April Packaging Properties Property nameDefault value if not specifiedTypeDescription package.autoreqprovyesstringCan be ‘yes’ or ‘no’. If it is set to yes, rpm will try to automatically detect requires and provides from the packaged files and libs, otherwise will only use specified entries Additional properties are available to set.pre/.post scripts and other various spec files optionsAdditional properties are available to set.pre/.post scripts and other various spec files options

INFSOM-RI ETICS 2 Induction Training - FZJ April How to Tell the Packager What to Package The packager relies on two elements from your metadataThe packager relies on two elements from your metadata –The install target –The installation prefix There are various conventions about how to pass this parameter depending on which installation method is used (autotools, ant, python distutils, etc)There are various conventions about how to pass this parameter depending on which installation method is used (autotools, ant, python distutils, etc) ETICS defines the following generic conventionETICS defines the following generic convention –whatever method is used to pass the prefix, the value of the prefix must be set in the commands as ${prefix}. The ETICS Packager will then resolve this value to an internal location of its own –Note that the same method is used to stage the files in the stage area

INFSOM-RI ETICS 2 Induction Training - FZJ April Installation Instructions Examples AutotoolsAutotools –init =./configure - -prefix=${prefix} –compile = make –install = make install python distutilspython distutils –compile = python setup.py build –install = python setup.py - -prefix=${prefix} Custom install scriptCustom install script –install =./my_installation_script - -install_location=${prefix}

INFSOM-RI ETICS 2 Induction Training - FZJ April PUBLISHING

INFSOM-RI ETICS 2 Induction Training - FZJ April The ETICS Publisher The ETICS Client comes with pre-defined publishing conventions that allow harvesting build artifacts (packages, logs, etc) from the various modules into a common location. The ETICS Publisher uses the following conventions:The ETICS Client comes with pre-defined publishing conventions that allow harvesting build artifacts (packages, logs, etc) from the various modules into a common location. The ETICS Publisher uses the following conventions: –All checkout and build logs are stored in the reports directory of the workspace. The main log file is called: build-status.xml and contains summary checkout and build information for each module in the current build run. Detailed logs of the checkout and build operations of each modules are saved in files of the format: [CHECKOUT | BUILD]- -.log –All packages are stored in the directory dist

INFSOM-RI ETICS 2 Induction Training - FZJ April Packages Locations The packages in dist are organized in a tree that mirrors the structure of the ETICS software repository: / / / / The packages in dist are organized in a tree that mirrors the structure of the ETICS software repository: / / / / The packages are taken by default from the following locations in each module root:The packages are taken by default from the following locations in each module root: –(S)RPMS: /RPMS –(src.)tar.gz: /tgz

INFSOM-RI ETICS 2 Induction Training - FZJ April Packaging Properties used by the Publisher Property nameDefault value if not specifiedTypeDescription package.tgzLocation${location}/tgzstringThe default location where generated tarballs are stored in each module at the end of a build and where the Publisher looks for them package.SRPMSLocation${location}/RPMSstringThe default location where generated source RPMS are stored in each module at the end of a build and where the Publisher looks for them package.RPMSLocation${location}/RPMSstringThe default location where generated binary RPMS are stored in each module at the end of a build and where the Publisher looks for them

INFSOM-RI ETICS 2 Induction Training - FZJ April The Remote Build Service Build/Test Artefacts Web Application Report DB Project DB Metronome Engine Clients Web Service Metronome Client Wrapper Via browser WNs ETICS Infrastructure Repository Service

INFSOM-RI ETICS 2 Induction Training - FZJ April Submitting Remote Builds Builds are submitted to the ETICS Service using the command etics-submit build|test [options] [module]Builds are submitted to the ETICS Service using the command etics-submit build|test [options] [module] The options can be categorized in two groups:The options can be categorized in two groups: –Checkout options: options passed to the remote checkout command –Build/test options: options passed to the remote build/test command

INFSOM-RI ETICS 2 Induction Training - FZJ April Remote Build Options Property nameDefault value if not specified TypeDescription --platformsCurrent local platformComma separated list of platform strings (do not use blanks or quote the entire string) The list of platforms where to build --projectCurrent project if definedDefine the name of the project to get for the build --project-configDefine a specific project configuration for the metadata --configDefine a specific configuration for the requested module. If this is missing the HEAD configuration is used

INFSOM-RI ETICS 2 Induction Training - FZJ April Remote Build Options Property nameDefault value if not specified TypeDescription --fromsourceWhen possible, check out source code instead of downloading binaries. This implies that the packages will not be built. --frombinaryWhen possible, download binaries instead of checking out source code. This implies that packages will be built (this is the default if not option is specified). --romsourceonlyCheck out source code only. This means that all the configurations, including dependencies have to be available in source code form or the operation will fail. --frombinaryonlyCheckout binaries only. This means that all the configurations, including dependencies have to be available in binary form, except the modules specified to the command.

INFSOM-RI ETICS 2 Induction Training - FZJ April Remote Build Options Property nameDefault value if not specified TypeDescription --registerAsk to register the artifacts in the permanent repository (only applies to locked configurations). --register-volatile= Ask to register the artifacts in a volatile repository. If not specified, the artifacts goes automatically in the default volatile repository In particular note the following requirement:In particular note the following requirement:--register used to upload build artifacts into the repository after the build

INFSOM-RI ETICS 2 Induction Training - FZJ April ETICS Production Setup etics.cern.chetics.cern.ch –Official ETICS submission node - production host –2250+ jobs (as of 22 Sept 2006) etics-01.cnaf.infn.itetics-01.cnaf.infn.it –200+ (as of 22 Sept 2006) “Grand Central” at University of Wisconsin“Grand Central” at University of Wisconsin –Hundred Thousands jobs used by several projects Future job migration between testbeds to cover the largest possible platform spectrum and increase resource availabilityFuture job migration between testbeds to cover the largest possible platform spectrum and increase resource availability

INFSOM-RI ETICS 2 Induction Training - FZJ April Currently Supported platforms 64 bits64 bits – slc3_ia64_gcc323 (ia64_slc_3) – slc3_x86_64_gcc323 (x86_64_slc_3) – slc4_x86_64_gcc346 (x86_64_slc_4) – macos10_ppc_gcc33 (ppc_macos_10.4) 32 bits32 bits – win32_vc71 (x86_winnt_5.1) – slc3_ia32_gcc323 (x86_slc_3) – slc4_ia32_gcc346 (x86_slc_4) – sl5_ia32_gcc411 (x86_sl_5) – fc3_ia32_gcc345 (x86_fc_3) – fc4_ia32_gcc345 (x86_fc_4) – rhel3_ia32_gcc323 (x86_rhes_3) – rhel4_ia32_gcc346 (x86_rhes_4) – deb4_ia32_gcc412 – centos4_ia32_gcc346 (x86_cent_4.3) – etc

INFSOM-RI ETICS 2 Induction Training - FZJ April

INFSOM-RI ETICS 2 Induction Training - FZJ April