B. Hegner, P. Mato, P. Mendez CERN, PH-SFT Group 1 ST FORUM CERN 28-SEP-2015 THE QUALITY AND TESTING INFRASTRUCTURE OF PH-SFT

Introduction The LCG build system Tools and core elements Test, nightly and release infrastructures Scheduler and Dashboard Summary and future plans 28/09/15Quality and Testing PH-SFT infrastructure 2 OUTLOOK

Development and maintenance of common scientific software for the experiments in collaboration with the PH and IT departments and external HEP institutes Areas covered by the group: Simulation: Geant4, GeantV Frameworks: ROOT Distributed systems: CERN VM, CERNVMFS 28/09/15Quality and Testing PH-SFT infrastructure 3 PH-SFT GROUP

 Goals of the project:  Provision of consistent releases of external and LCG application PKGs to both experimental and theoretical communities  Provide daily tests and previews of incoming releases to users 28/09/15Quality and Testing PH-SFT infrastructure 4 THE PH-SFT PACKAGES AND RELEASES PROJECT WITHIN THE LHC SOFTWARE STACK Event Experiments specific Packages External packages non-HEP specific Detector Calib Simul. Data Mngmt. Distrib. Analysis Core Libraries

28/09/15Quality and Testing PH-SFT infrastructure 5 OUR BUILD PACKAGES AND COMBINATIONS Externals (~140)MC Generators (~50)Group’s projects Python, GSL, Boost…Pythia Herwig Hepmcanalysis … ROOT GEANT4 About 10 Grid packages FTS, WMS, DPM… CompilersArchitecturesOperating systemsBuild types gcc4.8.4, gcc4.9.3, gcc5.1.0, gcc5.2.0 clang35, 37 Icc15 Native Intel arm64 SLC6 CentOS7 CentOS6 Ubuntu12, 14 Mac 10.8, 10.9, Release Debug Packages Combinations Packages and combinations agreed with experiments

1.Build results reproducibility 2.Manageable handling of package dependencies 3.End-systems independent installation procedures; reallocation 4.Proper quality assurance 5.Scalability of the infrastructure: increasing number of compilers, architectures 6.Fast improvement/feedback cycle Overall significant number of challenges common to many projects that could be commonly managed HEP Software Foundation (HSF) project  Promote and facilitate the creation, maintenance and lifecycle management of high quality common software of value to HEP and beyond  Forum for collaboration between software developers  Identification of community goals and deliverables and ease community contributions 28/09/15Quality and Testing PH-SFT infrastructure 6 THE PROJECT CHALLENGES

28/09/15Quality and Testing PH-SFT infrastructure 7 THE CORE ELEMENTS Toolset.cmake # Application Area Projects LCG_AA_project(COOL 3_0_4) LCG_AA_project(CORAL 3_0_4) LCG_AA_project(RELAX RELAX_1_4_2) LCG_AA_project(ROOT ) # Externals LCG_external_package(4suite 1.0.2p1 ) LCG_external_package(AIDA ) LCG_external_package(backports ) # Application Area Projects LCG_AA_project(COOL 3_0_4) LCG_AA_project(CORAL 3_0_4) LCG_AA_project(RELAX RELAX_1_4_2) LCG_AA_project(ROOT ) # Externals LCG_external_package(4suite 1.0.2p1 ) LCG_external_package(AIDA ) LCG_external_package(backports )

28/09/15Quality and Testing PH-SFT infrastructure 8 THE CORE ELEMENTS AFS Toolset.cmake LCGCMAKE Sources download Build instructions (Cmake, make) Build instructions (Cmake, make) Branch (release number, nightly) Architecture Compiler Build type Operating System GITLAB repository

28/09/15Quality and Testing PH-SFT infrastructure 9 THE CORE ELEMENTS AFS Toolset.cmake LCGCMAKE Packages download Machinery (Cmake, make) Machinery (Cmake, make) Packages/rpm repository Dependencies text (json) Installation scripts AFS/CVMFS

28/09/15Quality and Testing PH-SFT infrastructure 10 THE CORE ELEMENTS Toolset.cmake LCGCMAKE Packages download Machinery (Cmake, make) Machinery (Cmake, make) Packages/rpm repository Dependencies text (json) Installation scripts AFS/CVMFS Cdash publication Release summary web page

28/09/15Quality and Testing PH-SFT infrastructure 11 INFORMATION ACCESS FOR EXPERIMENTS AFS/CVMFS Cdash publication Experiments Release summary web page

 Build and testing infrastructure CMAKE based: Open-source system dedicated to build, test and install software Build specifications included in platform- independent list files Generation of CMakeFile that drives the full build Modular and scalable system Available in both AFS and CVMFS  SFT tool based on cmake: LCGCMAKE Based on “ExternalProject” CMAKE module implementation Enable the build of external software sources Easy way to control package dependencies Repository recently migrated to GITLAB 28/09/15Quality and Testing PH-SFT infrastructure 12 BUILDING FROM SOURCES

LCGPackage_Add( Python URL spi/external/tarFiles/Python-${Python_native_version}.tgz CONFIGURE_COMMAND /configure --prefix= -- enable-unicode=ucs4 ${Python_config_options} CPPFLAGS=- I${sqlite_home}/include INSTALL_COMMAND ${MAKE} -j1 install COMMAND ${CMAKE_SOURCE_DIR}/externals/Python_postinstall.sh DEPENDS ${Python_deps} sqlite ) LCGPackage_Add( Python URL spi/external/tarFiles/Python-${Python_native_version}.tgz CONFIGURE_COMMAND /configure --prefix= -- enable-unicode=ucs4 ${Python_config_options} CPPFLAGS=- I${sqlite_home}/include INSTALL_COMMAND ${MAKE} -j1 install COMMAND ${CMAKE_SOURCE_DIR}/externals/Python_postinstall.sh DEPENDS ${Python_deps} sqlite ) Modular file system based structure separated in blocks Few lines sufficient to describe the build steps per package Dependencies with other packages explicitly included 28/09/15Quality and Testing PH-SFT infrastructure 13 LCGCMAKE PACKAGE BUILD DESCRIPTION

 In big build systems cmake provides a full control of PKGs dependencies  Build Configuration outputs contains these dependencies 28/09/15Quality and Testing PH-SFT infrastructure 14 DEPENDENCIES CONTROL

Common installation tree in all builds for binaries / / _ / /… hash  Calculated from the list of package dependencies and the version Platform_tag  combination of architecture, OS, compiler version and build type: x86_64-slc6-gcc49-opt… RPM installation in both AFS and CVMFS areas Maintenance of all installed RPMs in AFS Management of installed RPMs through a specific DB Package relocation end-system independent 28/09/15Quality and Testing PH-SFT infrastructure 15 AFS/CVMFS DEPLOYMENT

1.Build results reproducibility 2.Manageable handling of package dependencies 3.End-systems independent installation procedures; reallocation 4.Proper quality assurance 5.Scalability of the infrastructure: increasing number of compilers, architectures 6.Fast improvement/feedback cycle 28/09/15Quality and Testing PH-SFT infrastructure 16 BACK TO SLIDE 6 - CHALLENGES

28/09/15Quality and Testing PH-SFT infrastructure 17 RELEASE PROCEDURES Experimental: (daily and on demand ) Experimental: (daily and on demand ) AFS CVMFS 3 Nightly builds (Daily) Dev2: ROOT 6.02 Dev3: ROOT MASTER Dev4: ROOT Nightly builds (Daily) Dev2: ROOT 6.02 Dev3: ROOT MASTER Dev4: ROOT 6.04 Releases LCG_XX Set of testing and nightly chains defined to ensure the quality of the products provided in releases Nightly branches copied to AFS to ensure a fast access to experiments Use of these external packages on the experimental builds and releases

Cross-platform continuous integration and delivery open-source application Read access to all CERN members with NICE account (SSO protected) and reachable outside CERN Test instance available to try new versions and plug-ins Characteristics of our system: Ca. 300 cores available for nightly and releases 10 big projects including a total of ~100 different jobs Replacement of the previous tool (Electric Commander) expected before the end of this year (2 weeks time) 28/09/15Quality and Testing PH-SFT infrastructure 18 BUILD ORCHESTRATION: JENKINS

 Build results sent from Jenkins to cdash at the end of the build  Enable the fast identification of errors and causes 28/09/15Quality and Testing PH-SFT infrastructure 19 BUILD VISUALIZATION: CDASH

28/09/15Quality and Testing PH-SFT infrastructure 20 LCG RELEASES PUBLICATION Using the information provided at the end of the build configuration

1.Build results reproducibility 2.Manageable handling of package dependencies 3.End-systems independent installation procedures; reallocation 4.Proper quality assurance 5.Scalability of the infrastructure: increasing number of compilers, architectures 6.Fast improvement/feedback cycle 28/09/15Quality and Testing PH-SFT infrastructure 21 BACK TO SLIDE 6 - CHALLENGES

28/09/15Quality and Testing PH-SFT infrastructure 22 SUMMARY OF OUR BUILDS CompilersOSRelease type BUILD CHAINS gcc48gcc49 gcc51 clang35 icc15 native CentOS7 SLC6 mac1010 mac109 ubuntu14 Release Debug Experimental ✔✔✔✔✔✔✔✔✔✔✔✔✔ dev2 ✔✔✔✔✔✔✔✔ dev3 ✔✔✔✔✔✔✔✔ dev4 ✔✔✔✔✔✔✔✔ Releases ✔✔✔✔✔✔ Clang37 to be included soon Already working in testing of gcc5.2.0

28/09/15Quality and Testing PH-SFT infrastructure 23 SUMMARY OF THE TOOLS Platform independent PKGs Dependence s handling End-system installation independent Easy “configure- build-install” approach Platform independent PKGs Dependence s handling End-system installation independent Easy “configure- build-install” approach Escalation and automation scheduling procedures Easy access to build and test results Fast feedback of experiments Easy access to build and test results Fast feedback of experiments Lcgcmake repository Facilitate specific branch procedures for nightly and releases Enable developers collaboration Lcgcmake repository Facilitate specific branch procedures for nightly and releases Enable developers collaboration CMAKE/LCGCM AKE JENKINS CDASH GIT

PH-SFT has established a robust build and test infrastructure to provide experiments with ca. 200 packages on a regular basis CMAKE based Quality focused release procedures Experimental  nightly (3)  Releases chain Automation and escalation infrastructure: Based on Jenkins Test and production instances available Results visualization and access enabled for experiments CDASH, Releases publication web Installation in AFS and CVMFS In a near future: Consolidation of the group projects into Jenkins Improvement of the Jenkins instance Fostering of CVMFS usage including results of the nightly builds Inclusion of new architectures and compilers lcgcmake documentation 28/09/15Quality and Testing PH-SFT infrastructure 24 SUMMARY AND FUTURE PLANS

 PH-SFT web page   HSF web page   GITLAB repository for LCGCMAKE   LCG Releases publication   PH-SFT Jenkins instance   Cdash instance  28/09/15Quality and Testing PH-SFT infrastructure 25 LIST OF POINTERS