SPI Report for the LHCC Comprehensive Review Stefan Roiser for the SPI project

19. Nov. 2007LHCC Comprehensive Review - SPI2 SPI services Software process support –Software testing, configuration & release Documentation support –Code documentation, web services Collaboration tools –Hypernews, Savannah Hardware infrastructure –Build servers, special purpose machines

Software

19. Nov. 2007LHCC Comprehensive Review - SPI4 Time to user The time from a bug report / feature request until it is used by the physicists in the experiments –Especially important to minimize this time when running with “live” data.

19. Nov. 2007LHCC Comprehensive Review - SPI5 Nightly Builds Recommendation of the 2006 internal review Motivation –Early integration into experiment software –Fast feedback to LCG/AA developers of changes Usage –Building with own experiment nightlies on top –Developers testing their code on LCG/AA stack Used by –LHCb –Atlas

19. Nov. 2007LHCC Comprehensive Review - SPI6 Nightly Builds (ctd.) 5 configurations LCG/AA projects 13 possible platforms Builds & Tests test evolution

19. Nov. 2007LHCC Comprehensive Review - SPI7 Operating System eg. Athena, CMSSW, Gaudi Linux (slc) 32/64 bit, Mac OSX, Windows External Software LCG / AA software Experiment software Operating System eg. Athena, CMSSW, Gaudi Linux (slc) 32/64 bit, Mac OSX, Windows External Software LCG / AA software Experiment software Software release 56 packages, eg. Boost, Python, Qt, etc… COOL, CORAL, POOL, RELAX, ROOT, SEAL LCG release rubberstamps all package versions provides several products afs installation web pages src tar balls bin tar balls documentation

19. Nov. 2007LHCC Comprehensive Review - SPI8 Software processes mysql extern s/w scramcmt install. mgr. cron / python tar balls xml / python nightly builds project s/w web sites docu xml historically grown

19. Nov. 2007LHCC Comprehensive Review - SPI9 Software processes new extern s/w install. mgr. tar balls nightly builds project s/w web sites docu Python / LCGCMT

19. Nov. 2007LHCC Comprehensive Review - SPI10 LCGCMT Is a CMT based project Contains the configuration information –How to build software packages from source Using sh/bat scripts –Extra information Eg. homepage, package maintainer, description, etc. –In addition for other CMT based projects How to use/build against the packages Runtime environment

19. Nov. 2007LHCC Comprehensive Review - SPI11 Example: package build package gsl use LCG_BuildPolicy v* LCG_Builders macro gsl_build_config_version "$(GSL_config_version)" set LCG_tarfilename "gsl-$(gsl_build_config_version).tar.gz" apply_pattern buildscripts_local #!/bin/sh cd ${LCG_builddir} tar xvfz ${LCG_tarfilename} cd gsl-${LCG_package_config_version}./configure --prefix=${LCG_extdir}/GSL/${LCG_package_config_version} /${LCG_CMTCONFIG} #!/bin/sh cd ${LCG_builddir}/gsl-${LCG_package_config_version} make #!/bin/sh cd ${LCG_builddir}/gsl-${LCG_package_config_version} make install lxplus:~ >setenv CMTCONFIG slc4_amd64_gcc34 lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_get lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_config lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_make lxplus:~/LCGCMT/LCG_Builders/gsl/cmt >cmt pkg_install

19. Nov. 2007LHCC Comprehensive Review - SPI12 Benefits of new s/w process Avoid duplication of information Easier maintenance / automation All parties use the same info (nightly builds, release, users rebuilding the software) Speedup time to user Current configuration LCG 53f has been built with these tools

Communication

19. Nov. 2007LHCC Comprehensive Review - SPI14 Hypernews Bulletin board system used by LHC experiments Moved to new IT provided hardware –1 production machine, 1 hot standby Helping with instance maintenance –Managing user DB, setup of for a Atlas and CMS heavy users –Also used in Alice and Totem –LHCb is rethinking their usage

19. Nov. 2007LHCC Comprehensive Review - SPI15 Savannah Savannah is the main bug reporting system in LCG/AA Moved to new IT provided hardware Used by many other parties (IT, Grid, etc.) Reasonable system / user maintenance

Hardware

19. Nov. 2007LHCC Comprehensive Review - SPI17 Hardware LCG / AA build servers –4 linux machines (managed by IT/FIO) –2 mac osx machines (one ppc) –2 windows machines Special purpose machines –Web / database servers Recycling old Hypernews machines (located in CC) –New machines for Savannah, Hypernews

Staff & Program of Work

19. Nov. 2007LHCC Comprehensive Review - SPI19 Staff Change of project leader –Andreas Pfeiffer left to CMS –Stefan Roiser took over 1 staff left, 1 fellow joined –Manuel Gallas left for Atlas –Piotr Kolet joined 1 summer student –Luana Micallef Currently 3.2 FTE

19. Nov. 2007LHCC Comprehensive Review - SPI20 Program of work Completion of the integrated software configuration / testing / release –Web pages derived from LCGCMT –Integrate remaining release steps Consistency checking Production of distribution kits Documentation generation –Completion of integration with nightly builds –Integration of installation manager

19. Nov. 2007LHCC Comprehensive Review - SPI21 Program of work (ctd.) Nightly build system –Integrate Geant4 builds Using IT infrastructure for –Setup of common LCG/AA web services –Setup of database services –Virtual windows machines for s/w build

19. Nov. 2007LHCC Comprehensive Review - SPI22 Program of work (ctd.) Hypernews –Investigate usage of IT provided SSO –Find solution for longterm support

19. Nov. 2007LHCC Comprehensive Review - SPI23 Conclusion Current focus on LCG/AA s/w stack –Optimize release process –Facilitate and unify processes for Nightly builds Release procedure User rebuilds Outsourcing of maintenance for other services, but still a lot of SPI manpower involved –Savannah –Hypernews –Hardware