1. The LCG SPI project in LCG Phase II CHEP’06, Mumbai, India Feb. 14, 2006 Andreas Pfeiffer -- for the SPI team andreas.pfeiffer@cern.ch

2. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 2 Overview Introduction and Motivation Services of SPI Savannah service External software and distribution services Configuration and Build Infrastructure Software testing and QA services Documentation service Outlook and future plans Summary

3. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 3 The SPI project in the LCG Applications Area Goal of the project Provide to the development projects of the LCG Application Area -Basic environment for physics SW development -General scientific libraries and class libraries -Software development tools -Documentation tools and document templates -Compiler expertise Have consistency and homogeneity in the development of the different packages of the LCG Application Area -Configuration management -Central software repositories -Common policies (naming, building, etc.)

4. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 4 Software Process and Infrastructure Work guided and prioritized by Architects Forum Providing services to the projects (and experiments) Channelling the requests from projects (and experiments) Providing the following services: Savannah service -issue and task tracker Software services -central installations of external software, software distribution Software development service -configuration mgt, pre- and post-build service, tools for testing and QA Web and Documentation

5. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 5 Savannah Service Web portal for LCG software projects Single entry point for all projects -Uniform “look and feel” for all projects Totally web based Used for various functionalities Bug/issue tracking Task management Download area Customized from GNU savannah project In collaboration with main developer

6. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 6 Adoption by projects 1575 users 176 projects 2770 tasks13500 bugs

7. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 7 Functional enhancements Project specific 'cookbooks' based on the tracker engine Inherits the tracker functionality while providing a friendly book-like item display. Export facility Helps extracting data to insert into reports (new releases, statistics, project monitoring, etc) Voting system Gives registered users the possibility to prioritize the way items are handled Additional browsing criteria Selection of items modified/unmodified since a certain date Extended transition mechanism Automatic update of multiple fields on specific value transitions

8. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 8 External Software Services Install software needed by LCG projects and experiments Open source and public domain software -Libraries and tools Standard structure of directories (central AFS repository) / / /(lib, src, bin, …) Build and dependency information now stored in XML files with scripts to build (semi-)automatically Very easy to build a new version or new platform Easy to convert to formats for other tools (e.g. pacman) Experiments and projects decide what is installed Through “Architects Forum”

9. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 9 External software service -- example

10. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 10 Software Distribution Service Existing solution since 2002 Script to download and install tar-balls of binaries for software of LCG AA projects with all dependencies Limited to binaries of platforms build at CERN Dependency information from configuration management and build tool (scram) Effort started to download source tarballs and (re-) build as needed Simplifies porting to new platforms (or development work) Re-using build and dependency information from XML files Will be extended to download/install binaries Work ongoing to fully automate this service

11. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 11 Configuration and Build Infrastructure LCG Librarian Maintaining configuration information centrally for build systems in use -XML files + converters to scram and cmt Coordinating releases and pre-releases Librarians integration group Discuss with experiments’ librarians/integrators additional needs on external packages Bi-weekly meeting (LIM) Developed simple tools to automate builds of LCG AA software Three phases: pre-build, build (on all platforms in parallel), post-build Work ongoing to improve automation Provide tools for release managers in projects to do this

12. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 12 Software Testing Service Providing Test frameworks, support, policies and documentation CppUnit, Oval Qmtest valgrind On different compilers/platforms Running the tests is an integral part of the development in the projects Unit tests, integration tests and regression tests Also available for experiments Initial setup User support

13. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 13 Quality Assurance Service Central QA checklist on each release of LCG AA projects Code coverage Providing statistics on tests and from savannah as a coherent view Automated procedure to generate reports At every release Published on the SPI web site Evaluation and usage of external tools E.g. RuleChecker, Logiscope

14. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 14 QA example (ROOT project)

15. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 15 QA example (POOL project)

16. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 16 Documentation Service Document LCG Policies Homogenize the projects “look and feel” CVS and release directory structures Software testing policies Code documentation Through SPI web site and wiki pages Tools used: doxygen and LXR Run for every release of LCG AA projects plus some external packages (CLHEP, ROOT, HepMC) -Procedure is now semi-automated Internal procedures documented “explicit knowledge” rather than “reading scripts”

17. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 17 Web and Documentation Maintain and improve existing web pages Update internal and external documentation (wiki pages) SPI and LCG Workbooks Automate content wherever possible Create more “active/dynamic” web pages -E.g.: status of external tools, configuration info -New: pages to do pre- and post- build procedures Update and clarify existing policies To be reviewed by AF Update web pages for external packages Responsibles, status, … for each package

18. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 18 Challenges for LCG phase II Servicing LHC experiments more than AA projects Discussion with experiments to (re-)define scope -What is needed, what can be improved Participation from people in experiments “On-the-spot” help with requests from experiments SPI provides a “frame” to do complex work -Guidelines, AFS space, web space for docs, … Coping with strong (50%) reduction of resources Optimization opportunities Further optimization and automatization of procedures

19. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 19 Summary SPI provides a large set of stable services that fully use the current resources Savannah, External Software, Testing Frameworks, Software Distribution, Build and Release, QA Activities SPI work is guided by the Architects Forum AF represents the users and steers our priorities In Phase II we will concentrate more on the experiments’ needs Who see LCG AA project software as “external” Ongoing discussion with AF and experiments To redefine the scope of the project -Resources are significantly reduced

20. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 20 “Thank you!” to the SPI team(s) 2005 Alberto Aimar, Johanne Benard, Ian McLaren, Yannick Patois, Yves Perrin, Andreas Pfeiffer, Eric Poinsignon 2006 Full time: Ian McLaren, Yves Perrin, Andreas Pfeiffer Part time: Bertrand Bellenot, Manuel Gallas, Stefan Roiser

21. 14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 21 Additional Information SPI project web: http://spi.cern.ch SPI project wiki pages: https://twiki.cern.ch/twiki/bin/view/SPI/WebHome LCG Applications Area web: http://lcgapp.cern.ch LCG web: http://cern.ch/lcg