1. SPI Software Process & Infrastructure
GRIDPP Collaboration Meeting - 3 June 2004
Jakub MOSCICKI

2. Collaborative Facilities Deployment and Installation
SPI Services Overview
General Services
CVS service
Collaborative Facilities
Task Management
Mailing Lists
Web Portal
External Software
Coding
Analysis and Design
Development
Release
Specifications
Testing
Build systems
Deployment and Installation
Documentation
Quality Assurance
Software Development
Provide General Services needed by each project
CVS repository, Web Site, Software Library
Mailing Lists, Bug Reports, Task Management, Collaborative Facilities
Provide solutions specific to the Software Development phases
Tools, Templates, Policies, Support, Documentation, Examples
J.Moscicki
SPI - Software Process & Infrastructure

3. SPI Project Guidelines
Have different and separated services
Simple solutions, easy to learn, commonly needed services
Leave any process for later
Work with the users
Develop as little as possible
Establish simple deliverables
Everything is done starting from existing infrastructure
LCG and LCG projects (Pool, Seal, etc)
LHC experiments
IT division
Big projects (G4, Root, etc)
We did not start from tools for requirements, design, etc.
We started from development-related work
repository, releases, testing, bug report, etc
 The rest of the talk describes SPI services
J.Moscicki
SPI - Software Process & Infrastructure

4. SPI - Software Process & Infrastructure
SPI Services (April 2004)
External Software
Savannah Project Portal
Testing Frameworks
Development of LCG policies, templates
QA checklists and reports
Software Distribution
LCG Software Configuration
CVS server and AFS management for LCG App. Area
Code Documentation (doxygen, lxr, viewcvs)
Automatic Nightly Build (Nicos)
Software Librarian, builds and releases (new, was Scram support)
Documentation and LCG Workbook
J.Moscicki
SPI - Software Process & Infrastructure

5. SPI Web Site - http://spi.cern.ch
J.Moscicki
SPI - Software Process & Infrastructure

6. SPI External Software Service
We install software needed by LCG projects.
Open Source and Public Domain software (libraries and tools) like:
Compilers (icc, ecc)
HEP made packages
Scientific libraries (GSL)
General tools (python)
Test tools (cppunit, qmtest)
Database software (mysql, mysql++)
Documentation generators (lxr, doxygen)
XML parsers (XercesC)
There are currently 50 different packages, plus others under evaluation. For more than 300 installations
The LCG projects (SEAL, POOL, PI, Simulation and SPI) propose what to install in agreement with LHC needs
The platforms, are decided by the Architect Forum
Linux RedHat 7.3 and compilers
gcc 3.2 and (rh73_gcc32)
icc 7.1 (rh73_icc71)
ecc 7.1 (rh73_ecc71)
Windows
Visual Studio .NET 7.1: (win32_vc7).
Mac OSX (osx103_gcc33)
Platforms always been reviewed
We also provide configuration for the LCG projects
A unique AFS location
Standard structure package_name/version/ platform/package_ content
J.Moscicki
SPI - Software Process & Infrastructure

7. External Software – http://spi.cern.ch/extsoft
J.Moscicki
SPI - Software Process & Infrastructure

8. SPI Savannah Portal Service
Functionality:
Bug tracking
Task management
Mailing lists, news, faqs
Access to CVS repository
Download area, etc
The Web portal for LCG software projects
Customized from GNU (SourceForge as origin)
Totally web based
Single entry point to all projects
Uniform access to project information
Set up common web infrastructure for a project without coding
What SPI changed
installation from GNU, general bug fixing and improvements
integration with AFS authentication
Integration with standard services already available
What SPI does
administration (project approval)
maintenance (submitted bugs)
development (support requests)
Status
>80 hosted projects
>550 registered users
J.Moscicki
SPI - Software Process & Infrastructure

9. Savannah Service – http://savannah.cern.ch
J.Moscicki
SPI - Software Process & Infrastructure

10. SPI - Software Process & Infrastructure
SPI Testing Services
Software testing should be an integral part of the software development in the LCG App Area
The goal was to provide something that can be run automatically as often as needed (releases, development, etc)
Integration
Tests
System
Examples
Software testing
Integration test
System test
Acceptance test
Unit test
Software
developer
Sw-testing
team
LHC
experiments
Use in Exp.
Work Package
Test
Automated testing
SPI provides
Test frameworks
CppUnit, Oval
Qmtest
Test support
Test policies
Test doc
Different platforms/compilers
J.Moscicki
SPI - Software Process & Infrastructure

11. Testing Support – http://spi.cern.ch/testing
J.Moscicki
SPI - Software Process & Infrastructure

12. Quality Assurance Service - http://spi.cern.ch/qa
The main goal of QA activity help LCG projects
assess and improve the quality of the software
provide tools to collect useful metrics/statistics which help to asses quality;
generate reports;
verify if project setup is correct with LCG policies.
QA Tools and Focus
Automatic reports
Development/integration of automatic tools
LCG Policies
agreed and defined by AF
SPI supports them in the tools and procedures and only helps to work them out
J.Moscicki
SPI - Software Process & Infrastructure

13. Quality Assurance Activities
QA Checklist on each Release
Build the release
Run automatic tests
Statistics
Test Inventory
Documentation/Examples Inventory
Savannah Statistics
Code Inventory
Rule Checker , Logiscope
LCG Policies
Configuration of a build system
CVS directory structure
Well-defined, transparent, open
clear rules and checklist of assessed items
anybody at anytime may see statistics
create reports themselves
anybody may contribute
QA Reports
Automatic reports
Generated at every release
Published on the SPI web site
Evaluation and usage of external tools
Rule Checker
Logiscope, Test coverage
SLOC, Valgrind, ignominy
J.Moscicki
SPI - Software Process & Infrastructure

14. SPI - Software Process & Infrastructure
QA Reports
J.Moscicki
SPI - Software Process & Infrastructure

15. SPI - Software Process & Infrastructure
LCG Policies
LCG Policies
CVS and Build Directory Policy
Software Testing Policies
Version Numbers, Tagging and Release Procedure
Installation Directory Structure
Platform string, binary names, debug flags and more
They are a needed by the LCG
They are defined by the LCG projects, collected by SPI
If everything is different is too difficult to use and to automate
compromising on our habits, for project needs
tell when they are not followed
First time that this exists at this extent, and that is checked
J.Moscicki
SPI - Software Process & Infrastructure

16. SPI Software Distribution Service
Simple solution to use
local installations (external sites, laptops,...)
using simplest approach
python downloader + tar format
replicate the central AFS tree (in a optimized way)
package dependency from SCRAM
...until a complete, long-term solution available
Looking into pacman as a suitable solution
SPI will adopt what LCG Grid Deployment decides to provide
Simple tool to install
successful for users:
Karlsruhe
BNL nightly builds, CMS
developers at home, etc
very easy to use and reliable
Different use-cases should have different solutions
Our tool is adequate as a temporary solution for LCG Application Area Distribution
but long-term solutions must be investigated:
pacman, LCFGng ....
GRID WN installations should be supported differently
J.Moscicki
SPI - Software Process & Infrastructure

17. Software Distribution – http://spi.cern.ch/lcgsoft
J.Moscicki
SPI - Software Process & Infrastructure

18. SPI - Software Process & Infrastructure
Summary
The set of services is working and fully available
Savannah Project Portal
Software Testing
External Software Service
Quality Assurance and Policies
Software Distribution
…and many more
We have followed the strategy defined
Work with the users
Ask their help
Develop as little as possible in order to have little maintenance
Provide simple and modular solutions
We have commitment to provide a sustainable service
Most people moved on to new projects, as it was planned
The services are used by LCG projects, and many outside LCG
Unlike in the past, we try to match the environment and habits of the way people work in HEP
J.Moscicki
SPI - Software Process & Infrastructure