1. EGEE is a project funded by the European Union under contract IST-2003-508833 Software Configuration Management and Integration Alberto Di Meglio EGEE JRA1 Integration Manager EGEE Kick-Off Meeting – Cork – 18-22/04/2004 www.eu-egee.org

2. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 2 Contents JRA1 Integration Mandate Team Objectives The SCM Plan: an Overview SCM and Integration Implementation Configuration and Version Control Build Systems Release Process Other Configuration and Change Control Procedures

3. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 3 JRA1 Integration Mandate EGEE JRA1 has a dedicated team for SCM and Integration activities The vision of the team is that SCM and Integration must start before a single line of code is written and must provide an integrated development and collaboration environment The mission is to provide a solid, reliable infrastructure to manage changes and release quality software based on industry-standard SCM procedures and tools throughout the entire duration of the project

4. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 4 Integration Team Objectives The Integration System must be simple, flexible and solid Excessive diversity in tools, languages, technologies must be avoided Centralization of processes and procedures is favoured Whenever possible industry-standard tools and methods with a large user base and reliable, official support must be used Making it work well on one platform is preferable to making it available on all platforms (at least initially) Extension to multiple platforms and usage scenarios will naturally follow for a well designed, developed and integrated software EGEE Middleware is a complex, multi-disciplinary, highly distributed project. This is part of the novelty and challenge of it and will require a novel approach to old and new problems and some courageous decisions

5. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 5 SCM Plan Overview A master Software Configuration Plan is being finalized now [1] It contains basic principles and rules about the various areas of SCM and Integration (version control, release management, build systems, bug tracking, etc) Compliant with internationally agreed standards (ISO 10007-2003 E [2], IEEE SCM Guidelines series [3]) Most EGEE stakeholders have already been involved in the process to make sure everybody is aware of, contributes to and uses the plan An EGEE JRA1 Developers’s Guide will follow shortly in collaboration with JRA2 (Quality Assurance) based on the SCM Plan It is of paramount importance to deliver the plan and guide as early as possible in the project lifetime

6. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 6 SCM and Integration Implementation As soon as the SCM Plan is finalized, we will start the implementation phase (some steps already taken) All tools, servers, procedures and communication channels will be put in place Developer’s and User’s Guides will be produced with clear instructions about how to use and interact with the project and manage changes in the correct way The complete infrastructure will be in place at the end of Month 3 (30 June) (MJRA1.1) Goals for the first 12 months are to release incremental code baselines to test with a very rapid lifecycle (weekly) and finally release v. 1.0.0 (main milestone is MJRA1.5 at Month 10: integrated EGEE RC 1 released to testing)

7. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 7 Version and Configuration Control Based on CVS using CERN Central IT CVS service Fixed directory structure for each module Rules for tagging and branching (e.g. bug fix branches) Common naming conventions for baseline and release tags Configuration files to automate creation of workspaces  Used to enforce build reproducibility  Used to create private workspaces for developers Authorization based on commitavail and tagavail according to SCM roles (developers, release managers, integrators, etc.) Supported protocols: K4, SSH (pserver only for anonymous read-only). A CERN AFS account is required

8. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 8 CVS Structure JRA1 EGEE CVS org.egee org.egee.subsys org.egee.subsys.comp1 org.egee.subsys.compN project src test doc interface examples config

9. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 9 Workspaces A Workspace is one of the basic SCM Patterns [4] It is a private area where a build can be produced under configuration control More independent workspaces can coexist on the same computer for parallel development The workspace is created using a Configuration Specification File (CSF) The CSFs contain all the information necessary to prepare the workspace extracting modules and other dependencies from CVS or the central software repository There is a CSF file for the overall org.egee Middleware suite and one CSF for each subsystem CSFs are stored in the corresponding modules in CVS and versioned so that any version of the MW suite and all its subsystems can be reproduced at any time

10. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 10 Build System Common Build System  A project-wide build system based on Ant to allow all developers, integrators and testers to manage private builds on their own computers with the same tools and settings (compilers, flags, configurations, etc)  Based on the concept of Private Workspaces, more than one independent workspace can be created on the same computer  Uses a set of common build and properties files to define targets and properties for all project modules  External dependencies are stored and versioned either in CVS or a central repository maintained by CERN SPI. All modules must use the SAME versions of the external dependencies across the project Central Build System  Located at CERN and maintained by the JRA1 Integration Team  Based on the Common Build System, but automated by means of CruiseControl and Maven to generate a continuous, automated integration process and the overall project web site (Apache style).

11. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 11 Build System (2) The Ant build system provides the common build orchestration for all modules, not only Java Modules can be built independently and use Ant or Make as specified by project requirements, independently from the module implementation language A set of common targets has been defined for all modules (environment checks, style checks, compilation, unit tests, distribution, etc) The entire MW suite, individual subsystems and individual modules can be built with the same set of targets Quality Assurance checks are built into the build process and are mandatory. Failure to pass a quality check (coding styles, unit testing, minimum test coverage, documentation, etc) fails the build

12. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 12 Build System (3) Five main types of builds:  Continuous Incremental Builds: every few hours, mail messages are sent to developers who break the build  Nightly build: every night from clean snapshot  Weekly builds: every week from clean project configurations  Maintenance builds: used to fix software defects on existing releases  Private builds: whenever needed on the private development computers

13. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 13 Build Tools Ant: used for the general build management and all Java modules. The advantages/disadvantages of using it also for C/C++ and other languages modules are under investigation Make + GNU Autotools: for C/C++ and other languages as necessary. Although this is the standard, familiar approach, it has a lot of drawbacks, especially for large, distributed projects (difficult to maintain and centralize, lots of external dependencies, poor internal dependency tracking, file based instead of task based, etc) CruiseControl: used to automate the integration builds Maven: used to automate the project web site, reports and documentation builds

14. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 14 Packaging and Configuration Installation packages for the EGEE Middleware will be produced by the integration team Packages will be provided for the entire middleware and for all subsystems and deployment modules Supported packaging formats are:  Source tarballs with build scripts  Native packages for supported platforms (e.g. RPMs for Red Hat Linux Enterprise 3.0, MSIs for Windows XP, etc) Extracting and using modules directly from CVS is not recommended, since there is no guarantee the dependencies are satisfied outside the build system context Configuration issues are being investigated with SA1 The goal is to have a simple, common format for all configuration items, usable by standard installation/configuration tools Some type of “Middleware Configuration Front-End” may also be provided in collaboration with SA1

15. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 15 Release Process 1. Developers works on code functionality and documentation and produce baseline candidates 2. Baseline candidates are continuously integrated and unit and smoke tested 3. A build is successful if all the functionality contained in the module is properly unit tested and documented. If a build is not successful, the baseline candidate is rejected and control returns to the developers 4. Baselines are automatically tagged if the build process is successful 5. When all baseline candidates in the application build list (Application CSF file) are successfully built, an application baseline is produced and passed to testing 6. If all test, certification and validation procedures are successful, the individual baselines in the build are considered eligible to be promoted to release if all required functionality is present 7. Development Release Managers promote eligible baselines to full releases 8. When all components have been released, functionality is frozen and the application is promoted to Release Candidate. 9. The Release Candidate is thoroughly tested, validated and certified, all critical and major bugs are fixed, most normal bugs are fixed. A number of Release Candidates are produced during a short number of iterations. 10. The final release is published

16. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 16 Change Control Board All public changes must go through a formal approval process The CCB is tasked to collect and examine the change requests Changes must be tracked and communicated as efficiently as possible The role of the CCB (and SCM in general) is not to prevent change, but to organize it so it has only positive impacts on the project It will meet periodically and on demand Need to define composition of the board, very likely it will be composed of the JRA1 activity and cluster managers plus representatives of the architecture and design teams, deployment and applications The relationship with the EGEE Architecture Oversight Committee (AOC) is under discussion

17. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 17 Defect Tracking System Bugzilla-like tool provided as part of the Savannah project portal Provided and supported by SPI team at CERN Need to add tools for collecting QA metrics (current reporting in Savannah is currently not sufficient) Used also for change requests (for example API changes, external libraries version changes, etc). In this case, request are assigned to the Change Control Board for further evaluation

18. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 18 Defect Tracking Cycle Ten states: Open, Analyzed, Accepted, In Progress, Ready for Integration, Ready for Test, In Test, Approved, Ready for Review, Closed Transition between two states subject to specific checks (Is it a defect? Can it be fixed? Test passed, Review passed?) ITeam are first filter to assign bugs to people or categories if required

19. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 19 Software Quality Assurance Software Metrics are collected as part of the build process Failure to pass a quality check fails the build Additional checks are implemented in the version control system (coding style checks) Software Defect and QA Metrics are collected from the defect tracking system All reports are published every night on the project web site Reports should be used by project and development managers to take corrective actions

20. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 20 Collaboration Tools Project Portal: Savannah from CERN SPI Single entry point for the project, news, tools, documentation Links to other relevant sites (CVS web browsing, builds status, package repository, project metrics and other artefacts) Mailing lists from CERN or Terena News, RSS feeds, etc, as necessary

21. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 21 Contacts ITeam mailing list project-eu-egee-middleware-integration@cern.ch List of the ITeam members and other activities representatives (closed list) Integration support project-egee-middleware-integration-support@cern.ch Public support list for reporting integration and build problems

22. EGEE Kick-Off Meeting – Cork – 18-22/04/2004 - 22 References [1] EGEE Middleware SCM Plan (https://edms.cern.ch/document/446241/0.6) [2] ISO 10007:2003 - Quality management systems -- Guidelines for configuration management, ISO, 2003 [3] http://standards.ieee.org/reading/ieee/std/se/828- 1998.pdf (requires a subscription) [4] Software Configuration Management Patterns, S.P. Berczuk, Software Patterns Series, Addison-Wesley, 2002