1. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS setup at CC-IN2P3 and Datagrid edg- build tools CVS management, centralized code checking and automatic building of rpm packages from CVS Yannick Patois E-mail patois@in2p3.fr

2. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Table Of Content  CVS service at CC-IN2P3 (Loïc Tortay) How it started Authorized user access in read/write setup Anonymous access setup CVSweb  Datagrid CVS tools (edg-build) General idea Code checking RPM building Package publishing RPM distribution management

3. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS service at CC-IN2P3 - starting point  To share code development among local users: CVS repository on AFS  But a project (Datagrid) needed a CVS repository open to more people Lab policy wouldn't allow local login for everyone Needed another solution  Once created and tested for Datagrid, this CVS service was extender to other experiments: cvs.in2p3.fr was created.  The presented setup has been set in place by Loïc Tortay at CC-IN2P3.  CVS.IN2P3.FR Dedicated authentication mechanism (not part of the generic users account policy) Offer open CVS services to experiements.

4. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS at CC-IN2P3 - Read/Write access setup  Authentication by ssh keys  Interactive login is disabled  CVS operations are performed through a specific pseudo-shell that only allow CVS operations to be executed.  rcvssh : remote CVS shell Build from 'anoncvssh' (OpenBSD project) Authentication managed above (by sshd) The shell ensure that the command and arguments are in a predefined set of allowed commands (cvs commands in our case) Can chroot the user to a subtree (anonymous access) Verbose login facility

5. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS setup at CC-IN2P3 - Anonymous access  Anonymous access is done on a chrooted cvs server  Use a different port than the authenticated access mechanism  The dedicated ssh daemon uses specifics configurations parameters (allow password authentication, allow empty password, only one account allowed) Picture taken from http://www.winternet.com/~mikelr/ with kind courtesy of Mike Reed.

6. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS setup at CC-IN2P3 - CVSweb and other tools  CVSweb setup Module based configuration Web access aliased cvs.in2p3.fr/experiment Apache access rights can be managed by experiment  CVS commit info on mailing list

7. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 CVS Services at CCIN2P3  cvs@in2p3 Service just started a few weeks ago A few experiments (EROS, SUPERNOVAE,..) uses it  Datagrid CVS repository Around 120 accounts 360Mb of sources A few hundreds connexions per day (more than one thousand if accounting for automatic connexion -like autobuild)

8. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Autobuild tools: The Datagrid collaboration  European grid project A Grid for  HEP  Biomedical applications (genomic and imaging)  Earth study (ESA)  Constraint Distributed development Several dependent modules (around 30)

9. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 The Datagrid Software Repository  The Datagrid Software Repository Hosted at CC-IN2P3 (Lyon) CNRS - France Main services:  Source code management for developers (CVS)  Packages distribution (RPMS)  The autobuild an check system (automatic building of packages)  Packages distribution Management http://datagrid.in2p3.fr/autobuild/rh6.2/

10. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Autobuild system - generalities  Autobuild Around 3000 SLOC of python One external Java program called A lot of os.system() calls to various unix utilities (from mkdir to cvs)  Context Run on RH6.2 and RH7.2, have been tested on Solaris CVS and package repository machine is remote (btw it's an AIX machine). Publication Machine IBM- AIX CVS repository Packages (rpm) repository Publication Machine IBM- AIX CVS repository Packages (rpm) repository Build Machine RedHat Linux Building, package managing Build Machine RedHat Linux Building, package managing CVS checkout ScpScp

11. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Autobuild system : what it does  Statistical informations about the project RPMS number/size SLOCs for every edg code  Auto-documentation pages Processed user doc (LaTeX files,...) Autogenerated code doc (Doxygen...)  Build platform informations Installed rpms, environments...  CVS module processing for each module Conformance check  README and INSTALL files  CVS tags Build  autotools  make  make doc  make check  make install  make dist  make rpm Publish build RPMs

12. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Autobuild : result sample

13. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Edg-release rpm lists

14. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Autobuild present and futur state  Autobuild program (called edg-build) can be downloaded from http://datagrid.in2p3.fr The code is most likely still very specific to Datagrid own needs, but is improving.  Current evolution plan Build on demand Email to developers on build failure Package managing capability (partially implemented)

15. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Creating RPM lists from dependencies I (idea)  Problem : Managing huge lists of RPM (edg-release) can be difficult.  Idea : Creating them from the RPM inter-dependencies Practical implementation Datagrid RPMS Striped RPMS (only headers) Custom “everything” RPM database Build Machine Local copy RPM task (virtual rpm pkgs) RPM task (virtual rpm pkgs) Full RPM list --whatprovide--whatrequire--whatprovide--whatrequire Custom list RPM database Rpm -Va Validation

16. Yannick Patois – CVS and Autobuild tools at CCIN2P3 – hepix - October, 24 2002 - n° 1 Creating RPM List from Dependencies II (example) Redhat-base-edg-0.0.1-1.i386.rpm redhat-kernel-edg-0.0.1-1.i386.rpm task-build-wp6-0.0.1-1.i386.rpm Redhat-base-edg-0.0.1-1.i386.rpm redhat-kernel-edg-0.0.1-1.i386.rpm task-build-wp6-0.0.1-1.i386.rpm Name: task-build-wp6 [...] # This is for edg-example Requires: libtool Requires: automake Requires: autoconf Requires: make Requires: c++ Requires: doxygen Requires: sgml-tools Requires: tetex-dvips Requires: ghostscript [...] Full RPM List