CVS Service at CERN status and LCG-dedicated service CERN IT/PS/UI October 2003

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 Outline CVS Service for LCG The challenge Architecture Failure recovery Advantages, disadvantages Plans for the future CERN Central CVS Service Overview What does it offer? Projects Architecture Status Interactions with users Failures and fail-over Tools

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Overview central service hosting CERN-related software projects created following a service request - collection of user requirements - architecture proposals - implementation based on assigned resources in production since the end of August 2002 currently hosting around 45 projects, 3 GB of data (major part is source code)

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Projects AB-CO FPGA designs AB-RF-CS Section Software ABs Common-Build ATLAS TDAQ RobIn Anaphe Atlas (migrating) Automatic Driver Code Generation BIC surveillance project CASTOR project CCDB CERN Central CVS Service CERN Oracle Grid Data Management Services CESAR Project CMW administration tool CVS test Computer Centre - Operational Procedures Management Cyan Jaguar Diana Network device information analysing and monitoring Project EJB Platform Project Front End Software Architecture GUI Platform project HEPiX scripts IT-FIO software IT-PS-UI software Itcobe Java Api for Parameter Control Java Data Viewer LCG Oracle AS + DB Tier 1 Deployment LHC Alarm Service Project LHC Logging Project LHC Orbit feedback LHCb software Machine Component State Manager Macsy MammoGrid project Middle Tier Server for CMW NA48 analysis software NA60 Offline Project OASIS software OPERA project PHOS Trigger Router Unit PS LICMON Software QUATTOR components for CERN Section Activities Administration System...

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service What does it offer? secure and robust CVS service with up-to-date server software data integrity (mirror every hour, daily archiving) several access methods: Kerberos IV, SSH, pserver service support through Remedy automatic CVS lock monitoring and reporting good performance Web interfaces: CVSWeb, ViewCVS BUT: the service is not a project management tool

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Architecture automatic and transparent load-balancing and fault tolerance (via an ISS DNS alias that distributes CVS requests among a farm of four servers) dependency on AFS and DNS monitoring availability every 10 min. service availability higher than 99.98% so far

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Failures and fail-over On average, one out of four nodes is often down, due to: - software upgrades - 3ware disk controllers failures - hardware tests performed to investigate hangs (lxcvs02 has hung more than 20 times this year) 4 disks failed (no problem since they are mirrored) Automatic fail-over made the disruptions mentioned above transparent to CVS users Total down-time of the service this year < 12 hours (est.): - Computer Centre power cuts - several short network interruptions (mainly at non-working hours) - some others partial interruptions: ·AFS problems - affecting only some users ·xinetd configuration (wrong pserver port number, limit on kserver processes)

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Status Software upgrades: -operating system → Red Hat 7.3 -CVS → (newest feature version) -applying patches when necessary Two additional nodes (SEIL 2x2.4GHz) have been added to the cluster – currently four servers Fully automated installation and configuration of the machines using WP4 tools (as in lxbatch): -CDB templates -kickstart files -... but still some SUE features to be translated into WP4 components

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Interactions with users Almost 200 user requests or questions received and answered so far - around 90 on the Remedy system - others via Full documentation on the Web ( ) - user documentation - manuals - howto's - list of CVS books - technical documentation for administrators Web tools for users: - configuring access type for web interfaces: Public, Restricted or None - modified CVSWeb and ViewCVS - encrypting passwords for pserver access

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CERN Central CVS Service Tools A series of tools were developed for service maintenance: detecting and increasing AFS volume size for projects which are about to get full CVS lock detection and reporting to librarians ISS statistics cluster nodes' information (availability; load; up-time; ISS information and enabling and disabling a node) cvsserver SUE feature (for automatic machine configuration) scripts for creating and deleting projects setup, availability and backup checking scripts etc.

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003

CVS Service for LCG The LCG challenge LCG (LHC Computing Grid – group has requested a non AFS-based CVS Service Several proposals were prepared to meet this demand One solution was chosen, implemented and is to be evaluated by LCG The old CVS Service will remain in production, available for non LCG-related projects

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CVS Service for LCG Architecture An „N+1” cluster: N active nodes with repositories on local file systems additional passive node („slave server”) – backup for all active servers data replication – repositories copied to the slave server for each repository there is a DNS alias pointing to the node hosting this repository when a node is down, aliases are redirected to the slave server currently 3+1 machines

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CVS Service for LCG Architecture repositories copies of repositories slave X.cvs.cern.ch Project X 12N servers

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CVS Service for LCG Advantages and disadvantages Advantages fastest possible access to the repositories (local file system) independent of AFS regular PC as servers (no special hardware bought) Disadvantages: constant mirroring may affect performance load-balance is on repository level (not request level) slave server down => no fault tolerance fail-over requires human intervention (for the time being) Plans for the future: Automatic fail-over (system decides when to switch to the slave server)

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 CVS Service for LCG Other information data replication is done by CVSup access to repositories and user home directories on a file-system level via NFS Web interface: CVSWeb (sample URL: instant DNS update CVS Service for LCG web page:

CVS Service – M. Guijarro, S. Lopienski (CERN IT/PS/UI) – October 2003 Thank you! Any questions? More information at: