Linux distributions HEPSYSMAN July 2004 P. D. Gronbech

Linux developments since Oct 03 Fermilab Solution Other independent options CERN negotiations

sent The following statement was sent a few days ago to CERN's Linux Certification List by Jan Iven, head of the Linux support team in CERN. I should add a few comments: 1. It is my understanding that the deal which Redhat was offering to CERN will also be available to all HEP sites in Europe. You should contact your local Redhat representative for details. 2. We should schedule a session at the forthcoming HEPiX meeting in late May in Edinburgh. I will organise this. Alan Silverman CERN Software Licence Office Dear list, our negotiations with Red Hat Europe were not progressing as intended and have now been stopped. We now propose to use a recompiled version of Red Hat Enterprise 3 for the next certification round, as has been previously discussed in the certification coordination meeting. IT management has accepted this proposal, so the (moderate) extra effort for recompilation is funded. We are also talking to Fermi to establish common procedures, even if we will not have a fully shared version at this moment. Both versions should be binary compatible with each other and with the official Red Hat Enterprise. Both versions and updates will be freely redistributable to outside institutes, however no extra support for external users is foreseen. We do continue a relationship with Red Hat and may reconsider licence and support options later this year. Best regards Jan Iven CERN Linux Support

On Fri, 27 Feb 2004, Michel Jouvin wrote: > One comment on your last mail about RH being ready to come to Edinburg > with > 'good news'. End of May is 'late'. In France (IN2P3), a discussion already > started following the Jan Iven's mail about the impact of CERN decision for > non CERN (and even non HEP - nuclear physics and astroparticles) > experiments. If there is still a chance for an agreement with Red Hat, it > would be nice to know about it as soon as possible to prepare the > discussion at Hepix rather than discovering new (attractive ?) proposals > from RH at Edinburgh. CERN has made a public commitment to produce a distribution which will be available free to client HEP sites. We will not be going back on this for this release. Our discussions with Redhat now focus on the next cycle. Edinburgh in May is plenty of time. We may even have some interesting experiences to relate by then. We will use the time from now to then to talk to clients about what might happen next cycle. I will be sending a mail next week. One of our current issues is from whom we get commitments. I can quote you 2 European sites where the techie says one thing and the manager, presumably, the budget holder, says another. The sites will need until May until they each speak with one voice!!!!!! Regards Alan

Sent: 10 March :01 To: Subject: [LCG-CORE] no phone conference this week Dear Core Sites, I am sorry, but due to overlap with some other meeting we will have no phone conference today. I haven't found the time to write proper minutes for the last meeting. Could someone please send an update on the dCache story to the list? Here is a short summary of the core points: dCache: Michael Ernst reported that almost everything was finished a version of the RM tools that doesn't use authentication on the data ports was needed for further testing. I brought this to the attention of Jean P. Baud. Work is expected to finish within a week. KERNELS: Sites have to look for secure kernels them self. Since Redhat stopped the support for 7.3 everyone is on his/her own. Our first tests with the CERN patched kernel showed that it brakes vfork We are testing the fedora kernel and will switch to this is needed. The most important nodes to be upgraded are the UIs. Migration LCG1 -> LCG2 Markus will (as soon as he finds time) send a mail that describes the migration plan from LCG1 to LCG2. Markus Schulz CERN IT

This last slide illustrates my worsed fears. LCG is based on RH7.3 now and it is no longer properly supported. The Cern Kernel does not work and so they are trying the fedora kernel. Individual sites will now be considering their options and a divergence of selected OS will probably be the result. If a deal could have been struck or a decision to just go with RHEL 3 made then all the institutes would have been common and all the benefits of sharing would continue….

STOP PRESS On Behalf Of Alan Silverman Sent: 22 March :48 To: Subject: CERN's position on Redhat For the past several months, CERN has been discussing with Redhat representatives both from Europe and from their US Headquarters. CERN's requirements can be summarised as an agreement which offers some added value in terms of support of a distribution which includes some modules modified by CERN for our needs and running on particular platforms chosen by CERN. - an agreement available at reasonable conditions to external clients of CERN's releases; if possible the agreement should be available to all European HEP sites - an agreement which is financially acceptable to CERN, both today and as we build towards LHC startup After considerable negotiation and pressure from both CERN and our partners, a framework agreement is emerging which appears to be acceptable to both sides. It is based on a modest per- node fee, special conditions for very large clusters, call-in support for Redhat modules and dedicated support for the overall release. In the coming weeks, the details should be clarified and CERN will license a certain limited number of nodes in order to evaluate the agreement internally and to get some practical experience with a view to basing future release cycles on this agreement. One important implication of this choice is that an external site wishing to use future releases from CERN will be asked to license with Redhat for a modest fee those nodes on which it runs the CERN-modified version of Redhat.

We are inviting Redhat to the next HEPiX meeting (end May in Edinburgh) to present the precise details of the deal but sites wishing information before then should contact their local Redhat sales unit. [For European sites, I can supply the name of a senior contact.] As we prepare technically for this new scheme, we seek feedback from such external clients, in particular are they prepared to license those nodes on which they wish to run CERN's future releases? The actual decision for future releases will depend on a number of factors including the experience gained in this round by SLAC and by CERN (in the latter case experience both of building our own release and in the use of Redhat-licensed nodes) and on your feedback. In the meantime, for reasons of timing and already-publicised commitment, CERN has recently begun the certification of a Linux release based on recompiled versions of the GPL sources of Redhat Enterprise WS version 3. This release should be available in about 2 months for general use, no restrictions, free access, no formal support from CERN or Redhat. There will be several presentations and discussions on this topic at HEPiX in Edinburgh in May (24 to 28) and all interested sites are encouraged to send appropriate representatives there. Alan Silverman Jan Iven CERN Software Licence Office CERN Linux Support

New solutions emerge from HEPIX Scientific Linux FNAL developed Fermi LTS which was based on recompiled RHEL 3 source rpms with Fermi customization Stripped off the fermi customization to produce Scientific Linux. Provided mechanism to select Fermi flavour or other at installation time

New solutions 2 CERN produce CEL Produced from recompiled RHEL 3 source rpms with CERN modifications applied HEP wide deal struck with Red Hat Approx 28 euro’s for RHEL WS Magic words are ‘HEPIX deal’ Extra cost for Satellite servers and node connection Contact if local sales reps don’t know about

Proposal for LCG to use SL The attached was discussed at the Edinburg HEPiX, with many positive responses, and is now put forward to the HEPiX list. We are proposing that LCG and/or EGEE standardize on Scientific Linux as the base Linux distribution. This is an "almost-fnal" draft, please send your comments, cc'ing the list. Thanks. Proposal for LCG and/or EGEE to standardize on SL as the base Linux platform Problem: LHC experiments expect to find compatible runtime environments at all participating laboratories. LCG uses tag-based job allocation that cannot easily cope with a variety of operating environments. Validation of a new platform for all experiments' use is largely manual and cannot be repeated too often. There is a clear need for standardization. Sites need to customize their operating environment, for example, to enforce local security policies, this is expected to continue into the future. Sites also have non-matching release cycles and different updating policies.

Recent developments: Most of the HEP sites see Red Hat Enterprise 3 as a viable successor to the various Red Hat-based solutions currently deployed [HEPiX Vancouver/Edinburgh]. This is available as a supported commercial product from Red Hat, and freely (but unsupported) in recompiled form by various groups, including CERN and Fermi. Security updates are available via a vendor support contract or in source form for free. All of these versions are binary compatible with each other (this is an explicit design goal of the recompilation efforts), but the installed software packages may vary between sites. Fermi has made an effort to separate their site customizations from a reusable set of core packages. Fermi also provides a set of scripts to allow for easy per-site customizations, including the facility to "re-brand" and to add/replace software packages. Core packages and scripts have been released to the HEP community under the name of "Scientific Linux". Fermi will base their next production release on this, CERN is evaluating whether a switch can be done this late in the ongoing certification process. Several HEP institutes (e.g. TRIUMF) have expressed strong interest, and others were following CERN's distribution in the past and will probably do so in the future (e.g. NIKHEF). Several of the US DOE labs have purchased commercial Red Hat under a recently negotiated agreement. A similar agreement is expected to apply to other HEP labs. SLAC currently runs the commercial version in production. This presents a window of opportunity to get a large part of HEP to agree on a (set of) common (binary compatible) distributions). Initial discussion shows support for common packaging and deployment policies (prefer back ported fixes to "core" packages, do not replace "core" packages except for security), but neither complete synchronization between sites nor "free" support for other institutes should be expected.

Proposal: The common set of packages in Scientific Linux (before site customization) will be binary compatible with the commercial Red Hat Enterprise 3 Linux. Scientific Linux "sites" will be discouraged from modifying this in incompatible ways. It will be possible to certify physics applications and middleware on either Scientific Linux or commercial Red Hat Linux interchangeably. LCG should ask collaborating experiments to validate their production software on one of the Scientific Linux derivates or a suitably restricted subset of Red Hat Enterprise. Since nearly all of these will require per-site modifications, care has to be taken during the build process not to pick up add-on or customized packages. Such software would then be expected to run on all SL-derived operating environments without further per-site certification. It should also show a high degree of resiliency against the inevitable configuration "dithering" from software updates. Automatic testing/ validation suites are still recommended to allow easy transitions between releases. Development should be encouraged to stay with as few versions of libraries as possible, to facilitate later rollout into a new "common" release. The practice of "locking" onto certain library versions conflicting with the standard system installation is expensive to maintain (indefinite availability required), and the "production" releases should able to use pre-installed software as far as possible.

Scope: Initially until 12/2005 (current minimum lifetime across Fermi and CERN). Extension is possible given that the underlying base product (Red Hat Enterprise 3) will get security updates until Endorsement: written by Mark Kaletka (Fermi), Jan Iven (CERN) Footnote: Red Hat has offered serious price discounts for the commercial version, but seems to have not yet grasped the uniqueness of the HEP community. Furthermore, it looks like only a combined package of all their software management products and services would allow replacement of local support efforts. (Our current understanding is the base "entitlement" price includes only security updates with no direct technical support, with updates directly from Red Hat which do not scale well and are without admin control. High-level consultant-like support service comes only with the Technical Account Manager (TAM), this requires additional per-node support. Software management on-par with existing implementations requires per- node fee and "Satellite Server" product.) These offers are still being tested. CERN and Fermi have announced that their next production versions will be based on a "free" alternative, the next opportunity to switch would be in 2005.