The gLite middleware distribution Oliver Keeble CERN

Overview Why is gLite? What is gLite? Services and status Roadmap Release process The future The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Middleware What’s so great about being in the middle anyway? The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Why is gLite? So we need software to manage the infrastructure, but why does EGEE produce a distribution? Simplify availability of services and updates Provide integrated 'services‘ from components In the absence of standards, promote interoperability Identify endorsed projects at certified versions Provide an entry point for the uninitiated The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Where does gLite begin... and end? gLite is not just somewhere we stick stuff which might be useful for grids Adding something to gLite has a cost for the project Core services Relationship with Applications RESPECT Operational Tools The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

How is gLite put together? gLite is… A distribution With its own eponymous development project The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

gLite 3.1 The current release is gLite 3.1 It is updated almost every week Its purpose is to provide a stable platform for production grid usage Some highlights Data Management Workload Management Information System Security and Authorisation The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

What’s in there? The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

What’s in there… continued The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Running jobs The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Workload Management Just getting a job started involves a lot of services VOMS BDII WMS UI MyProxy WN TORQUE CE The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Data Management The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

The future The EGEE/gLite development activity is JRA1 Most of their effort is earmarked for maintenance Logging, monitoring, docs Address the bugs as found on the EGEE Production and Pre-Production Systems Provide support Address the short/medium term requests of the applications as decided by the TMB Improve the manageability of the middleware in terms of deployability, reliability and usability. Provide the internal unit testing of new or modified software Participate in the definition of the gLite releases together with the SA3 activity Maintain a web page with the relevant information needed by middleware users The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Authorisation service The single biggest chunk of development anticipated for EGEE-III What will this bring? Consistency across services Distributed Policy Management Auditing and Logging Most components will have to be adapted The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Authorisation Service The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

CREAM The infrastructure currently runs with the lcg-CE CREAM is the scheduled replacement Computing Resource Execution And Management service Scalability Parameter passing Direct submission WS interface job management (inc proxy renewal) Improved service administration A BES/JSDL interface is planned A first release is ready now for production The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

WMS/ICE WMS Submission through the WMS CREAM Direct Job Submission Job management tasks are delegated to the gLite Workload Management Service Can now sustain 30k jobs per day under normal operations The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Support for pilot jobs A pilot job is a shell which pulls in the ‘real’ job when its execution starts This is now a common way for VOs to manage their workloads As it runs something on behalf of another user, the mapping to local accounts has to be redone; glexec This mapping must be consistent across a site to prevent two different users sharing an account; SCAS The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

SAML2-XACML2 Query Protocol glexec and SCAS Site Query for Authorisation decisions on behalf of users gLExec-on-WN LCMAPS SAML2-XACML2 Query Protocol GUMS & SAZ SCAS LCAS & LCMAPS Site Central AuthZ Service The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Encrypted Data Storage gLite provides solutions for managing sensitive data One application is medical data management, where DICOM is a standard for image acquisition and analysis gLite will release new services for this The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

The great thing about standards… Nordugrid CREAM ARC EGEE PBS/Torque LSF Condor Load Leveler Sun Grid Engine GRAM v2 Unicore OSG DEISA GRAM v4 NAREGI Teragrid Naregi There are as many Computing Interfaces as Batch Systems! The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Some standards relevant to gLite Glue2 Encapsulates the ‘accumulated wisdom’ of a number of years of grid operations Fixes problems Better model e.g. relationships between main entities in compute model have been improved Buy-in BES/JSDL Proposed as solutions by OGF Viability has not been established, investigations are ongoing SRM2 The Storage Resource Manager (SRM) is a middleware component whose function is to provide dynamic space allocation and file management on shared storage components on the Grid. The DPM supports SRM2.2 IGTF / x509 The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Glue 2 The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

How gLite is put together We allow independent things to remain independent We make a gLite release almost every week! The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Patch States [ ] Closed States In Configuration Configured [yes] Acceptance Criteria Config Required With Provider Ready for Integration Ready For Certification [yes] [no] Open [no] [yes] Pass [yes] Pass Pass In Pre Production In PPS-PDT Certified In Certification [yes] [no] [no] [no] Other Update Available SA1 SA3 JRA1 [no] [yes] [ In Production Rejected Obsolete ] Closed States The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Testing infrastructure Framework SAM, Nagios, Manual Testing Regression Tests Extensive use of virtualisation ETICS Move testing as far upstream as possible Deployment tests Regression tests Unit tests The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

A gLite 3.1 update announcement This indicates to an administrator which services they should update The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Multiplatform Support gLite currently runs on SL3 and SL4 ‘generations’, including selected services on 64bit A release of the WN on SL5 is nearly ready More to come Work continues on Debian 4 Now profiting from earlier work to make the codebase more portable Our build programme is being overhauled to be more multiplatform friendly The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Challenges for gLite Release mechanisms IPv6 What is the best way to deliver the software? IPv6 Response to virtualisation Now the resource can adapt to the application, not the other way around Distribution of VMs for services or applications Support for running jobs in VMs Retirement and end-of-life for services and platforms Fight the Complexity Partially a result of heterogeneity Improve Agility The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

gLite Consortium EGI as currently proposed does not directly cover middleware development The EGEE PMB recommended the construction of a new organisation capable of guaranteeing the support and evolution of gLite in the EGI timeframe Would allow gLite to be properly represented within EGI or a UMD project A proposal for the gLite consortium is currently under discussion The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN

Summary gLite is critical in the realisation of one of the principal goals of EGEE; Maintain, enhance and simplify the use of the production quality computing infrastructure for an increasing range of researchers in diverse scientific fields gLite is enabling grids for e-science in Europe, across a multitude of scientific disciplines While there are plenty of new ideas in the pipeline, the overarching goal is that stated above, and the primary focus is now on consolidation The gLite middleware - EGEE 08 Istanbul - Oliver Keeble, CERN