gLite and LCG-2: an overview Dr. Mike Mineter National e-Science Centre, Edinburgh / UK Dr. Rüdiger Berlich, Forschungszentrum Karlsruhe / Germany Brisbane, 2 February 2006 Slides contributed by EGEE Team

Outline Grid concepts The LCG/EGEE relationship Background to gLite gLite services gLite status LCG-2 & gLite Overview, Brisbane,January 2006

Grid concepts LCG-2 & gLite Overview, Brisbane,January 2006

A multi-VO Grid EGEE is establishing a production grid infrastructure to support multiple, diverse VO’s User Interface User Interface Grid services Blue VO: people cooperating and sharign resources in different places. Similarly the red VO. The resources that have been made available to the red VO are circled in red; similarly for blue. LCG-2 & gLite Overview, Brisbane,January 2006

A multi-VO grid INTERNET Users join VOs Virtual organisation contributes resources & negotiates access Grid middleware runs on each resource “Storage elements” “Compute elements” Additional services (both people and grid middleware) enable the grid Effect: “virtual computing” across administrative domains empowering collaboration INTERNET LCG-2 & gLite Overview, Brisbane,January 2006

VO’s and middleware Application Application toolkits, standards Basic Grid services: AA, job submission, info, … Middleware: “collective services” Application toolkits, standards Application The tools, services used by the VO’s applications Community-specific standards Application development environment, portals, semantics, workflow In EGEE: Mainly VO-specific Emphasis of EGEE middleware Workflow –WILL allow user to “see” an application, the developer to code components, and these to be assembled into distributed services Different organisational structures and different application types imply different middleware fucntionality: So short-lived operations “supercharging” a GIS require interactive performance – not the batch oriented pattern that mainly characterises Grids LCG-2 & gLite Overview, Brisbane,January 2006

An EGEE-centric view of the middleware “ecosystem” 2001 EDG Globus MyProxy Condor ... VDT . . . LCG OSG, … DataTAG CrossGrid ... SRM 2004 GridCC NextGrid EGEE DEISA … This slide does not include many nationally funded initiatives – neither is it comprehensive in showing US and EU projects. Its here partly so that the relationship between VDT, LCG and EGEE – the ones heard most often – are seen, and are seen to be a small part of a complex story. The projects shown are not the only ones of importance. Grid middleware has evolved very fast, in many places. It entails different attempts to solve a range of problems; these tools need to become interoperable. It leaves Grid middleware in need of a view from a step back, and it has come in the convergence of Grids with web services. The development of Grid computing spans oceans – so the Univ of Wisconsin is one of two sites running the new prototype middleware for EGEE now. The scale of cooperation demands standards exist. The last years have begun to see these standards for Grids build on and with those from web services. USA EU Used in Future grids With apologies to many missing projects! LCG-2 & gLite Overview, Brisbane,January 2006

Computation services in EGEE Jobs are: Run from batch queues, termed “computing elements” CE’s Described in “Job Description Language” files gLite also supports Interactive jobs Jobs run in batch mode – “listener” receives messages from CE Parallelism using MPI MPI jobs can run on CE’s that support MPI not across administrative domains (not MPICH-G) Workflow (DAGs, from Condor) Checkpointing Partitioned jobs (soon) – e.g. Monte-Carlo LCG-2 & gLite Overview, Brisbane,January 2006

Data services in EGEE Simple data Structured data Files Requires Replica files Move data to computation Virtual filesystems Metadata for files File transfer These services are amongst those provided by EGEE Structured data RDBMS, XML databases Require extendable middleware tools to support computation near to data easy access, controlled by AA integration and federation Hence OGSA-DAI DAI: Data Access and Integration OGSA-DAI is NOT currently being ported to gLite Is being used in earth science VO however – see website from EGEE conference, Pisa, 2005 LCG-2 & gLite Overview, Brisbane,January 2006

Current production middleware LCG-2 LCG-2 & gLite Overview, Brisbane,January 2006

LCG and EGEE LCG : Large Hadron Collider Compute Grid EGEE committed to “hit the ground running” EGEE profits from the resources - no funded computing/data resources in EGEE Provided by the VOs LCG obtains additional production and operation efforts LCG experiments now comprise several of the many VOs in EGEE Current service (“LCG-2”) based on work done in LCG Middleware components to be upgraded by “gLite” services as they are proven “gLite 3” will be forged from LCG 2.7 + gLite services LCG : Large Hadron Collider Compute Grid LCG-2 & gLite Overview, Brisbane,January 2006

Current production grid: LCG-2 Computing cluster Network resources Data storage Operating system Local scheduler File system User access Security Data transfer Information schema Workload management Data management App monitoring system User interfaces Applications Hardware System software “Basic” services “Collective” services Application level services HPSS, CASTOR… Scientific Linux NFS, … PBS, Condor, LSF,Torque… VDT (Condor, Globus, GLUE) EU DataGrid Information system LCG-2 & gLite Overview, Brisbane,January 2006

Current production m’ware: LCG-2 Replica Catalogue Logging & Book-keeping Resource Broker Storage Element Computing Element = a batch queue Information Service Job Status DataSets info Author. &Authen. Job Submit Event Job Query Input “sandbox” Input “sandbox” + Broker Info Output “sandbox” Publish SE & CE info “User interface” LCG-2 & gLite Overview, Brisbane,January 2006

Background to gLite LCG-2 & gLite Overview, Brisbane,January 2006

If “The Grid” vision leads us here… Annapurna range – photo by Mike … then where are we now? LCG-2 & gLite Overview, Brisbane,January 2006

Grids: where are we now? Many key concepts identified and known Many grid projects have tested, and benefit from, these Major efforts now on establishing: Standards (a slow process) (e.g. Global Grid Forum, http://www.gridforum.org/ , OASIS, W3C, IETF ) Production Grids for multiple VO’s – EGEE the most international and biggest by far “Production” = Reliable, sustainable, with commitments to quality of service One stack of middleware that serves many communities Operational procedures and services (people, policy,..) New user communities From physics via bioinformatics, chemistry, earth science towards infrastructure for widest spectrum of collaboration … whilst research & development continues “Service orientation” widely seen as the way to build grids LCG-2 & gLite Overview, Brisbane,January 2006

Service Orientation Service Oriented Architecture Components are loosely coupled by messages; messages define the interfaces – not reliant on a pre-existing client library Facilitates interoperability Allows easier compliance with upcoming standards, hosting environments, toolkits Architecture is not bound to specific implementations Heterogeneous resources (storage, computation…) Services can be deployed and used independently “Easily used to build new systems”… comparatively!! Facilitates development of clients for different architectures The gLite service decomposition has been largely influenced by the work performed in the LCG project Alert to the WSRF standardization initiatives but can’t wait for these! Start with plain WS (WS-I) LCG-2 & gLite Overview, Brisbane,January 2006

gLite LCG-2 & gLite Overview, Brisbane,January 2006

Evolution of EGEE middleware LCG Security GSI Job Management Condor + Globus CE, WN Logging & Bookkeeping Data Management LCG services Information & Monitoring BDII (evolution of MDS) Grid Access CLI + API Operating system Red Hat 7.3 gLite Security GSI and VOMS Job Management Condor + Globus + blahp CE, WN Logging & Bookkeeping Job Provenance Package management Data Management gLite-I/O + FiReMan Information & Monitoring R-GMA + Service Discovery Grid Access CLI + API + Web Services Easier installation / configuration Currently Scientific LINUX, will be available on others, incl. Windows VOMS: roles are supported in VO, proxy contains VO infomration unlike in GSI “blahp” another q manager. Jobs come via torgu R-GMA better w realtime R-GMA and VOMS are already used in LCG… a pattern that will continue with subsequent gLite services being adopted, LCG-2 & gLite Overview, Brisbane,January 2006

gLite components overview Near Future Access Services Grid Access Service API CLI now Security Services Information & Monitoring Services Authorization Auditing Information & Monitoring Job Monitoring Service Monitoring Authentication Dynamic Connectivity Service Discovery Data Services Job Management Services Metadata Catalog File & Replica Catalog Accounting Job Provenance Package Manager Site proxy – allows outbound connectivity from hidden networks Storage Element Data Movement Computing Element Workload Management Site Proxy LCG-2 & gLite Overview, Brisbane,January 2006

Summary gLite, the EGEE middleware: Is exiting prototyping phase and entering real production phase LHC first real data are under 2 years away from now! Implements a full and complete stack of grid services Migrating from LCG to gLite incrementally gLite development is seeking to balance Conforming to (emerging) standards Need to deliver a production service that demands efficiency, speed LCG-2 & gLite Overview, Brisbane,January 2006

Further information EGEE http://public.eu-egee.org/ gLite http://www.glite.org/ EGEE Middleware Architecture https://edms.cern.ch/document/594698/ LCG http://lcg.web.cern.ch/LCG/ LCG User Guide https://edms.cern.ch/file/454439//LCG-2-UserGuide.pdf User Scenario https://edms.cern.ch/file/498081//UserScenario2.pdf LCG-2 & gLite Overview, Brisbane,January 2006