1. gLite and LCG-2: an overview
Dr. Mike Mineter
National e-Science Centre, Edinburgh / UK
Dr. Rüdiger Berlich,
Forschungszentrum Karlsruhe / Germany
Brisbane, 2 February 2006
Slides contributed by EGEE Team

2. Outline Grid concepts The LCG/EGEE relationship Background to gLite
gLite services
gLite status
LCG-2 & gLite Overview, Brisbane,January 2006

3. Grid concepts
LCG-2 & gLite Overview, Brisbane,January 2006

4. A multi-VO Grid
EGEE is establishing a production grid infrastructure to support multiple, diverse VO’s
User Interface
User Interface
Grid services
Blue VO: people cooperating and sharign resources in different places. Similarly the red VO.
The resources that have been made available to the red VO are circled in red; similarly for blue.
LCG-2 & gLite Overview, Brisbane,January 2006

5. A multi-VO grid INTERNET Users join VOs
Virtual organisation contributes resources & negotiates access
Grid middleware runs on each resource
“Storage elements”
“Compute elements”
Additional services (both people and grid middleware) enable the grid
Effect: “virtual computing” across administrative domains empowering collaboration
INTERNET
LCG-2 & gLite Overview, Brisbane,January 2006

6. VO’s and middleware Application Application toolkits, standards
Basic Grid services: AA, job submission, info, …
Middleware: “collective services”
Application toolkits, standards
Application
The tools, services used by the VO’s applications
Community-specific standards
Application development environment, portals, semantics, workflow
In EGEE: Mainly VO-specific
Emphasis of EGEE middleware
Workflow –WILL allow
user to “see” an application,
the developer to code components,
and these to be assembled into distributed services
Different organisational structures and different application types imply different middleware fucntionality:
So short-lived operations “supercharging” a GIS require interactive performance – not the batch oriented pattern that mainly characterises Grids
LCG-2 & gLite Overview, Brisbane,January 2006

7. An EGEE-centric view of the middleware “ecosystem”
2001
EDG
Globus
MyProxy
Condor
...
VDT
. . .
LCG
OSG, …
DataTAG
CrossGrid
...
SRM
2004
GridCC
NextGrid
EGEE
DEISA …
This slide does not include many nationally funded initiatives – neither is it comprehensive in showing US and EU projects.
Its here partly so that the relationship between VDT, LCG and EGEE – the ones heard most often – are seen, and are seen to be a small part of a complex story. The projects shown are not the only ones of importance.
Grid middleware has evolved very fast, in many places. It entails different attempts to solve a range of problems; these tools need to become interoperable.
It leaves Grid middleware in need of a view from a step back, and it has come in the convergence of Grids with web services.
The development of Grid computing spans oceans – so the Univ of Wisconsin is one of two sites running the new prototype middleware for EGEE now. The scale of cooperation demands standards exist. The last years have begun to see these standards for Grids build on and with those from web services.
USA EU
Used in
Future grids
With apologies to many missing projects!
LCG-2 & gLite Overview, Brisbane,January 2006

8. Computation services in EGEE
Jobs are:
Run from batch queues, termed “computing elements” CE’s
Described in “Job Description Language” files
gLite also supports
Interactive jobs
Jobs run in batch mode – “listener” receives messages from CE
Parallelism using MPI
MPI jobs can run on CE’s that support MPI not across administrative domains (not MPICH-G)
Workflow (DAGs, from Condor)
Checkpointing
Partitioned jobs (soon) – e.g. Monte-Carlo
LCG-2 & gLite Overview, Brisbane,January 2006

9. Data services in EGEE Simple data Structured data
Files
Requires
Replica files
Move data to computation
Virtual filesystems
Metadata for files
File transfer
These services are amongst those provided by EGEE
Structured data
RDBMS, XML databases
Require extendable middleware tools to support
computation near to data
easy access, controlled by AA
integration and federation
Hence OGSA-DAI DAI: Data Access and Integration
OGSA-DAI is NOT currently being ported to gLite
Is being used in earth science VO however – see website from EGEE conference, Pisa, 2005
LCG-2 & gLite Overview, Brisbane,January 2006

10. Current production middleware LCG-2
LCG-2 & gLite Overview, Brisbane,January 2006

11. LCG and EGEE LCG : Large Hadron Collider Compute Grid
EGEE committed to “hit the ground running”
EGEE profits from the resources - no funded computing/data resources in EGEE
Provided by the VOs
LCG obtains additional production and operation efforts
LCG experiments now comprise several of the many VOs in EGEE
Current service (“LCG-2”) based on work done in LCG
Middleware components to be upgraded by “gLite” services as they are proven
“gLite 3” will be forged from LCG gLite services
LCG : Large Hadron Collider Compute Grid
LCG-2 & gLite Overview, Brisbane,January 2006

12. Current production grid: LCG-2
Computing cluster
Network resources
Data storage
Operating system
Local scheduler
File system
User access
Security
Data transfer
Information schema
Workload management
Data management
App monitoring system
User interfaces
Applications
Hardware
System software
“Basic” services
“Collective” services
Application level services
HPSS, CASTOR…
Scientific Linux
NFS, …
PBS, Condor, LSF,Torque…
VDT (Condor, Globus, GLUE)
EU DataGrid
Information system
LCG-2 & gLite Overview, Brisbane,January 2006

13. Current production m’ware: LCG-2
Replica
Catalogue
Logging &
Book-keeping
Resource
Broker
Storage
Element
Computing
Element = a batch queue
Information
Service
Job Status
DataSets info
Author.
&Authen.
Job Submit Event
Job Query
Input “sandbox”
Input “sandbox” + Broker Info
Output “sandbox”
Publish
SE & CE info
“User interface”
LCG-2 & gLite Overview, Brisbane,January 2006

14. Background to gLite
LCG-2 & gLite Overview, Brisbane,January 2006

15. If “The Grid” vision leads us here…
Annapurna range – photo by Mike
… then where are we now?
LCG-2 & gLite Overview, Brisbane,January 2006

16. Grids: where are we now? Many key concepts identified and known
Many grid projects have tested, and benefit from, these
Major efforts now on establishing:
Standards (a slow process) (e.g. Global Grid Forum, , OASIS, W3C, IETF )
Production Grids for multiple VO’s – EGEE the most international and biggest by far
“Production” = Reliable, sustainable, with commitments to quality of service
One stack of middleware that serves many communities
Operational procedures and services (people, policy,..)
New user communities
From physics via bioinformatics, chemistry, earth science towards infrastructure for widest spectrum of collaboration
… whilst research & development continues
“Service orientation” widely seen as the way to build grids
LCG-2 & gLite Overview, Brisbane,January 2006

17. Service Orientation Service Oriented Architecture
Components are loosely coupled by messages; messages define the interfaces – not reliant on a pre-existing client library
Facilitates interoperability
Allows easier compliance with upcoming standards, hosting environments, toolkits
Architecture is not bound to specific implementations
Heterogeneous resources (storage, computation…)
Services can be deployed and used independently
“Easily used to build new systems”… comparatively!!
Facilitates development of clients for different architectures
The gLite service decomposition has been largely influenced by the work performed in the LCG project
Alert to the WSRF standardization initiatives but can’t wait for these!
Start with plain WS (WS-I)
LCG-2 & gLite Overview, Brisbane,January 2006

18. gLite
LCG-2 & gLite Overview, Brisbane,January 2006

19. Evolution of EGEE middleware
LCG
Security
GSI
Job Management
Condor + Globus
CE, WN
Logging & Bookkeeping
Data Management
LCG services
Information & Monitoring
BDII (evolution of MDS)
Grid Access
CLI + API
Operating system
Red Hat 7.3
gLite
Security
GSI and VOMS
Job Management
Condor + Globus + blahp
CE, WN
Logging & Bookkeeping
Job Provenance
Package management
Data Management
gLite-I/O + FiReMan
Information & Monitoring
R-GMA + Service Discovery
Grid Access
CLI + API + Web Services
Easier installation / configuration
Currently Scientific LINUX, will be available on others, incl. Windows
VOMS: roles are supported in VO, proxy contains VO infomration unlike in GSI
“blahp” another q manager.
Jobs come via torgu
R-GMA better w realtime
R-GMA and VOMS are already used in LCG… a pattern that will continue with subsequent gLite services being adopted,
LCG-2 & gLite Overview, Brisbane,January 2006

20. gLite components overview
Near Future
Access Services
Grid Access Service
API
CLI
now
Security Services
Information & Monitoring Services
Authorization
Auditing
Information & Monitoring
Job
Monitoring
Service
Monitoring
Authentication
Dynamic
Connectivity
Service
Discovery
Data Services
Job Management Services
Metadata Catalog
File & Replica Catalog
Accounting
Job Provenance
Package Manager
Site proxy – allows outbound connectivity from hidden networks
Storage Element
Data Movement
Computing Element
Workload Management
Site Proxy
LCG-2 & gLite Overview, Brisbane,January 2006

21. Summary gLite, the EGEE middleware:
Is exiting prototyping phase and entering real production phase
LHC first real data are under 2 years away from now!
Implements a full and complete stack of grid services
Migrating from LCG to gLite incrementally
gLite development is seeking to balance
Conforming to (emerging) standards
Need to deliver a production service that demands efficiency, speed
LCG-2 & gLite Overview, Brisbane,January 2006

22. Further information EGEE http://public.eu-egee.org/
gLite
EGEE Middleware Architecture
LCG
LCG User Guide
User Scenario
LCG-2 & gLite Overview, Brisbane,January 2006