1. Slides contributed by EGEE Team
EGEE Middleware
Valeria Ardizzone
INFN Catania
Roma, April 2006
Slides contributed by EGEE Team

2. Outline Grid “ecosystem” Middleware in EGEE Processes and Releases
Summary
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

3. The Grid “ecosystem” OSG, … SRM Used in 2001 2004 EDG Globus MyProxy
Condor
...
VDT
. . .
LCG
OSG, …
DataTAG
CrossGrid
...
SRM
This slide does not include many nationally funded initiatives – neither is it comprehensive in showing US and EU projects.
Its here partly so that the relationship between VDT, LCG and EGEE – the ones heard most often – are seen, and are seen to be a small part of a complex story. The projects shown are not the only ones of importance.
Grid middleware has evolved very fast, in many places. It entails different attempts to solve a range of problems; these tools need to become interoperable.
It leaves Grid middleware in need of a view from a step back, and it has come in the convergence of Grids with web services.
The development of Grid computing spans oceans – so the Univ of Wisconsin is one of two sites running the new prototype middleware for EGEE now. The scale of cooperation demands standards exist. The last years have begun to see these standards for Grids build on and with those from web services.
GridCC
NextGrid
2004
EGEE
DEISA …
USA EU
Used in
Future grids
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

4. Middleware in EGEE

5. Middleware in EGEE Applications Higher-Level Grid Services
by R.Jones, 24/10
Provide specific solutions for supported applications
Host services from other projects
More rapid changes than Foundation Grid Middleware
Deployed as application software using procedure provided by grid operations
Higher-Level Grid Services
Workload Management
Replica Management
Visualization
Workflows
Grid economies
etc.
application independent “Grid Foundation Middleware”
all services that need to be deployed on a production Grid infrastructure in order to provide a consistent, dependable service. It can be regarded as the “Middleware Infrastructure”.
higher level “Grid Services”.
comprise higher level services that certain, but not all, VOs require.
Foundation Grid Middleware
Security model and infrastructure
Computing (CE) & Storage Elements (SE)
Accounting
Information providers and monitoring
Application independent
Evaluate/adhere to new stds
Emphasis on robustness/stability over new functionality
Deployed as a software distribution by grid operations
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

6. Grid Foundation Middleware (1/2)
A Service Oriented Architecture (SOA) approach is essential to allow different Grid foundation services to interface with services not developed inside EGEE-II.
Adherence to widely accepted international standards (WSRF)
Components
Security model and infrastructure
define and enforce policies within VOs as well as those needed by the resource provider (i.e. between VOs).
resource access control, resource access auditing and VO membership management.
policy definition and enforcement, dynamic connectivity, auditing, and interoperability with Shibboleth-based authentication and authorisation infrastructures.
Computing Element (CE)
set of services that provide homogenous, managed, and secure access to heterogeneous, remote computing resources.
to directly submit jobs or to dynamically deploy VO specific schedulers.
collaboration w/ Condor and Globus.
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

7. Grid Foundation Middleware (2/2)
Storage Element (SE)     
provides homogeneous access to storage resources, including managed data transfer.
POSIX-like I/O and FTS.
Accounting
collecting the relevant information locally at the resources and making it available at a global or VO level (in a secure manner) for statistical, billing, or scheduling purposes.
Prototype.
Information and monitoring
Information on resources must be accessible to other services in a dependable and timely manner.
Supporting: monitoring of resources and allow user level monitoring (information and monitoring infrastructure, and service discovery).
Allowing free information flow across different Grid flavours.
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

8. Middleware in EGEE Applications Higher-Level Grid Services
by R.Jones, 24/10
Provide specific solutions for supported applications
Host services from other projects
More rapid changes than Foundation Grid Middleware
Deployed as application software using procedure provided by grid operations
Higher-Level Grid Services
Workload Management
Replica Management
Visualization
Workflows
Grid economies
etc.
application independent “Grid Foundation Middleware”
all services that need to be deployed on a production Grid infrastructure in order to provide a consistent, dependable service. It can be regarded as the “Middleware Infrastructure”.
higher level “Grid Services”.
comprise higher level services that certain, but not all, VOs require.
Foundation Grid Middleware
Security model and infrastructure
Computing (CE) & Storage Elements (SE)
Accounting
Information providers and monitoring
Application independent
Evaluate/adhere to new stds
Emphasis on robustness/stability over new functionality
Deployed as a software distribution by grid operations
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

9. Grid Services
Grid Services comprise higher level services typically VO specific
exploit the Grid Foundation Middleware for achieving their purpose.
Grid Services not only from within EGEE-II
Produce a middleware infrastructure that is general and capable of hosting Grid Services coming from other sources and projects.
Services:
workload management services
Grid scheduler like functionality
logging, bookkeeping, and job provenance services
keep track of actions performed at the Grid level
replica management
services that reliably schedule data movement and catalog updates,
visualization services
for various information collected on the Grid,
workflow services
application specific abstraction of the workflow,
Grid economies
economy based scheduling, advanced reservation, etc.
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

10. gLite Grid Middleware Services
Access
CLI
API
Security Services
Authorization
Information & Monitoring Services
Information & Monitoring
Application
Monitoring
Auditing
Authentication
The gLite Grid services follow a Service Oriented Architecture:
facilitate interoperability among Grid services
allow easier compliance with upcoming standards
Architecture is not bound to specific implementations
services are expected to work together
services can be deployed and used independently
The gLite service decomposition has been largely influenced by the work performed in the LCG project
Data Management
Workload Mgmt Services
Metadata Catalog
File & Replica Catalog
Job Provenance
Package Manager
Accounting
Storage Element
Data Movement
Computing Element
Workload Management
Connectivity
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

11. Process and Releases

12. gLite Software Process
Development
Integration
Testing
Deployment Packages
Software Code
Fail
Pass
Testbed Deployment
Integration Tests
EMT defines release contents
Based on work plan progress
Based on essential items needed
so far mainly for the HEP experiments, BioMed and Operations needs
Decide on target dates for tags
Taking into account enough time for integration & testing
Integration Team
produces Release Candidates
based on received tags
Performs
Build, Smoke Test, Deployment Modules
Iterate with developers
Once Release Candidate passed functional tests
produces documentation, release notes and final packaging
announce the release on the glite Web site and the glite-discussion mailing list.
Testing Team
Test Release candidates on a distributed testbed
CERN, RRZN Uni Hannover, Imperial College
Raise bugs as needed
Iterate with Integrators & Developers
…..
Fix
Fail
Functional Tests
Pass
Installation Guide, Release Notes, etc
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

13. File Placement Service
gLite Releases
gLite 1.1.2
Special Release for SC
File Transfer Service
gLite 1.4.1
Service Release
~60 Defect Fixes
gLite 1.1.1
Special Release for SC
File Transfer Service
gLite 1.5
Shallow resub. in WMS
GPBOX
gLite I/O encryption
Amga
Hydra
gLite 1.3
File Placement Service
FTS multi-VO
Refactored RGMA & CE
gLite 1.4
VOMS for Oracle
SRMcp for FTS
WMproxy
LBproxy
DGAS
gLite 1.0
Condor-C CE
gLite I/O
R-GMA
WMS
L&B
VOMS
Single Catalog
gLite 1.2
File Transfer Agents
Secure Condor-C
gLite 1.1
File Transfer Service
Metadata catalog
QF1.4.1__33_2005
QF1.4.1__34_2005
QF1.4.1__31_2005
QF1.4.1__32_2005
QF1.3.0_22_2005
QF1.4.1__27_2005
QF1.3.0_20_2005
QF1.3.0_21_2005
QF1.4.1__28_2005
QF1.4.1__29_2005
QF1.1.2_11_2005
QF1.3.0_19_2005
QF1.4.1__30_2005
Functionality
QF1.4.1__26_2005
QF1.1.0_09_2005
QF1.3.0_18_2005
QF1.4.1__25_2005
QF1.0.12_04_2005
QF1.1.0_10_2005
QF1.3.0_17_2005
QF1.1.0_07_2005
QF1.0.12_02_2005
QF1.1.0_08_2005
QF1.1.2_16_2005
QF1.0.12_03_2005
QF1.1.0_05_2005
QF1.1.2_13_2005
QF1.1.0_06_2005
QF1.2.0_14_2005
QF1.0.12_01_2005
QF1.2.0_15_2005
QF1.1.2_12_2005
QF1.3.0_24_2005
QF1.3.0_23_2005
April 2005
May 2005
June 2005
July 2005
Aug 2005
Sep 2005
Oct 2005
Nov 2005
Dec 2005
Jan 2006
Feb 2006
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

14. After gLite 1.5.0
Converge from LCG and gLite to a single middleware stack called gLite. The first version will be gLite 3.0.0
Process controlled by the Technical Coordination Group
gLite and LCG have been the last independent releases
Components in gLite 3.0.0
Certified:
All components already in LCG plus upgrades
this already includes new versions of VOMS, R-GMA and FTS
The Workload Management System (with LB, CE, UI) of gLite 1.5.0
Tested to some degree and with limited deployment support:
The DGAS accounting system
Data management tools as needed by the Biomed community
Hydra, AMGA, secure access to data
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

15. Schedule for gLite 3.0.0 After gLite 3.0.0:
February
March
April
May
June
Certification
PPS
Deploy in prodn
PRODUCTION!!
Thursday 1/6/06
LCG Service Challenge 4
(SC4) starts!!
Tuesday 28/2/06
gLite 3.0.0β exits certification and enters pre-production
Wednesday 15/3/06
gLite 3.0.0β available to users in the PPS
Deployment of gLite 3.0.0β in PPS
Continual bug fixing and patches
passed to PPS
Friday 28/4/06
PPS phase ends.
gLite passes from PPS to Production.
After gLite 3.0.0:
March 31st: code freeze for development release gLite 3.1.0
April 30th: end of integration
May 31st: end of certification. Deployment on PPS
July 31st: release of production version gLite Start deployment at sites
September : gLite installed at sites and usable.
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

16. Summary Complete middleware stack
security infrastructure
information system and monitoring
workload management
data management
Developed according to a well defined process
Controlled by the EGEE Technical Coordination Group
gLite will be available on the production infrastructure in less than 2 months
Development is continuing to provide increased robustness, usability and functionality
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome

17.
Valeria Ardizzone, INFN Catania, EGEE gLite Tutorial, April 2006 Rome