1. EGEE Project and Middleware Overview
Marco Verlato
(INFN-Padova)
CYCLOPS First Training Workshop
11-13 April 2007
Bologna, Italy
Slides from recent EGEE talks and GILDA team

2. The EGEE Middleware: gLite GILDA – the grid we will use in practicals
Outline
The EGEE project
Motivations
Organisation
Infrastructure
The EGEE Middleware: gLite
GILDA – the grid we will use in practicals
Long-term view
Conclusions and useful links

3. The Grid metaphor Mobile Access G R Supercomputer, PC-Cluster I D M LW A
Supercomputer, PC-Cluster
Workstation
Data-storage, Sensors, Experiments
Visualising
Internet, networks

4. The EGEE project EGEE EGEE-II Objectives 1 April 2004 – 31 March 2006
71 partners in 27 countries,
federated in regional Grids
EGEE-II
1 April 2006 – 31 March 2008
91 partners in 32 countries
Objectives
Large-scale, production-quality grid infrastructure for e-Science
Attracting new resources and users from industry as well as science
Maintain and further improve “gLite” Grid middleware

5. EGEE Activities Service Activities Joint Research Activities
SA1 – Grid Operations, Support and Management (CERN)
SA2 – Networking Support (CNRS)
SA3 – Integration, Testing and Certification (CERN)
Joint Research Activities
JRA1 – Middleware Re-engineering (INFN)
JRA2 – Quality Assurance (CS-SI)
Networking Activities
NA1 – Management (CERN)
NA2 – Dissemination, Outreach and Communication (CERN)
NA3 – Training and Induction (UEdin)
NA4 – Application Identification and Support (CNRS)
NA5 – Policy and International Cooperation (GRNET)

6. EGEE Applications
Multitude of applications from a growing number of domains
Astrophysics
Computational Chemistry
Earth Sciences
Financial Simulation
Fusion
Geophysics
High Energy Physics
Life Sciences
Multimedia
Material Sciences
…..
Book of abstracts:
App. Deployment Plan:

7. EGEE operations - Operations Coord. Centre (OCC) - Regional Operations
Centres (ROC)
Front-line support for user and operations issues
Provide local knowledge and adaptations
One in each region – many distributed
Manage daily grid operations – oversight, troubleshooting, “Operator on Duty”
Run infrastructure services

8. EGEE Production Resources
EGEE today:
> 180 sites, 39 countries
> 30,000 processors, > 9 PB storage
> 30K jobs/day

9. EGEE Production Usage ~17.5 million jobs run (6450 cpu-years) in 2006;
Workloads of the “not HEP VOs” is now significant – approaching 8-10K jobs per day; and 1000 cpu-months/month
one year ago this was the overall scale of work for all VOs

10. EGEE Related projects & others Grids
Potential for linking ~80 countries

11. EU Related projects Name Description BalticGrid
EGEE extension to Estonia, Latvia, Lithuania
EELA
EGEE extension to Brazil, Chile, Cuba, Mexico, Argentina
EUChinaGRID
EGEE extension to China
EUMedGRID
EGEE extension to Malta, Algeria, Morocco, Egypt, Syria, Tunisia, Turkey
EU-IndiaGrid
EGEE extension to India
eIRGSP
Policies
ETICS
Repository, Testing
OMII-Europe
to provide key software components for building e-infrastructures;
BELIEF
Digital Library of Grid documentation, organisation of workshops, conferences
BIOINFOGRID
Biomedical
Health-e-Child
Biomedical – Integration of heterogeneous biomedical information for improved healthcare
ICEAGE
International Collaboration to Extend and Advance Grid Education

12. EGEE Middleware Distribution
gLite
Exploit experience and existing components from VDT (Condor, Globus), EDG/LCG, and others
Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Life Sciences are pilot applications)
Pluggable components – cater for different implementations
Follow SOA approach, WS-I compliant where possible
Focus is on re-engineering and hardening
Business friendly open source license
Moving to Apache-2

13. Parts of the Grid “ecosystem”
2001
EDG
Globus
MyProxy
Condor
...
VDT
. . .
LCG
OSG, …
DataTAG
CrossGrid
...
SRM
2004
GridCC
NextGrid
EGEE
DEISA …
interactive
USA EU
Used in
Future grids

14. gLite layered description
Service-oriented approach:
Lightweight services
Allow for multiple interoperable implementations
Easily and quickly deployable
Use existing services where possible
Applications
by R.Jones, 24/10/05
Higher-Level Grid Services
Workload Management
Replica Management
Visualization
Workflows
Grid economies
etc.
Provide specific solutions for supported applications
Host services from other projects
More rapid changes than Foundation Grid Middleware
Deployed as application software using procedure provided by grid operations
Foundation Grid Middleware
Security model and infrastructure
Computing (CE) & Storage Elements (SE)
Accounting
Information providers and monitoring
Application independent
Evaluate/adhere to new stds
Emphasis on robustness/stability over new functionality
Deployed as a software distribution by grid operations

15. gLite Grid Services Decomposition
Access
CLI
API
Security
Information & Monitoring
Authorization
Auditing
Information & Monitoring
Application
Monitoring
Authentication
Data Management
Job Management
Accounting
Metadata Catalog
File & Replica Catalog
Job Provenance
Package Manager
Storage Element
Data Movement
Computing Element
Workload Management
Overview paper

16. How can EGEE middleware support collaboration and resource sharing within and between many diverse VO’s ?

17. Grid Middleware When using a Grid you
When using a PC or workstation you
Login with a username and password (“Authentication”)
Use rights given to you (“Authorisation”)
Run jobs
Manage files: create them, read/write, list directories
Components are linked by a bus
Operating system
One admin domain
When using a Grid you
Login with digital credentials (“Authentication”)
Use rights given you (“Authorisation”)
Run jobs
Manage files: create them, read/write, list directories
Services are linked by the Internet
Middleware
Many admin domains
“services” = data storage, computers

18. Typical current grid INTERNET
At each site that provides computation:
Local resource management system
(= batch queue)
Condor
PBS
Torque …
EGEE term: queue is a “Computing element”
Grid middleware runs on each shared resource
Data storage
(Usually) batch queues on pools of processors
Users join VO’s
Virtual organisation negotiates with sites to agree access to resources
Distributed services (both people and middleware) enable the grid, allow single sign-on
INTERNET

19. Authorisation, Authentication (AA)
Users in many locations and organisations
Access services (“user interface”) : logon, upload credentials, run m/w commands
GRID SERVICES
Build on Grid Security Infrastructure
“Gate keeping”: map user’s credential to local user id / account
Resources in many locations and organisations
PBS, Condor, LSF,…
System software
NFS, …
Operating system
File system
Local scheduler
Hardware
HPSS, CASTOR…
Computing clusters,…
Network resources
Data storage

20. Basic job submission Tools that:
Users
Tools that:
copy files to and between CE’s and data storage
Submit job to a CE
Monitor job
Get output
How do I run a job on a compute element (CE) ? (CE =batch queue)
Resources
Compute elements
Network resources
Data storage

21. Information service (IS)
Users
Information Service (IS):
Resources send updates to IS
Grid services query IS before running jobs
How do I know which CE could run my job?
Which is free?
Resources
Compute elements
Network resources
Data storage

22. File management Storage Transfer Replica management
Users
Storage
Transfer
Replica management
My data are in files, and I’ve terabytes
We’ve terabytes of data in files.
Our data are in files, and I’ve terabytes
EGEE data: primarily file-based
services for databases used by some VO’s, not in release yet
Resources
Compute elements
Network resources
Data storage

23. Main components
User Interface (UI): The place where users logon to the Grid
Resource Broker (RB): Matches the user requirements with the available
resources on the Grid
Information System: Characteristics and status of CE and SE (Uses “GLUE schema”)
Computing Element (CE): A batch queue on a site’s computers where
the user’s job is executed
Storage Element (SE): provides (large-scale) storage for files

24. Job submission example
voms-proxy-init
glite-job-submit myjob.jdl
Myjob.jdl
Executable = “gridTest”;
StdError = “stderr.log”;
StdOutput = “stdout.log”;
InputSandbox = {“/home/joda/test/gridTest”};
OutputSandbox = {“stderr.log”, “stdout.log”};
InputData = “lfn:testbed ”;
DataAccessProtocol = “gridftp”;
Requirements = other.Architecture==“INTEL” && \ other.OpSys==“LINUX”;
Rank = “other.GlueHostBenchmarkSF00”;
User Interface
Information
Service
Input Sandbox
JDL
Replica
Catalog
JDL
Job Submit
Event
Author.
Service
Output Sandbox
Input Sandbox
Resource
Broker
Output Sandbox
Storage
Element
Job
Logging &
Book-keeping
Job Status
GSI data acc/transf
Job Submission
Service
Computing
Element

25. t-Infrastructure Why t-infrastructure?
Training is necessary!
e-Infrastructure for production
t-Infrastructure for training
Need guaranteed response for tutorials; limit the vulnerability of production systems
use training grid
have training CA
able to change middleware to prepare participants for future releases on production system
need safe resources for installation training
easy entry point for new communities

26. The GILDA t-Infrastructure (https://gilda.ct.infn.it)
19 sites in 3 continents
> 3000 certificates issued, >15% renewed at least once
> 100 tutorials and demos perfor-med in 23 months
> 1,000,000 hits (> 50,000 unique visits) on (of) the web site from 10’s of different countries
> 0.6 TB of training material downloaded from the web site

27. The Future of Grids
Increasing the number of infrastructure users by increasing awareness
Dissemination and outreach
Training and education
Grids offer new opportunities for collaborative work
Increasing the number of applications by improving application support and middleware functionality
Increase stability, scalability, and usability
Major efforts needed particularly on VO management, security infrastructure, data management, and job management
High level grid middleware extensions
Increasing the grid infrastructure
Increase manageability of Grid services
Reducing the cost of operation
Ensuring interoperability between infrastructures
Protecting user investments
Better involvement of industry
Move towards a sustainable grid infrastructure

28. Sustainability: Beyond EGEE-II
Need to prepare for permanent Grid infrastructure
Ensure a reliable and adaptive support for all sciences
Independent of short project funding cycles
Infrastructure managed in collaboration with national grid initiatives

29. Summary
EGEE operates the world’s largest multi-disciplinary grid infrastructure for scientific research
In constant and significant production use
gLite 3.0 production middleware
Upon which can be built toolkits and services for new application communities
Supporting effective new user communities
t-Infrastructure for training
GILDA
Next 2 years: progress towards
Federated Grid
Sustainable organisation

30. Further information EGEE’07 Conference: 1-5 October 2007, Budapest
NEW!!!
EGEE’07 Conference: 1-5 October 2007, Budapest
EGEE User Forum, 7-9 May 2007, Manchester
EGEE digital library:
Needs certificate (GILDA or national CA in browser)
EGEE
gLite
LCG
Open Grid Forum
Globus Alliance
VDT