1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org EGEE Middleware reengineering Claudio Grandi – JRA1 Activity Manager - INFN EGEE Final EU Review 23-24 May 2006 www.glite.org

2. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 2 Content Main accomplishments Feedback on reviewers recommendation Future look and issues Summary

3. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 3 Main accomplishments Release of gLite 1.5 (January) Quick-fixes to previous releases Started migration of the build system to ETICS –ETICS project started in January Reorganization of the work according to the new process –Technical Coordination Group and Task Forces (see the following) –Fusion of the JRA1 testing and the SA1 certification teams Convergence of gLite 1.5 and LCG 2.7.0 that was on the production infrastructure to a unique middleware stack –Major effort since LCG 2.7.0 and gLite 1.5 were developed in not completely consistent environments Release of gLite 3.0 (May) –Version number is 3.0 because it is higher than 1.5 and 2.7.0 Development for the next gLite releases

4. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 4 SA3 Testing & Certification Functional Tests Testbed Deployment gLite Software Process JRA1 Development Software Error Fixing SA3 Integration Deployment Packages Integration Tests Installation Guide, Release Notes, etc SA1 Pre- Production Scalability Tests Pre-Production Deployment Fail Pass SA1 Production Infrastructure Release Problem Serious problem Directives

5. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 5 Integration and Testing Testing activity staffing now appears to be adequate: –There is now a unique middleware stack to test and deploy –The Pre-Production System has been recovered to its original role: test the software that is supposed to go on the Production System –The JRA1 testing and SA1 certification teams have been merged Integration migrated to use the ETICS tools –ETICS system based on the EGEE JRA1 experience!!! Integration and testing are now in the SA3 activity

6. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 6 Middleware structure Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory Foundation Grid Middleware will be deployed on the EGEE infrastructure –Must be complete and robust –Should allow interoperation with other major grid infrastructures –Should not assume the use of Higher-Level Grid Services

7. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 7 Middleware components Some components are on the production infrastructure since a long time and have been used for applications challenges. They are all in gLite 3.0 Some components are in gLite 3 with limited support: –Didn’t go through the full certification process –Clients are deployed on the infrastructure, while servers should be deployed by the interested users Some components are not in gLite 3.0. Development continues until maturity

8. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 8 Grid Foundation: Security, Accounting VOMS and VOMSAdmin Provide a way to add attributes to a certificate proxy. Enables VO policies In gLite 3.0 Other Security components See JRA3 presentation DGASCollects, stores and transfers accounting data. Compliant with privacy requirements Not in gLite 3.0. Will be deployed during the summerAPELUses R-GMA to propagate and display job accounting information for infrastructure monitoring and reporting In gLite 3.0

9. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 9 Grid Foundation: Information Systems BDIIInformation System by LCG based on LDAP. In gLite 3.0 R-GMAProvides a uniform method to access and publish distributed information and monitoring data In gLite 3.0 used for job monitoring Service Discovery Provides a standard set of methods for locating Grid services. Currently supports R-GMA, BDII and XML files as backends In gLite 3.0 used by WMS and DM components CEMonWeb service to publish the status of a computing resource and of running jobs. In gLite 3.0, but the WMS uses the BDII

10. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 10 Grid Foundation: Resource access LCG-CEComputing Element based on GT2, by LCG. To be replaced by gLite-CE In gLite 3.0 gLite CE Computing Element based on GSI enabled Condor-C. Deployed in parallel to the LCG-CE In gLite 3.0 CREAMLightweight Computing Element with web service interface Not in gLite 3.0 BLAHLayer between the CE and the local batch system In gLite 3.0, in the gLite CE Storage Elements DPMdCacheCastor Use solutions developed by LCG and other projects: DPM, dCache, Castor all with SRM interface In gLite 3.0 GFALPosix-like file access by LCGIn gLite 3.0 gLiteIOPosix-like file access. To be replaced by GFAL Limited support in gLite 3.0

11. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 11 High level data management services LFCFile Catalog by LCGIn gLite 3.0 FiReManFile Catalog. To be replaced by LFCLimited support in gLite 3.0 AMGAGeneral purpose Metadata Catalog. Developed with NA4 Limited support in gLite 3.0. Full support foreseen HydraKey-store for data encryption. Used by gLiteIO, will be used also by GFAL Limited support in gLite 3.0. Full support foreseen File Transfer System Reliable, scalable and customizable File Transfer System. Supports “retries”, channel management (authZ, VO shares) and VO plug-ins In gLite 3.0. Used under stress for Service Challenges!

12. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 12 High level job management services LCG-RBResource brokering, input/output management based on GT2, by LCG. To be replaced by gLite-WMS In gLite 3.0 gLite-WMSResource brokering, input/output management, workflow management). WMProxy Web service interface (WMProxy) Deployed in parallel with the LCG-RB Is in gLite 3.0. ICESubmission to CREAM CEs in the gLite WMS. Is not in gLite 3.0 Logging and Bookkeeping Tracks jobs during their lifetimeIn gLite 3.0 Job Provenance Long term job information storage. Allows job re-run Is not in gLite 3.0 GPBOXManages and distributes VO policies for resource access based on VOMS Is not in gLite 3.0

13. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 13 Feedback to recommendation 26 “Have more direct pathways and programs where the teams from developers, testers, infrastructural providers, and above all, application developers and users spend focused efforts to identify the usage and concerns with the current LCG-2 and gLite, instead of relying on a fairly long pathway from the application end to the development” Adoption of the process a-la EGEE II in fall 2005 –Process controlled by the Technical Coordination Group (TCG) keeps the development focused –Task Forces with applications, developers, testers and deployment experts Convergence of the LCG and gLite middleware stacks –gLite 3.0 released in May  On the Pre-Production System (PPS) and exposed to the users in mid-March  Being deployed on the Production System now

14. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 14 Feedback to recommendation 27 “Continuously assess user application feedback, especially in the light of introduction of new services, in order to be able to judge whether continued investment into the R&D of that particular feature would have high return on its value” Achieved with the creation of TCG and Task Forces –Prioritization of the inclusion of gLite 1.5 and LCG 2.7.0 components into gLite 3.0 –Definition of high priority additions for the next releases –Feed-back from the Task Forces about functionality and performance of the components deployed on the PPS Still to be done: –Creation of a “preview” system when new components may be exposed to the users at an early stage of development  Early feedback before effort is spent in integration and testing

15. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 15 Feedback to recommendation 28 “Clarify and advertise a more conservative (in term of time span) and comprehensive release cycle plan for gLite” After gLite 3.0 adopt a continuous release process –No more big-bang releases with fixed deadlines for all –Develop components as requested by users and sites –Deploy or upgrade as soon as testing is satisfactory Major releases synchronized with large scale activities of VOs (SCs) –Include what is available at that time Support more versions of client code Keep servers backward compatible as much as possible The release is handled by SA3 in the EGEE II process –Integration process using ETICS tools –The new testing team includes the former JRA1 testing and SA1 certification teams

16. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 16 Feedback to recommendation 29 “Revise the gLite development process to fully integrate the Technical Coordination Group and application developers” Started in fall 2005 with TCG and Task Forces

17. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 17 Feedback to recommendation 30 “ Investigate the deliverables of other international grid R&D activities and identify where deliverables could be shared in a mutually collaborative fashion to achieve rapid international interoperations with grids outside of EU” Design Team meetings –Include Condor, Globus and OSG experts Frequent bi-lateral meetings –OSG, NAREGI, NorduGrid Joint efforts: –GLUE (OSG, Globus, NorduGrid) –glexec (GridPP, Condor) –CREAM (OMII-EU) Components developed by other projects, often with EGEE support: –Build system (ETICS) –Condor-C / gLite CE (Condor) –MyProxy (Univ. of Illinois) –VOMRS (Fermilab) Components developed by EGEE included in other projects toolkits: –VOMS (VDT) –MyProxy/VOMS interface (VDT) –BLAHPD (VDT) –CEMon (VDT) –L&B (will be in VDT) –LCMAPS (Globus) –AAA toolkit (Globus) Note that development with external projects that do not have commitments on the development or simply different time scales may be very difficult

18. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 18 Feedback to recommendation 31 “Identify in the middleware stack which parts of gLite is “conformant” to standards activities within GGF and where it is currently not“ Standards adoption –Security:  Supporting international standards (mainly form OGSA-Auth) either directly or through plug-ins –Data Management:  Standard interfaces for SE (SRM) –Workload Management:  SOAP, WSDL and WS-I used to access services; building JDL to JSDL translator –Resource description:  GLUE schema is widely used; information exchange between GLUE and CIM under study –Information systems:  Specification defined within GGF- INFOD –Accounting  Extends GGF-UR specifications –Networking  NPM compliant with GGF-NM New proposed standards –VOMS Attribute Certificate –CREAM contribution to GGF-BES –FTS contribution to GGF-DAIS Not fully compliant with WS-* –May be disruptive –Adopting standards too early is dangerous Note that some standards are newer than the corresponding gLite middleware –Difficult to adopt a new standard without breaking the existing functionalities

19. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 19 Future look and issues Support for middleware deployed on the production infrastructure –Bug fixing, second-line user support Improve security, robustness and usability Participate to TCG Task Forces –Direct application support and gathering of user requirements...and following TCG decisions: –Bring components not yet included in release to maturity –Improve functionality –Improve the compliance to international standards Issue: inclusion of improvements and new functionalities vs stability of the code and of the production infrastructure  Critical role of the TCG!!!

20. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 20 Summary EGEE provides a comprehensive middleware stack –gLite 3.0 will be available on the production infrastructure in June Developed according to a well defined process –Controlled by the EGEE Technical Coordination Group –Steered by applications Activities of the integration team lead to the creation of the ETICS project Many gLite components are included in other toolkits (e.g. VDT) and used in other infrastructures

21. Enabling Grids for E-sciencE INFSO-RI-508833 EGEE Final EU Review 23-24 May 2006 - JRA1 - Claudio Grandi 21 www.glite.org