LCG 4-Feb-2004GridPP-9 : EdinburghSlide 1 ARDA Architectural Roadmap towards Distributed Analysis

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 2 What is ARDA ? Is it middleware (e.g. “second generation” ) ? Is it a common application layer above middleware ? Is it a co-ordination project ? Is it about distributed analysis or wider HEP grid use ? Is it “All things to all men” ? Is it the solution to all our (Grid) problems ? Possibly ALL the above …and more !! Does any one understand what this all means ? A call for volunteers !!

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 3 Outline History –HEPCAL I –GAG and HEPCAL II ARDA Research & Technology Assessment Group –Set up Spring 2003; Reported late Autumn 2003 –ToR –Survey –Conclusion –Recommendations Workshop (21 st /22 nd Jan 2004) Status

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 4 Some History

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 5 What we want from a GRID (EDG meeting, NIKHEF, March 2001, © fca 2004 ) OS & Net services Bag of Services (GLOBUS) GRID middleware HEP VO common application layer Earth Obs.Biology ALICEATLASCMSLHCb Specific application layer WP9WP 10 March

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 6 What we have (HEPCAL presentation, CHEP 2003, © fca 2004 ) OS & Net services Bag of Services (GLOBUS) Middleware ALICEATLASCMSLHCb Specific application layer WP1WP2WP3WP4WP5 Semantic gap

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 7 A proposal (HEPCAL presentation, CHEP 2003, © fca 2004 ) OS & Net services Bag of Services (GLOBUS) Specific application layer DataGRID middleware WP1WP2WP3WP4WP5 Common use cases VO common application layer If we manage to define Middleware WP1WP2WP3WP4WP5 It will be easier for them to arrive at ALICEATLASCMSLHCb

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 8 HEPCAL II (If HEPCAL I was Batch, then HEPCAL II is Analysis) Analysis Execution Models –Support for queries by common layers and at dataset level –Support for job pipelines Users, Groups, Quotas and Permissions Interactive.vs. Batch Grid Activity System Requirements –Provenance & Job Traceability –Log Book and Reports –Persistent Interactive Environment –Analysis Software deployment Use cases (as sequence of user operations) –Production Analysis –(Sub-)Group level Analysis –End-user level Anlaysis

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 9 Enter ARDA

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 10

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 11 RTAG ToR Motivation: To agree on requirements as laid out in a first step by recent work within the GAG and identify commonalities within the current projects that might allow the LCG (both in the AA and GTA areas) to provide a focus of effort. To provide guidance to the LCG on future Middleware development directions and interfacing work to match the experiment requirements To build on the richness of the current technical solutions to avoid duplication of efforts To clearly identify the roles and responsibilities of the components/layers/ services in the experiment DA planning To give guidance to the community on the expected division of work between the experiments, the LCG and the external projects.

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 12 RTAG ToR Mandate: To review the current Distributed Analysis (DA) activities and to capture their architectures in a consistent way To confront these existing projects to the HEPCAL II use cases and the user's potential work environments in order to explore potential shortcomings To consider the interfaces between Grid, LCG and experiment-specific services –Review the functionality of experiment-specific packages, state of advancement and role in the experiment –Identify similar functionalities in the different packages –Identify functionalities and components that could be integrated in the generic Grid middleware To confront the current projects with critical Grid areas To develop a roadmap specifying wherever possible the architecture, the components and potential sources of deliverables to guide the medium term (2 year) work of the LCG and the DA planning in the experiments

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 13

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 14 The Report Reviewed existing projects –PROOF and the Grid –AliEn – web services (ALICE) ** –Clarens – web services (CMS) –DIAL (ATLAS) – workflow –GANGA (LHCb/ATLAS) – high level job submission –DIRAC (LHCb) – distributed MC production ** selected for further analysis (best meets requirements of experiment, used in anger by an experiment…) The ARDA Blueprint –Service Descriptions + APIs (access to services) Information, Authentication, Authorisation, Audit, Accounting, Workload Management, Job Provenance, File Catalogue, Metadata Catalogue, Data management, Site Gatekeeper, Storage Element, Computing Element, Job Monitoring, Package Manager, Grid Monitoring

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 15 Example : AliEn

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 16 Example : AliEn

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 17 Example : Clarens Web Services -Secure file access -SRB -POOL -VO mgmt

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 18 Expanded AliEn Services

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 19 Set of Services

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 20 The ARDA Prototype “The main goal of an ARDA prototype is to provide a more complete blueprint and to develop the specifications and interfaces for the ARDA services and API” “We recommend that the LCG setup a project to develop a prototype…” 6 months for first prototype (driven by need for TDRs in 2005 ?) ARDA Project –Careful definition of work areas –Define project constituency –Define project lead -> project team –Work plan, schedule, milestones Particularly plan for interfacing to and engaging with LHC experiments and the LHC related Grid community

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 21 RTAG Recommendations RTAG report proposed a four-prong approach : Re-factoring of AliEn and other services into ARDA, with an initial release; consolidation of the API working with the experiments and the LCG-AA; release of a fully functional prototype. Subsequently implementation of agreed interfaces, testing and release of the prototype implementation. Modelling of an OGSI-based services infrastructure, performance tests and quality assurance of the prototype implementation Interfacing to LCG-AA software like POOL and ROOT Interfacing to experiment's frameworks, with specific meta-data handlers and experiment specific services n.b. OGSI  WSRF (or perhaps WS in the first instance)

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 22 Message of ARDA Report (© Miron Livny) Deliver end-to-end capabilities ( from user to fabric ) and stability ( deployable ) at the price of services offered ( functionality ) Services provide a natural abstraction and powerful software engineering constructs AliEn provides a useful and stable suite of services as it meets the expectations of the Alice experiment

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 23 The ARDA Workshop

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 24 ARDA Workshop - Goals CERN : 21 st /22 nd January 2004 : (Les Robertson) Define the ARDA project The scope of the distributed analysis requirements that should be addressed The scope of a generic middleware component, –the approach to implementation –target timescales Which HEP-specific components should or could be done in common in the LCG Applications Area –e.g. POOL, collections, meta-data, SEAL,.. A process for agreeing on the specification of ARDA services – middleware projects, experiments The framework for an ARDA implementation project, coordinating – Middleware ↔ LCG AA ↔ experiment analysis s/w ↔ end-users

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 25 Workshop Agenda Requirements (HEPCAL II)- Federico Carminati ARDA Architecture- Lothar Bauerdick Experiment expectations from ARDA – 4 talks Generic Middleware- Frederic Hemmer - Miron Livny Applications Area involvement in ARDA- Torre Wenaus POOL-ARDA collaboration- Dirk Duellmann Discussion…

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 26 Experiments Sumamry (© Andreas Peters)

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 27 Experiments Summary (© David Adams) ATLAS strategy –Use grid service model –Quickly define high-level service interfaces and implement services and clients –Deliver end-to-end system to users –Frequently re-design and re-implement based on user feedback ARDA collaboration –Ideally we would come to consensus within ARDA on a high-level interface along the lines of AJDL and share end-to-end effort –In any case, we will work closely with ARDA to define middleware services

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 28 Experiments Summary (© Julian Bunn (et al)) CMS Distributed Production –CMS is already using successfully LCG and VDT grid middleware to build its distributed production system –A distributed batch-analysis system is being developed on top of the same LCG and VDT software –CMS suggests that in the short term (  6 months) ARDA extends the functionality of the LCG middleware to meet the architecture described in the ARDA document Grid Analysis Environment –CMS asks that the GAE be adopted as the basis for the ARDA work on a system that supports interactive and batch analysis. Support for Interactive Analysis is the crucial goal in this context.

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 29 Experiments Summary (© Andrei Tsaregorodtsev ) LHCb Expected from ARDA –More efficient development process: Rapid development cycles; Keeping the functional core and adding functionality incrementally; Emphasis on intensive testing while the development. –Concurrent development of components to try out different ideas and to enhance the quality by competition. Participation –Participate to the definition of the services interfaces, testing and feedback; –Prototyping ARDA components using OGSI compliant implementations; –Developing the DIRAC WMS into an ARDA compliant service The first tests of the distributed analysis will be done during the DC2004 (May) using LHCb developed and LCG tools; We see the further evolution of the LHCb distributed analysis tools within the context of the ARDA architecture and the proposed development process.

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 30 Experiments Summary Summary ! A cynical view…!! We think ARDA’s the way forward We will be ARDA compliant …but would like ARDA to be based on us as a starting point !!

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 31 Middleware (from report to prototype) (© Miron Livny) Understand AliEn (including services) Identify potential contributions from existing middleware Understand requirements (how does analysis differ from “production”?) Develop a plan – what, how, when, who –Semantic of exposed services –Authentication/protection model –Integration/testing/deployment procedures –Documentation –… Execute the plan!

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 32 Middleware (working document) (© Miron Livny) Abstract: This working document is used to break down the high level services defined by ARDA to actual components and tries to map these components to existing implementations coming from AliEn, EDG, and VDT. The structure and initial AliEn input is taken from Chapter 5 of Draft v0.2 of the ARDA document (unpublished) Started after the December meeting as a vehicle to exchange and record information and ideas among the middleware providers. –Identification of services Service interplay and semantics –Understand how existing MW could implement these services Input from AliEn, EDG, VDT, commercial, …. (others?) –Specify interfaces to applications

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 33 General ARDA-AA Objectives (© Torre Wenaus) Common software above the middleware layer –Adapting, extending, interfacing AA software for ARDA –Participating in ARDA interface definition; ensuring AA requirements met –Applying lower level middleware services in specialized higher level services directed at HEP and analysis Early PEB agreement on ARDA: Middleware covers as much as possible; remaining higher levels covered by AA (if common) or experiments (if not) Integration and validation –Integrating ARDA middleware services and analysis application level services into end-to-end distributed analysis prototype –Assisting integration of distributed analysis prototype or components thereof into experiment environments –Validation of the prototype [and feedback to middleware providers] Proposal to use the GAG as the principal feedback channel seems a very good one

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 34 Summarizing TW’s Current Thoughts on WPs (© Torre Wenaus) 1)Integration and Validation – Primarily providing coordination, communication, coherence for efforts residing in the experiments and projects Some similarity to Physics Validation in the simulation project Though the (majority of the) work will go on in the experiments and projects, a common focal point is needed if it is to be a common effort 2)Event data management – Physics-driven event collections – Joint WP with POOL Collections 3)Framework integration –‘Thin’ adaptation of middleware services to whatever is required for integration in experiment analysis frameworks –Joint WP with SEAL

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 35 Status

AlAtCL POOL SEAL GAG EGEE/ VDT ARDA coordination integration, validation development specific services ARDA Services Grid middleware POOL’, SEAL’, HEP-specific SEAL’ Services POOL’ Services Grid Middleware Services Experiment Prototype Applications Al At C L Experiment Requirements and Use Cases Consolidated Requirements Operational experience physicists with real requirements ADAGAE DIRACALIEN grid technology and experience Experiment LCG Grid MW ARDA AliEn EDG VDT.... Other Common project HEP-common Service Nordu (© Les Robertson)

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 37 GAG LCG PEB AliceATLASCMSLHCb Generic Middleware Resource providers Requirements Guidelines Service specs Integration: workers’ forum LCG AA projects ARDA – “A Realisation of Distributed Analysis”

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 38 Postscript HEPCAL II provides reasonable set of use cases as input –But must be complemented by use cases from non-HEP (within EGEE) Proposed set of services for DA (under HEPCAL II) is reasonable Prototypes should leverage existing technologies and experience within a web services framework Prototype(s) needed (urgently) within 6 months Applications middleware must interface (POOL, LCG-AA, ROOT, GAE, …) as should experiments software The PEB are still working on the details

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 39 If you think ARDA is a threat … –It’s here to stay, so get used to it !! If you think ARDA is an opportunity –Then grasp it (but don’t hang around as timescales are ridiculously short (what’s new ?)) !! It’s for you to choose !! “ We should not miss this opportunity to put it all together! ” (© fca 2004)

LCG 4-Feb-2004GridPP-9 : EdinburghSlide 40 The End