Thoughts on Applications Area Involvement in ARDA Torre Wenaus, BNL/CERN LCG Applications Area Manager ARDA Workshop Jan 22, 2004

The main inputs so far… ARDA AA Meeting Nov 27 Small first meeting with the AA LCG people we expect to be involved with a substantial amount of their time Derek Feichtinger, Juha Herrala, Kuba Moscicki, Frederick Orellana Plus Frederic Hemmer, Predrag Buncic, Dirk Duellmann, Alberto Aimar and myself Had an overview of ARDA from Predrag, then discussed possible areas of AA activity Frederic’s initial ARDA middleware meeting in early Dec TW attended (some of it) as AA rep PEB and SC2 presentations of (almost) this talk in Dec Workshop yesterday Didn’t lead me to change anything substantive in this talk from the version presented Dec 9 (not necessarily a good thing)

General ARDA AA Objectives Common software above the middleware layer Adapting, extending, interfacing AA software for ARDA Participating in ARDA interface definition; ensuring AA requirements met Applying lower level middleware services in specialized higher level services directed at HEP and analysis Early PEB agreement on ARDA: Middleware covers as much as possible; remaining higher levels covered by AA (if common) or experiments (if not) Integration and validation Integrating ARDA middleware services and analysis application level services into end-to-end distributed analysis prototype Assisting integration of distributed analysis prototype or components thereof into experiment environments Validation of the prototype [and feedback to middleware providers] Proposal to use the GAG as the principal feedback channel seems a very good one to me

We expect ARDA will use SPI services and policies Possible work areas Event data management and access Framework integration services Interactive analysis tools Analysis environment integration and validation … and with thoughts on work package organization We expect ARDA will use SPI services and policies

Event data management and access Event collections, physics-level datasets, physics queries Efficient sparse data access Data access below file level (event objects) Splitting at physics dataset level A mix of interface development, POOL work, ROOT work Collections work currently going on in a POOL WP, but this work needs an ‘analysis’ perspective and not just a ‘persistency’ perspective – ARDA can provide that Make this ARDA work package a joint work package with POOL Collection WP Will be addressed by Dirk in the next talk

Framework integration services Interfacing/integrating framework-level distributed services Distributed messaging, error handling, logging, … Interactive interface; Python, ROOT bindings Framework access to more sophisticated middleware services? Workflow management, replication, … Probably mostly a very ‘thin’ activity not developing services, or even the interfaces But contributing to interface definition just packaging/integrating them for the AA architecture Maybe participation in some specialization of generic services closely connected to the framework? (e.g. managing provenance data) The long-empty ‘grid based services’ box in SEAL Joint ARDA/SEAL WP

Interactive Analysis Tools Interfacing to tools supporting interactive (low-latency, rapid-response) analysis ROOT, PROOF integration User interfaces to tools supporting analysis workload management User level management/monitoring User level reservations (‘what’ and ‘when’) Interfacing to tools supporting dynamic job interaction/control AIDA integration Needs will vary from experiment to experiment; probably mostly experiment-specific integration If there is an AA element, fold into the next WP…

Analysis Environment Integration & Validation ARDA integration as an analysis system in experiment environments Integrating experiment specific front end and service components with ARDA components into an end-to-end system Early priority: users in experiments testing detailed use cases using experiment-integrated ARDA Get ARDA in the hands of physicists doing analysis as soon as possible (as soon as there is a tool of interest to attract them – experiment ARDA teams need to sell the product) The key work package Support and assist four distinct but collaborative ARDA integration efforts in the experiments Coordination for the higher level elements of ARDA not coming from the middleware Coordinate gathering of feedback from experiment ARDA teams/users Use GAG for middleware feedback channel Provide overall coordination/coherence for ARDA end-to-end analysis systems

Summarizing My Current Thoughts on WPs Integration and Validation Primarily providing coordination, communication, coherence for efforts residing in the experiments and projects Some similarity to Physics Validation in the simulation project Though the (majority of the) work will go on in the experiments and projects, a common focal point is needed if it is to be a common effort Event data management Physics-driven event collections Joint WP with POOL Collections Framework integration ‘Thin’ adaptation of middleware services to whatever is required for integration in experiment analysis frameworks Joint WP with SEAL

Slide Title : Overview of Main Activities and Deliverables Experiment Prototype Applications ALIEN ADA GAE DIRAC physicists with real requirements Al At C L Other Common project ARDA Services Grid middleware POOL’, SEAL’, HEP-specific Operational experience Al At C L HEP-common Service SEAL ARDA coordination integration, validation development specific services GAG SEAL’ Services Consolidated Requirements POOL’ Services POOL Slide Title : Overview of Main Activities and Deliverables Slide shows how ARDA project could relate to those existing activities that have a relevant role to play (circles). It also shows the main deliverables (boxes) Shows scope of ARDA in green (indicates added value of project and in what context ARDA effort is working) ARDA should provide a consolidated set of middleware services to the experiments that then use them to produce rapid prototypes of realistic applications, gaining operational experience which is fed back to the project. Consolidated sets of requirements and use cases are provided by a cross experiment forum of appropriate experts (GAG – already exists and could assume this role with appropriate membership) The basic (non-HEP-specific) services are developed by a joint EGEE/VDT effort: delivery of an early prototype and further releases under short evolutionary cycles uses experience and technology from existing grid development projects (e.g. AliEn, EDG, NorduGrid, VDT ..) re-engineering according to agreed architectural models (OGSI) and implementation technology choices HEP-specific services may also be identified and developed in an appropriate context, e.g. POOL for data management related services SEAL for common services involving integration with other standard software components (UI, Job control, etc.) Main role of ARDA itself is one of coordination, integration, validation, development of other specific services Callout emphasises importance of involvement of physicists doing real distributed analysis, having real requirements ARDA itself may comprise people from the project and from the experiments. Project members will all have an experiment label (AACL) but will work together as a team. Experiment Requirements and Use Cases Grid Middleware Services Experiment AliEn grid technology and experience EGEE/ VDT LCG EDG Grid MW Nordu . . . . ARDA VDT