David Adams ATLAS ADA, ARDA and PPDG David Adams BNL June 28, 2004 PPDG Collaboration Meeting Williams Bay, Wisconsin

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Contents Definitions ADA/DIAL model Current status of ADA Connections to PPDG Conclusions

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Definitions ADA – ATLAS Distributed Analysis ATLAS project to deliver an end-to-end analysis system ARDA – A Realisation of Distributed Analysis for LHC CERN project to coordinate the development of prototype distributed analysis systems for the LHC experiments Prototypes are based on the EGEE middleware (gLite) DIAL – Distributed Interactive Analysis of Large Datasets BNL project to study interactive analysis of large amounts of data Delivered framework is also suitable for non-interactive jobs AJDL – Abstract Job Definition Language Generic and extensible definition of jobs intended for use with any batch or workload management system Components: transformation (application and task), dataset and job

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, ADA/DIAL model User interaction scenario: Select a dataset –AJDL, not HEPCAL Examine dataset –Display content (event ID’s, type-keys) –Location of data (logical files, physical files, DB, …) >Including means to localize data Define transformation –Application that processes the data –Task carrying data to configure application –Select these from catalog and modify Create job to apply transformation to dataset –Submit app, task and dataset to scheduler (e.g. analysis service) –Result will be a new dataset

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, ADA/DIAL model (cont) User interactions (cont) Monitor job –Status (running, done, failed, …) –Status of sub-jobs –Examine partial result Examine final result (output dataset) Repeat Following block diagram shows major components Client High-level services Middleware services

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, ADA/DIAL model (cont) Release 1.0 Started

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Current status of ADA Framework based on DIAL 0.90 Clients ROOT from DIAL (using rootcint) Python from GANGA (using lcgdict with DIAL classes) GUI being developed by GANGA

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Current status of ADA (cont) Analysis services DIAL provides the starting point for all the following Interactive service (i.e. low latency for short jobs) –Currently uses a fast LSF queue –Condor COD being tested –Running at BNL –Will be deployed at other sites –May also be deployed for long-running jobs Service to insert jobs in ATLAS production queue is being developed –Jobs may be run on any of the ATLAS grids ARDA service based on gLite being developed –Jobs sent to gLite WMS

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Current status of ADA (cont) Applications DC1 CBNT  histograms in place DC2 reconstruction for release in place Working on DC2 event data  histograms Capability to reuse production xforms being developed Datasets Generic classes in DIAL DC1: CbntDataset and HbookDataset DC2: AtlasPoolEventDataset and RootHistogramDataset Catalogs AMI adding required tables and user interface –Repositories, selection catalogs, … Programmatic access via AMI web service

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG DIAL Provides many parts of ADA –The overall framework –The ROOT client –Infrastructure for all analysis services delivered and under development –Connection to LSF, Condor (including COD) Depends on PPDG for support and exchange of ideas

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG (cont) AJDL Facilitates the development of clients or services without DIAL –E.g. plan for GANGA python interface is to drop dependence on DIAL C++ libraries >Lightweight client for any platform Goal to standardize within PPDG –Send non-ATLAS jobs to ATLAS analysis service –And vice versa

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG (cont) U.S. grid infrastructure U.S. sites important to ATLAS –And especially to USATLAS Assume continuing support from PPDG –Grid3, grid3+, OSG, … Most sites continue taking jobs from ATPROD queues A few sites to also host interactive analysis service

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG (cont) Monitoring and modeling US monitoring data fed into ATLAS computing model –Job data from ADA catalogs –Site data from grid monitoring ACM is interested in constructing models to understand analysis patterns –Collaboration with CS-11?

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG (cont) EGEE middleware Install gLite on one or more U.S. sites –At least services required to support WMS >CE, SE, … Submit jobs from the ATLAS-ARDA prototype Common project –Also submit jobs from CMS and Alice prototypes Will U.S. sites deploy EGEE middleware? –Important issue for evolution of ADA

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Connections to PPDG (cont) File cataloging and movement SRM likely to become standard –Licensing issues? Many replica catalogs relevant to ATLAS –Magda (phased out for DC2) –EDG RLS (LCG catalog) –ISI RLS (separate catalogs for U.S. and NorduGrid) –EGEE/AliEn file catalog (required for ARDA prototype?) Also many options for high-level interfaces –ATLAS internal system Don Quijote –CMS file transfer system –EGEE file transfer system (same as CMS?) –LBL RMS/RRS PPDG can help sort this out

David Adams ATLAS ATLAS and ARDA ARDA WorkshopJune 21, Conclusions ATLAS has benefited in may ways from PPDG Magda, DIAL, AJDL, SRM, Grid3, … Ideas in HEPCAL and ARDA Continue to look to PPDG Forum to exchange ideas Provider or identifier of useful tools and services Support for U.S. grid infrastructure