29 May 2002Joint EDG/WP8-EDT/WP4 MeetingClaudio Grandi INFN Bologna LHC Experiments Grid Integration Plans C.Grandi INFN - Bologna

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 2 Claudio Grandi INFN Bologna Introduction What do LHC experiments have to do: Build a distributed computing system for the experiment Test the prototypes: tools evaluation data challenges: tests with big data flows What do LHC experiments have today: Software EDG release 1.1 (1.2 coming) VDT release 1.0 (includes Globus, Condor-G, GDMP, …) Other tools for distributed computing (e.g. Web services, …) Test facilities EDG test bed EDT supported experiment test beds US GriPhyN/PPDG/iVDGl experiment test beds

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 3 Claudio Grandi INFN Bologna ALICE: Overview Application: AliRoot Data Persistency: Root Job parallelisation: Root/PROOF Job submission/control/monitoring: AliEn Data Catalogue: AliEn AliRoot AliEn API/PROOF AliEn Services ROOT IVDGLEDT By P.Cerello

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 4 Claudio Grandi INFN Bologna ALICE: Sites Houston Univ. - Texas OSU/OSC - Ohio LBL - California CERN - CH INFN Cagliari/Catania/ CNAF/Salerno/Torino - I NIKHEF - NL (?) By P.Cerello

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 5 Claudio Grandi INFN Bologna ALICE: Integration Items Job submission: EDG/Resource Broker as an AliEn client Condor-G as an AliEn client Data Management: register/access the AliEn Data Catalogue from an EDG/iVDGL/... Job MetaData Catalogue: implement a Job MetaData Catalogue Test multiple accesses and concurrent updates Evaluate Spitfire to manage it Spawn PROOF sub-jobs on any GRID By P.Cerello

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 6 Claudio Grandi INFN Bologna ALICE: Integration Plans AliEn Job EDG RB EDG CE EDG WN AliEn Data Catalogue EDG SE Condor-G Alice Farm Alice User I-LFN I-PFN O-PFN O-LFN,O-PFN MetaData Catalogue By P.Cerello

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 7 Claudio Grandi INFN Bologna ALICE: Integration Plans EDG WN AliEn Data Catalogue EDG SE1 I-LFN(s) O-PFN3 I-PFN(s) I-PFN3 MetaData Catalogue I-LFN(s) EDG SE3 EDG SE2 EDG CE2 EDG CE3 I-PFN2 I-PFN1 O-PFN2 O-PFN1 PROOF By P.Cerello

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 8 Claudio Grandi INFN Bologna ATLAS: DC1 & 2 DC1/1: simulation of 10**7 evts for HLT TDR started 15 April. Bulk of production will start after 15 June DC1/2: stress is on new software (20-9 till Xmas) Use of currently avalable Grid tools favoured but not mandatory: start with EDG 1.2 in EU and VDT 1.0 in US: Some US sites deploy EDG software and vice-versa. Integration of Magda (US data catalog+some replica management) with EDG RC Critical issue the EDG stability and reliability DC2: start spring 03 Use EDG release 2 Grid API to be inserted in ATHENA framework Magda and WP2 convergency plan to be detailed and executed Integrate VDT components

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 9 Claudio Grandi INFN Bologna ATLAS: Overview Main tools for data management on the grid: Magda - MAnager for Grid Data GANGA - Gaudi and Grid Alliance (with LHCb) Other tools in development: –GridView - simple script tool to monitor status of testbed (Java version being developed) –Gripe - unified user accounts –Pacman - package management and distribution tool –Grappa - web portal based on active notebook tech. –GRAT (GRid Application Toolkit): distribution of ATLAS software on the Grid Integration with EDG is being discussed

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 10 Claudio Grandi INFN Bologna ATLAS: Magda Magda: MAGD Magda: MAnager for Grid-based Data PPDG Deliverable for the Particle Physics Data Grid MySQL database at the core of the system web interface User interaction via command line and web interface File replication via GSIFtp Currently in use for ATLAS distributed data management prototyping and design 260k files, ~11TB cataloged at present Synchronization between Magda & EDG catalogues ATLAS applications query Magda’s MySQL catalogue; EDG Resource Broker queries the Globus Replica Catalog. By D.Rebatto

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 11 Claudio Grandi INFN Bologna ATLAS: Magda Magda Export Catalogue Import Catalogue Local Catalogue Magda Export Catalogue Import Catalogue Local Catalogue Host AHost B GDMP Globus replica catalog Magda MySQL catalog USER B: gdmp_host_subscribe -r host_b -p my_file A file named my_file is created at Host A Local Catalogue my_file USER A: gdmp_register_local_file -p my_file GDMP Local Catalogue my_file USER B: gdmp_replicate_get GDMP GSIFtp my_file Magda Local Catalogue my_file GDMP B: magda_putfile my_file --registeronly GDMP GSIFtp my_file GDMP_STAGE_TO_MSS GDMP Export Catalogue Import Catalogue my_file USER A: gdmp_publish_catalogue GDMPMagda GDMP Export Catalogue Import Catalogue my_file GDMP A: magda_putfile my_file --registeronly GDMP By D.Rebatto

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 12 Claudio Grandi INFN Bologna CMS: Overview Build a unique CMS-GRID framework (EU+US) EU and US grids not interoperable today Help from DataTAG-iVDGL-GLUE Work in parallel in EU and US Main US activities: MOP Virtual Data System Interactive Analysis Main EU activities: Integration of IMPALA with EDG WP1+WP2 sw. Batch Analysis: user job submission & analysis farm CMS Production Framework

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 13 Claudio Grandi INFN Bologna CMS: Schedule : A few sites in production. Some grid tools used “in production” (GDMP). 2002: World-Wide productions. Prototypes of “distributed” production sites deployed in US and EU. DAQ TDR. 2003: Integration activities: US/EU grid interoperability. First grid-enabled sites used “in production”. 2004: 5% data challenge. Deployment of delivered LCG prototypes. CCS TDR. 2005: preparation for 20% data challenge. Build "final" computing environment. Physics TDR. Long term schedule being revised now!

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 14 Claudio Grandi INFN Bologna Data Management components -Replica Manager -Virtual Data Catalogue -Replica Catalogue -… Planner (uses CMS production SW) Storage Element Reference DB Local analysis tools with plugin to access remote data Physics Data Production & Analysis Portal Computing Element EDG SW = data flow or invocation = read data from = contains reference to = active component = catalog or database job CMS: EU-US integration 02

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 15 Claudio Grandi INFN Bologna Computing Element Local Scheduler GRAM Replica Catalogue Maps each logical file name to one or more physical file names Job Submission Service Resource Broker Finds suitable Location for execution Condor-G Worker nodes CMSIM ORCA Storage Element Local Storage Local Objy FDDB Reference DB has all information needed by IMPALA to generate a dataset User Interface job = data flow or invocation = read data from = contains reference to = active component = catalog or database Information Services LDAP server Resource information job Production portal IMPALA Get request for a production Create location independent jobs BOSS CMS: EDG integration 02

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 16 Claudio Grandi INFN Bologna LHCb: Overview GANGA (Gaudi and Grid Alliance) : In collaboration with Atlas developing user interface for all levels of user (physicist,production manager,developer) Control and Monitoring system for distributed data production: Using PVSS today Data challenges: DC1: 3-30 june + 26 august -22 september 2002 DC2: 2003 DC3: 2004 DC4: 2005

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 17 Claudio Grandi INFN Bologna LHCb: DC1 Computing tests –prototype of new data management databases –configuration of LHCb environment at remote sites and installation kits –monitoring and control with PVSS –integrability of EDG testbed into our production system –data quality checking, evaluate DaVinci based tools –use "push" MC job submission initially, perhaps test first version of pull mechanism during second half –executables to be tested in production: sicbmc, Brunel, DaVinci Data production Produce useful data, for example for Physics studies with this facility, if the software is ready… Resources: 100 – 150 CPU at CERN & ~ CPUs outside (Bologna, IN2P3, RAL, Oxford, Bristol, Edinburgh, Nikhef, Cambridge, Barcelona, Moscow, Amsterdam VU?) By J.Closier

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 18 Claudio Grandi INFN Bologna LHCb: DC2 (2003) Computing tests –production version of new data management databases –implementation of "pull" MC job submission system, exploiting EDG resource broker –stress test Grid "philosophy" of submitting jobs without worrying where the job will run and where the output will be stored –monitoring and control, evaluate EDG middleware (how does it work with PVSS? can it replace it?) –make job submission tools independent of remote sites –test of first Ganga prototype –test of some Gaudi-Grid interfaces e.g. event selector –test of automatic data quality histogram creation and checking in production –executables to be tested: (sicbmc), Gauss, Brunel, DaVinci, Gaudi Resources Same sites as DC1 + Switzerland, Germany, Poland By J.Closier

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 19 Claudio Grandi INFN Bologna LHCb: DC3&4 (2004/5) Computing tests for DC3 –test of first prototype of the analysis model (display events that were analysed on the grid) –test of Ganga production version –full use of Grid in production –further tests of Gauss in production –tests of data quality checking in production –executables to be tested: Gauss, Brunel, DaVinci, Gaudi Computing tests for DC4 test of production analysis model the other points also mentioned previously By J.Closier

29 May 2002 Joint EDG/WP8-EDT/WP4 Meeting 20 Claudio Grandi INFN Bologna Summary Experiments are integrating Grid tools in their computing environment Most of the experiments are using Grid tools delivered by more than one Grid Project Most of the experiments are not planning to use the tools out of the box but to adapt them to their needs  Need help by middleware experts for interoperability Need non-standard facilities to develop and test the experiment computing environment. Help by middleware experts is needed for that too