ATLAS Data Challenges NorduGrid Workshop Uppsala November 11-13; 2002 Gilbert Poulard ATLAS DC coordinator CERN EP-ATC

November 11, 2002G. Poulard - NorduGrid Workshop2 Outline nIntroduction nDC0 nDC1 nGrid activities in ATLAS nDCn’s nSummary DC web page:

November 11, 2002G. Poulard - NorduGrid Workshop3 Data challenges nWhy? u In the context of the CERN Computing Review it has been recognized that the Computing for LHC was very complex and requested a huge amount of resources. u Several recommendations were made, among them:  Create the LHC Computing Grid (LCG) project  Ask the experiments to launch a set of Data Challenges to understand and validate Their computing model; data model; software suite Their technology choices The scalability of the chosen solutions

November 11, 2002G. Poulard - NorduGrid Workshop4 ATLAS Data challenges nIn ATLAS it was decided u To foresee a “serie” of DCs of increasing complexity  Start with data which looks like real data  Run the filtering and reconstruction chain  Store the output data into the ‘ad-hoc’ persistent repository  Run the analysis  Produce physics results u To study  Performance issues, persistency technologies, analysis scenarios,... u To identify  weaknesses, bottle necks, etc… (but also good points) u In using both the hardware (prototype) the software and the middleware developed and/or deployed by the LCG project

November 11, 2002G. Poulard - NorduGrid Workshop5 ATLAS Data challenges nBut it was also acknowledged that: u Today we don’t have ‘real data’  Need to produce ‘simulated data’ first  So: Physics Event generation Detector Simulation Pile-up Reconstruction and analysis  will be part of the first Data Challenges u we need also to “satisfy” the requirements of the ATLAS communities  HLT, Physics groups,...

November 11, 2002G. Poulard - NorduGrid Workshop6 ATLAS Data challenges nIn addition it is understood that the results of the DCs should be used to u Prepare a computing MoU in due time u Perform a new Physics TDR ~one year before the real data taking nThe retained schedule was to: u start with DC0 in late 2001  Considered at that time as a preparation one u continue with one DC per year

November 11, 2002G. Poulard - NorduGrid Workshop7 DC0 nWas defined as u A readiness and continuity test  Have the full chain running from the same release u A preparation for DC1; in particular  One of the main emphasis was to put in place the full infrastructure with Objectivity (which was the base-line technology for persistency at that time) u It should also be noted that there was a strong request from the physicists to be able to reconstruct and analyze the “old” physics TDR data within the new Athena framework

November 11, 2002G. Poulard - NorduGrid Workshop8 DC0: Readiness & continuity tests ( December 2001 – June 2002) n“3 lines” for “full” simulation u 1) Full chain with new geometry (as of January 2002) Generator->(Objy)->Geant3->(Zebra->Objy)->Athena rec.->(Objy)->Analysis u 2) Reconstruction of ‘Physics TDR’ data within Athena (Zebra->Objy)->Athena rec.-> (Objy) -> Simple analysis u 3) Geant4  Robustness test Generator-> (Objy)->Geant4->(Objy) n“1 line” for “fast” simulation Generator-> (Objy) -> Atlfast -> (Objy) Continuity test: Everything from the same release for the full chain (3.0.2)

November 11, 2002G. Poulard - NorduGrid Workshop9 Atlsim/G3 Hits/Digits MCTruth Atlsim/G3 Pythia 6 H  4 mu Objectivity/DB HepMC Event generation Schematic View of Task Flow for DC0 Detector Simulation Hits/Digits MCTruth Athena Hits/Digits MCTruth Athena Hits/Digits MCTruth Hits/Digits MCTruth Hits/Digits MCTruth Zebra Data Conversion Athena AOD Athena AOD Athena AOD Reconstruction

November 11, 2002G. Poulard - NorduGrid Workshop10 DC0: Readiness & continuity tests ( December 2001 – June 2002) nTook longer than foreseen u Due to several reasons  Introduction of new tools  Change of the base-line for persistency Which has as a major consequence to divert some of the man power  Under-evaluation of the statement “have everything from the same release” u Nevertheless we learnt a lot nWas completed in June 2002

November 11, 2002G. Poulard - NorduGrid Workshop11 ATLAS Data Challenges: DC1 nOriginal goals (November 2001): u reconstruction & analysis on a large scale  learn about data model; I/O performances; identify bottlenecks … u data management  Use/evaluate persistency technology (AthenaRoot I/O)  Learn about distributed analysis u Need to produce data for HLT & Physics groups  HLT TDR has been delayed to mid 2003 Study performance of Athena and algorithms for use in HLT High statistics needed –Scale: few samples of up to 10 7 events in days, O(1000) PC’s –Simulation & pile-up will play an important role u Introduce new Event Data Model u Checking of Geant4 versus Geant3 u Involvement of CERN & outside-CERN sites: Worldwide excersise u use of GRID middleware as and when possible and appropriate nTo cope with different sets of requirements and for technical reasons (including software development, access to resources) decided to split DC1 into two phases

November 11, 2002G. Poulard - NorduGrid Workshop12 ATLAS DC1 nPhase I (April – August 2002) u Primary concern is delivery of events to HLT community u Put in place the MC event generation & detector simulation chain u Put in place the distributed MonteCarlo production nPhase II (October 2002 – January 2003) u Provide data with (and without) ‘pile-up’ for HLT studies u Introduction & testing of new Event Data Model (EDM) u Evaluation of new persistency technology u Use of Geant4 u Production of data for Physics and Computing Model studies u Testing of computing model & of distributed analysis using AOD u Use more widely GRID middleware

November 11, 2002G. Poulard - NorduGrid Workshop13 DC1 preparation nFirst major issue was to get the software ready u New geometry (compared to December-DC0 geometry) u New persistency mechanism n… Validated n… Distributed u “ATLAS kit” (rpm) to distribute the software nAnd to put in place the production scripts and tools (monitoring, bookkeeping) u Standard scripts to run the production u AMI bookkeeping database ( Grenoble) u Magda replica-catalog (BNL)

November 11, 2002G. Poulard - NorduGrid Workshop14 DC1 preparation: software (1) nNew geometry (compared to December-DC0 geometry) u Inner Detector  Beam pipe  Pixels: Services; material updated; More information in hits; better digitization  SCT tilt angle reversed (to minimize clusters)  TRT barrel: modular design  Realistic field u Calorimeter  ACBB: material and readout updates  ENDE: dead material and readout updated (last minute update to be avoided if possible)  HEND: dead material updated  FWDC: detailed design  End-cap Calorimeters shifted by 4 cm.  Cryostats split into Barrel and End-cap u Muon  AMDB p.03 (more detailed chambers cutouts)  Muon shielding update

November 11, 2002G. Poulard - NorduGrid Workshop15 ATLAS Geometry -Inner Detector -Calori meters -Muon System

November 11, 2002G. Poulard - NorduGrid Workshop16 ATLAS/G3 Few Numbers at a Glance n25,5 millions distinct volume copies n23 thousands different volume objects n4,673 different volume types nFew hundred pile-up events possible nAbout 1 million hits per event on average

November 11, 2002G. Poulard - NorduGrid Workshop17 DC1 preparation: software (2) nNew persistency mechanism u AthenaROOT/IO  Used for generated events  Readable by Atlfast and Atlsim nSimulation still using zebra

November 11, 2002G. Poulard - NorduGrid Workshop18 DC1/Phase I preparation: kit; scripts & tools nKit u “ATLAS kit” (rpm) to distribute the software  It installs release (all binaries) without any need of AFS  It requires : Linux OS (Redhat 6.2 or Redhat 7.2) CERNLIB 2001 (from DataGrid repository) cern i386.rpm (~289 MB)  It can be downloaded : from a multi-release page (22 rpm's; global size ~ 250 MB ) “tar” file also available nScripts and tools (monitoring, bookkeeping) u Standard scripts to run the production u AMI bookkeeping database

November 11, 2002G. Poulard - NorduGrid Workshop19 Atlsim/Geant3 + Filter 10 5 events Atlsim/Geant3 + Filter Hits/ Digits MCTruth Atlsim/Geant3 + Filter As an example, for 1 sample of di-jet events:  Event generation: 1.5 x 10 7 events in 150 partitions  Detector simulation: 3000 jobs Pythia 6 Di-jet Athena-Root I/O Zebra HepMC Event generation DC1/Phase I Task Flow Detector Simulation (5000 evts) (~450 evts) Hits/ Digits MCTruth Hits/ Digits MCtruth

November 11, 2002G. Poulard - NorduGrid Workshop20 DC1 preparation: validation & quality control nWe defined two types of validation u Validation of the sites:  We processed the same data in the various centres and made the comparison To insure that the same software was running in all production centres We also checked the random number sequences u Validation of the simulation:  We used both “old” generated data & “new” data Validation datasets: di-jets, single ,e, ,H  4e/2  /2e2  /4  About 10 7 evts reconstructed in June, July and August  We made the comparison between “old” and “new” simulated data

November 11, 2002G. Poulard - NorduGrid Workshop21 DC1 preparation: validation & quality control nThis was a very “intensive” activity u Many findings: simulation or software installation sites problems (all eventually solved) u We should increase the number of people involved nIt is a “key issue” for the success!

November 11, 2002G. Poulard - NorduGrid Workshop22 Example:  jets distribution (di-jets sample) Old sim sample New sim sample 22 Comparison Reappearance of an old dice version in a site installed software

November 11, 2002G. Poulard - NorduGrid Workshop23 Validation samples (740k events)  single particles (e, , ,  ), jet scans, Higgs events Single-particle production (30 million events)  single  (low p T ; p T =1000 GeV with 2.8<  <3.2)  single  (p T =3, …, 100 GeV)  single e and  different energies (E=5, 10, …, 200, 1000 GeV) fixed  points;  scans (|  |<2.5);  crack scans (1.3<  <1.8) standard beam spread (  z =5.6 cm); fixed vertex z-components (z=0, 4, 10 cm) Minimum-bias production (1.5 million events)  different  regions (|  |<3, 5, 5.5, 7) Data Samples I

November 11, 2002G. Poulard - NorduGrid Workshop24 QCD di-jet production (5.2 million events)  different cuts on E T (hard scattering) during generation large production of E T >11, 17, 25, 55 GeV samples, applying particle-level filters large production of E T >17, 35 GeV samples, without filtering, full simulation within |  |<5 smaller production of E T >70, 140, 280, 560 GeV samples Physics events requested by various HLT groups (e/ , Level-1, jet/E T miss, B-physics, b-jet,  ; 4.4 million events)  large samples for the b-jet trigger simulated with default (3 pixel layers) and staged (2 pixel layers) layouts  B-physics (PL) events taken from old TDR tapes Data Samples II

November 11, 2002G. Poulard - NorduGrid Workshop25 ATLAS DC1/Phase I: July-August 2002 Goals : Produce the data needed for the HLT TDR Get as many ATLAS institutes involved as possible Worldwide collaborative activity Participation : 39 Institutes in 18 countries nAustralia nAustria nCanada nCERN nCzech Republic nDenmark nFrance nGermany nIsrael nItaly nJapan nNorway nRussia nSpain nSweden nTaiwan nUK nUSA

November 11, 2002G. Poulard - NorduGrid Workshop26 ATLAS DC1 Phase I : July-August 2002 nCPU Resources used : u Up to 3200 processors (5000 PIII/500 equivalent) u 110 kSI95 (~ 50% of one Regional Centre at LHC startup) u CPU*days u To simulate one di-jet event : SI95sec nData Volume : u 30 Tbytes u files u Output size for one di-jet event (2.4 Mbytes) u Data kept at production site for further processing  Pile-up  Reconstruction  Analysis

November 11, 2002G. Poulard - NorduGrid Workshop CPU‘s 110 kSI CPU days 5*10* 7 events generated 1*10* 7 events simulated 3*10* 7 single particles 30 Tbytes files ATLAS DC1 Phase I : July-August institutions in 18 countries

November 11, 2002G. Poulard - NorduGrid Workshop28 ATLAS DC1 Phase 1 : July-August 2002

November 11, 2002G. Poulard - NorduGrid Workshop29 ATLAS DC1 Phase II nProvide data with and without ‘pile-up’ for HLT studies u new data samples (huge amount of requests) u Pile-up in Atlsim u “Byte stream” format to be produced nIntroduction & testing of new Event Data Model (EDM) u This should include new Detector Description nEvaluation of new persistency technology nUse of Geant4 nProduction of data for Physics and Computing Model studies u Both ESD and AOD will be produced from Athena Reconstruction u We would like to get the ‘large scale reconstruction’ and the ‘data-flow’ studies ready but not be part of Phase II nTesting of computing model & of distributed analysis using AOD nUse more widely GRID middleware (have a test in November)

November 11, 2002G. Poulard - NorduGrid Workshop30 Pile-up nFirst issue is to produce the pile-up data for HLT u We intend to do this now u Code is ready u Validation is in progress u No “obvious” problems

November 11, 2002G. Poulard - NorduGrid Workshop31 Luminosity Effect Simulation Aim Study Interesting Processing at different Luminosity L Separate Simulation of Physics Events & Minimum Bias Events Merging of Primary Stream (Physics) Background Stream (Pileup) Primary Stream (KINE,HITS) Background Stream (KINE,HITS DIGITIZATION Bunch Crossing (DIGI) 1 N( L )

November 11, 2002G. Poulard - NorduGrid Workshop32 Pile-up features nDifferent detectors have different memory time requiring very different number of minimum bias events to be read in u Silicons, Tile calorimeter: t<25 ns u Straw tracker: t<~40-50 ns u Lar Calorimeters: ns u Muon Drift Tubes: 600 ns nStill we want the pile-up events to be the same in different detectors !

November 11, 2002G. Poulard - NorduGrid Workshop33 Higgs into two photons no pile-up

November 11, 2002G. Poulard - NorduGrid Workshop34 Higgs into two photons L=10^34 pile-up

November 11, 2002G. Poulard - NorduGrid Workshop35 Pile-up production nScheduled for October-November 2002 nBoth low (2 x ) and high luminosity (10 34 ) data will be prepared nResources estimate: u CPU days (NCU) u 70 Tbyte of data u files

November 11, 2002G. Poulard - NorduGrid Workshop36 ATLAS DC1 Phase II (2) nNext steps will be to u run the reconstruction within Athena framework  Most functionality should be there with release Probably not ready for ‘massive’ production  Reconstruction ready by the end of November u produce the “byte-stream” data u perform the analysis of the AOD nIn parallel the dedicated code for HLT studies is being prepared (PESA release 3.0.0) nGeant4 tests with a quite complete geometry should be available by mid-December nLarge scale Grid test is scheduled for December n“Expected “end” date 31st January 2003 n“Massive” reconstruction is not part of DC1 Phase II

November 11, 2002G. Poulard - NorduGrid Workshop37 ATLAS DC1 Phase II (3) nCompared to Phase I u More automated production u “Pro-active” use of the AMI bookkeeping database to prepare the jobs and possibly to monitor the production u “Pro-active” use of the “magda” replica catalog nWe intend to run the “pile-up” production as much as possible where the data is u But we have already newcomers (countries and institutes) nWe do not intend to send all the pile-up data to CERN nScenari to access the data for reconstruction and analysis are being studied u Use of Grid tools is ‘seriously’ considered

November 11, 2002G. Poulard - NorduGrid Workshop38 ATLAS DC1/Phase II: October 2002-January 2003 Goals : Produce the data needed for the HLT TDR Get as many ATLAS institutes involved as possible Worldwide collaborative activity Participation : 43 Institutes nAustralia nAustria nCanada nCERN nChina nCzech Republic nDenmark nFrance nGermany nGreece nIsrael nItaly nJapan nNorway nRussia nSpain nSweden nTaiwan nUK nUSA

November 11, 2002G. Poulard - NorduGrid Workshop39 ATLAS Planning for Grid Activities nAdvantages of using the Grid: u Possibility to do worldwide production in a perfectly coordinated way, using identical software, scripts and databases. u Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system. u Possibility to execute tasks and move files over a distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords). nWhere we are now: u Several Grid toolkits are on the market. u EDG – probably the most elaborated, but still in development. u This development goes way faster with the help of the users running real applications.

November 11, 2002G. Poulard - NorduGrid Workshop40 Present Grid Activities nAtlas already used Grid test-beds in DC1/1 u 11 out of 39 sites ( ~5% of the total production) used Grid middleware:  NorduGrid (Bergen, Grendel, Ingvar, ISV, NBI, Oslo, Lund, LSCF) all production done on the Grid  US Grid test-bed (Arlington, LBNL, Oklahoma; more sites will join in the next phase) used for ~10% of US DC1 production (10% = 900 CPUdays)

November 11, 2002G. Poulard - NorduGrid Workshop in addition nATLAS-EDG task-force u with 40 members from ATLAS and EDG (led by Oxana Smirnova) u used the EU-DataGrid middleware to rerun 350 DC1 jobs in some Tier1 prototype sites: CERN, CNAF, Lyon, RAL, NIKHEF and Karlsruhe ( CrossGrid site)  done in the first half of September) nGood results have been achieved: u A team of hard-working people across the Europe u ATLAS software is packed into relocatable RPMs, distributed and validated u DC1 production script is “gridified”, submission script is produced u Jobs are run at a site chosen by the resource broker nStill work needed (in progress) for reaching sufficient stability and easiness of use u Atlas-EDG continuing till end 2002, interim report with recommendations is being drafted

November 11, 2002G. Poulard - NorduGrid Workshop42 Grid in ATLAS DC1/1 US-ATLAS EDG Testbed Prod NorduGrid

November 11, 2002G. Poulard - NorduGrid Workshop43 Plans for the near future nIn preparation for the reconstruction phase (spring 2003) we foresee further Grid tests in November u Perform more extensive Grid tests. u Extend the EDG to more ATLAS sites, not only in Europe. u Test a basic implementation of a worldwide Grid. u Test the inter-operability between the different Grid flavors.  Inter-operation = submit a job in region A, the job is run in region B if the input data are in B; the produced data are stored; the job log is made available to the submitter.  The EU project DataTag has a Work Package devoted specifically to interoperation in collaboration with US IvDGL project: the results of the work of these projects is expected to be taken up by LCG (GLUE framework).

November 11, 2002G. Poulard - NorduGrid Workshop44 Plans for the near future (continued) nATLAS is collaborating with DataTag-IvDGL for interoperability demonstrations in November u How far we can go we will see during the next week(s) when we will discuss with technical experts. nThe DC1 data will be reconstructed (using ATHENA) early 2003: the scope and way of using Grids for distributed reconstruction will depend on the results of the November/December tests. u ATLAS is fully committed to LCG and to its Grid middleware selection process  our “early tester” role has been recognized to be very useful for EDG.  We are confident that it will be the same for LCG.

November 11, 2002G. Poulard - NorduGrid Workshop45 Long Term Planning nWorldwide Grid tests are essential to define in detail the ATLAS distributed Computing Model. n ATLAS members are already involved in various Grid activities and take also part in inter-operability tests. In the forthcoming DCs this will become an important issue. nAll these tests will be done in close collaboration with the LCG and the different Grid projects.

November 11, 2002G. Poulard - NorduGrid Workshop46 DC2-3-4-… nDC2: Q3/2003 – Q2/2004 u Goals  Full deployment of EDM & Detector Description  Geant4 replacing Geant3 (fully?)  Pile-up in Athena  Test the calibration and alignment procedures  Use LCG common software (POOL; …)  Use widely GRID middleware  Perform large scale physics analysis  Further tests of the computing model u Scale  As for DC1: ~ 10 7 fully simulated events nDC3: Q3/2004 – Q2/2005 u Goals to be defined; Scale: 5 x DC2 nDC4: Q3/2005 – Q2/2006 u Goals to be defined; Scale: 2 X DC3

November 11, 2002G. Poulard - NorduGrid Workshop47 Summary (1) nWe learnt a lot from DC0 and the preparation of DC1 u The involvement of all people concerned is a success u The full production chain has been put in place u The validation phase was “intensive”, “stressing” but it is a “key issue” in the process u We have in hands the simulated events required for the HLT TDR u Use of Grid tools looks very promising

November 11, 2002G. Poulard - NorduGrid Workshop48 Summary (2) nFor DC1/Phase II: u Pile-up preparation is in good shape u The introduction of the new EDM is a challenge by itself u Release 5 (November 12) should provide the requested functionality u Grid tests are scheduled for November/December u Geant4 tests should be ready by mid-December

November 11, 2002G. Poulard - NorduGrid Workshop49 Summary (3) nAfter DC1 u New Grid tests are foreseen in 2003 u ATLAS is fully committed to LCG  As soon as LCG-1 will be ready (June 2003) we intend to actively participate to the validation effort  Dates for next DCs should be aligned to the deployment of the LCG and Grid software and middleware

November 11, 2002G. Poulard - NorduGrid Workshop50 Summary (4): thanks to all DC-team members A-WP1: Event generation A-WP2: Geant3 simulation A-WP3: Geant4 Simulatio n A-WP4: Pile-up A-WP4: Pile-up A-WP5: Detector response A-WP6: Data Conversio n A-WP6: Data Conversio n A-WP7: Event filtering A-WP9: Analysis A-WP10: Data Management A-WP10: Data Management A-WP8: Reconstructio n A-WP11: Tools A-WP12: Teams Production Validation …. A-WP12: Teams Production Validation …. A-WP13: Tier Centres A-WP14: Fast Simulatio n