2007 Workshop on INFN ComputingRimini, 7 May 2007 Conditions Database Service implications at Tier1 and Tier2 sites Andrea Valassi (CERN IT-PSS)

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 What are Conditions Data? Describe the detector at the time of data taking –Event data (beam collisions) vs. conditions data (detector environment) –Conditions data vary with time and may exist in several versions Several sources of conditions data –Online: DCS (temperatures…), configuration (trigger…) and monitoring Generally single version (results of a measurement) –Offline: calibration and alignment Generally multi-version (results of a computation) This talk will focus on offline conditions databases –Online and offline conditions needed for offline event reconstruction –Main use case: retrieve the conditions data valid for one event/run

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Conditions DB in the 4 experiments ALICE –AliRoot (Alice-specific software) for time/version handling –ROOT files with AliEn file catalog ALICE-managed database infrastructure (AliEn MySQL at T0) CMS –CMSSW (CMS-specific software) for time/version handling –Oracle (via POOL-ORA C++ API) with Frontier web cache 3D + CMS database infrastructure (Oracle/Frontier at T0, Squid at T1/T2) ATLAS and LHCb –COOL (LCG AA common software) for time/version handling –Oracle, MySQL, SQLite, Frontier (via COOL C++ API) 3D + Alice/LHCb database infrastructure (Oracle at T0/T1 with Streams)

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ALICE – use Alien/ROOT ALICE has no special needs for DB deployment/replication at T1 and T2 sites for managing its conditions data. AliEn Grid file catalog uses a 2-node MySQL database cluster at T0 (not replicated at T1/T2)

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ALICE – OCDB Grid file catalog Offline Calibration DB: conditions data in AliEn file catalog –Metadata for ROOT files containing calibration and alignment data Files produced by many sources are registered into OCDB by a ‘shuttle’ –OCDB ROOT files are replicated to all T1’s (and on request to T2’s) Two types of conditions data –“Short-lived” (one update/run, ~2000/year) MB per data set –“Long-lived” (few updates/year) MB per data set –Total data size per year ~200 GB (short-lived) plus ~1 GB (long-lived) R/O data access from Grid jobs –ROOT files are downloaded to the local worker node

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ALICE – Dress Rehearsal 2007

Andrea Valassi Conditions DatabasesRimini, 7 May D project (CMS, ATLAS, LHCb) Barbara Martelli, INFN T1/T2 Cloud Workshop, November 2006

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Oracle Streams (Capture, Propagation, Apply) Barbara Martelli, INFN T1/T2 Cloud Workshop, November 2006

Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Streams downstream capture Objective –Source database (CERN) isolation against network or database problems at the replicas Status and plans –Four nodes allocated –Production setup with two servers (Atlas and LHCb) and two spares For LHCb will also be used for LFC replication Eva Dafonte Perez (IT-PSS)

10 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Frontier Marc Paterno, CHEP 2004 T1/T2: Squid T0: Squid T0: Tomcat T0: Oracle

11 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 CMS – use Oracle/Frontier Vincenzo Innocente, CMS Software Week April 2007

12 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 CMS – conditions data at CERN Vincenzo Innocente, CMS Software Week, April 2007 ORCON and ORCOFF conditions data are in POOL_ORA format ORCON-ORCOFF Oracle Streams prototype set up (integration RAC). Production set up later in 2007.

13 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 CMS – conditions data distribution Vincenzo Innocente, CMS Software Week April 2007

14 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 CMS – deployment status Conditions data access via Frontier already successfully tested in CSA Many performance improvements since that time (improve cache reuse, fix many problems – D. Dykstra)

15 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 COOL: LCG Conditions database –Common development of IT-PSS, LHCb and ATLAS –Manage time variation and versioning of conditions data Four supported relational technologies (via CORAL) –Oracle and MySQL database servers –SQLite files –Frontier (read-only ): Squid + Tomcat + Oracle server COOL service deployment model –Based on generic 3D distributed db deployment model Oracle at T0 and T1 (Streams replication with downstream capture) Other technologies elsewhere if at all needed –Details depend on each experiment’s computing model ATLAS and LHCb – use COOL P. Mato

16 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 LHCb computing model COOL only stores the conditions data needed for offline event reconstruction –Low data volume ~ 1 GB/year COOL deployment model –Oracle at Tier0 –Oracle at Tier1’s (six sites) –Streams replication T0-T1 –COOL not needed at Tier2’s (only MC production there) SQLite files may in any case be used for any special needs (Marco Clemencic, 3D workshop 13 Sep 2006)

17 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 LHCb – COOL service model Two servers at CERN – essentially for online and offline –Replication to Tier1’s from the online database is a two-step replication –Online server at the pit (managed by LHCb); offline server in the CC GRIDKA RAL IN2P3 CNAF SARA PIC (Marco Clemencic, 3D workshop 13 Sep 2006) COOL (Oracle)

18 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 LHCb - status and plans Setup and tests of Oracle servers and Streams replication –CERN Online (prototype in CC), CERN Offline (integration RAC) and five Tier1 sites (GRIDKA, RAL, IN2P3, CNAF, SARA), with Streams downstream capture –Tested 2-step replication (Online-Offline-T1) at rates higher than expectations All OK with 100 IOVs/sec for one hour and 1 IOVs/sec for 24 hours –Tested simple read-back from T1 database Verified that latency in tag replication is negligible Future milestones –May 2007: complete T0/T1 deployment (CERN production offline RAC, add PIC) –June 2007: software release using LFC replica service (CORAL 1.8.0) DB lookup and secure user authentication using grid certificate (retrieve encrypted username and password – later on also secure SSH data transfer in LFC) –Summer 2007: test reconstruction at T1 sites using Oracle Streams replicas Conditions data for the ongoing data challenge are distributed as SQLite files –Later in 2007: online database server at the pit Online database prototype in the CC has already been switched off

19 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – COOL service model COOL Oracle services at Tier0 and ten Tier1’s –Two COOL servers at CERN for online/offline (similar to LHCb) Online database within the Atlas pit network, but physically in the CC –In addition: Oracle at three ‘muon calibration center’ Tier2’s Online OracleDB Offline master CondDB Tier-0 Sqlite replica (1 file/run) Tier-1 replica Tier-1 replica Online / PVSS / HLT farm Tier-0 farm Dedicated 10Gbit link ATLAS pitComputer centreOutside world CERN public network Calibration updates Streams replication ATLAS pit network (ATCN) gateway (Sasha Vaniachine and Richard Hawkings, 3D workshop 14 Sep 2006) GRIDKA TAIWAN RAL IN2P3 CNAF SARA BNL TRIUMF PIC Nordugrid

20 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – production setup Gancho Dimitrov, Florbela Viegas, Marco Clemencic 3D Amsterdam, March 2007 Online  Offline: both COOL and non-COOL

21 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – current test setup Gancho Dimitrov, Florbela Viegas, Marco Clemencic 3D Amsterdam, March 2007

22 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – COOL schemas Gancho Dimitrov, Florbela Viegas, Marco Clemencic 3D Amsterdam, March 2007 DCS ~300 GB/year (largest component) ONL ~few GB/year OFL ~60 GB/year

23 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – status and plans Setup and tests of Oracle servers and Streams replication –Production Online and Offline RAC at CERN and six Tier1 sites (GRIDKA, RAL, IN2P3, CNAF, SARA, Taiwan), with Streams downstream capture –Ongoing “mini-CDC” production with 2-step replication (Online-Offline-T1) Continuous insertion of calibration-like CLOB data (30MB every 30min) Read back T1 data ~30 minutes after insertion at T0 –5% failure for most distant site (Streams latency – but the data gets there eventually) Future plans/milestones –May 2007: mini-CDC tests with DCS-like data (higher LCR/s for the same MB/s) –May 2007: software release 13 with replica sorting in dblookup.xml (CORAL 1.8.0) –June 2007: “closed loop” calibration/alignment challenge Realistic calibration setup (but lower data rates than current mini-CDC tests) The muon calibration centers will be tested later (~September) –Summer 2007: complete T1 deployment (add PIC, Nordugrid; fix BNL, Triumf)

24 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – open issues Can the replication throughput meet the Atlas requirements? –Intrinsic limitation of Oracle Streams: single row updates in apply step Each LCR (logical change record) is replicated individually Current mini-CDC tests are very important to validate rates for COOL –Investigating transportable tablespaces for TAGS Higher data volume (6TB/year), much higher LCR/s (finer granularity) Replication to Tier2’s –Several use cases Simulation, calibration, end user analysis –Investigating several possibilities SQLite files are enough for some use cases (e.g. simulation) Evaluating COOL ‘dynamic replication’ to MySQL at remote sites Evaluating COOL Frontier backend (performance, cache consistency…)

25 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – Streams tests Barbara Martelli, May 2007 Current Atlas mini-CDC tests (CNAF replica) Steady-state operation at ~80 LCR/s Gancho Dimitrov et al, SARA, March 2007 Oracle Streams stress tests (RAL replica) Maximum achieved ~1400 LCR/s

26 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 ATLAS – muon calibration centers (Sasha Vaniachine and Joe Rothberg, 3D workshop 14 Sep 2006) Muon calibration centers: Roma, Michigan, Munich Streams set up between Michigan (source) and CERN (target)

27 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Summary *Not present in the table: online conditions other than DCS/PVSS (configuration for DAQ/trigger, monitoring…) Data volume for these data is generally smaller than for calibration/alignment or DCS CondDB. CondDB technologyCalibration and alignment (~Offline conditions) DCS from PVSS/Oracle (~Online conditions *) ALICEROOT files + Alien file catalog Alien uses MySQL cluster at T0 (DB is not replicated at T1/T2) CondDB: 200 GB/year ~100 MB/run (short-lived) ~150 MB/month (long-lived) (CondDB: included in 200 GB/y) PVSS archive to ROOT files Full PVSS archive: 7 TB/year CMS POOL-ORA (Oracle at T0) FroNTier launchpad at T0 Squid web cache at T0/T1/T2 CondDB: <500 GB/year (DB disk space requirements) 100 GB/Squid, 2-3 Squid/site (CondDB: included in <500 GB/y) P5 online DB disk space 500 GB ATLASCOOL Oracle at T0/T1 (Streams/3D) Sqlite/MySQL/Frontier at T2? CondDB: 60 GB/year ~20 GB/year (first-pass rec.) ~40 GB/year (re-rec.) CondDB: 300 GB/year One PVSS2COOL job/15 min Full PVSS archive ~1-2 TB/year LHCbCOOL Oracle at T0/T1 (Streams/3D) Sqlite at T2 (only if needed) CondDB: 1 GB/year? (CondDB: included in 1 GB/y) DCS CondDB filled by PVSS job Full PVSS archive < 100 GB/year?

28 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Conclusions ALICE: use ROOT and Alien (no db services outside CERN) CMS: Oracle and Frontier at T0, Squid web caches at T1/2 –Frontier already successfully tested during CSA06 and 2007 productions –Oracle Streams between online and offline servers at T0 –No database service is required outside CERN for CMS LHCb (COOL): Oracle at T0 and T1 with Streams replication –Two servers at CERN: one at the pit (online) and one in the CC (offline) –No database service is required at T2’s for LHCb –Test Grid reconstruction (reading COOL from 6 T1 replicas) in summer 2007 Atlas (COOL): Oracle at T0 and T1 with Streams replication –Two servers at CERN, both in the CC (online server in the online network) –“Mini-CDC” high-rate tests ongoing with 6 T1 sites –Test “closed loop calibration” in summer 2007 –No single solution for Tier2 deployment – evaluating MySQL and Frontier

29 Andrea Valassi Conditions DatabasesRimini, 7 May 2007 Acknowledgements ALICE: Latchezar Betev CMS: Lee Lueking, Nicola De Filippis, Vincenzo Innocente Atlas: Richard Hawkings, Gancho Dimitrov, Jim Cook LHCb: Marco Clemencic, Philippe Charpentier IT-PSS, 3D: Eva Dafonte, Barbara Martelli, Dirk Duellmann Thanks to all for your help!