LHCb Readiness for Run 2 2015 WLCG Workshop Okinawa Stefan Roiser / CERN IT-SDC for LHCb Distributed Computing

LHCb Run2 Readiness - StR Content Online changes with impact for Offline Offline Data Processing Offline Data Management Services LHCb relies on 11 April '15 LHCb Run2 Readiness - StR

Preamble – LHC Evolution Run 1 Planned for Run 2 Max beam energy 4 TeV 6.5 TeV Transverse beam emittance 1.8 μm 1.9 μm β* (beam oscillation) 0.6 m / LHCb 3 m 0.4 m / LHCb 3 m Number of bunches 1374 2508 Max protons per bunch 1.7 * 1011 1.15 * 1011 Bunch spacing 50 ns 25 ns LHC Maximum Luminosity 7.7 * 1033 cm-2s-1 1.6 * 1034 cm-2s-1 LHCb Maximum Luminosity 4 * 1032 cm-2s-1 LHCb μ (avg # collisions/crossing) 1.6 1.2 ATLAS & CMS LHCb NB: LHCb uses “luminosity leveling”, ie. the “in time pile up” and so the instantaneous luminosity stays constant for LHCb during an LHC fill 11 April '15 LHCb Run2 Readiness - StR

Pit & Online

LHCb Run2 Readiness - StR Trigger Scheme Hardware trigger reduce event rate to ~ 1 MHz High Level Trigger computing farm split into HLT1 with partial event reconstruction, output will be buffered on local disks HLT 1 output used for detector calibration and alignment (O(hours)). (was done offline in Run 1) HLT2 runs deferred with signal event reconstruction very close to offline reconstruction 12.5 kHz event rate to OFFLINE At ~ 60kB event size this is ~ 750 MB/s Event rate was 4.5 kHz in Run 1 NB: Because of deferred trigger, very little availability of HLT for offline data processing See also Marco’s talk tomorrow on further evolution for future Runs 11 April '15 LHCb Run2 Readiness - StR

HLT Output Stream Splitting 12.5 kHz to Storage 10 kHz Full (+Parked) Stream 2.5 kHz Turbo Stream 10 kHz go to classic Offline reconstruction / stripping on distributed computing resources If needed part of this can be “parked” and processed in LS 2 New concept of “Turbo Stream” in Run 2 for ~ 2.5 kHZ i.e. wherever sufficient, take the HLT output with its event reconstruction directly for physics analysis Initially RAW information included, will be stripped off Offline S. Benson, “The LHCb Turbo Stream”, T1, Village Center, Thu 10am, 11 April '15 LHCb Run2 Readiness - StR

Data Processing

LHCb Run2 Readiness - StR Legend: Offline Processing Workflow Application File Type Storage Element (M)DST Stripping RAW Reconstr. X 5GB, 1x FULL.DST Tape X RAW 24h Reconstr. 5GB, 1x FULL.DST Buffer 3GB, 1x Buffer 6h Stripping The RAW input file is available on Disk Buffer Reconstruction runs ~ 24 h, 1 input RAW, 1 output FULL.DST to Disk Buffer Asynchronous migration of FULL.DST from Disk Buffer to Tape Stripping (DaVinci) runs on 1 or 2 input files (~ 6h/file), output several unmerged (M)DST files (one per “stream”) to Disk Buffer Input FULL.DST removed from Disk Buffer asynchronously Rerun the above workflows for one run Once a stream reaches 5 GB of unmerged (M)DSTs (up to O(100) files), Merging runs ~ 15 – 30 mins, output one merged (M)DST file to Disk Input (M)DST files removed from Disk Buffer asynchronously … unmerged (M)DST O(MB) 1x Buffer X Merging 30m (M)DST … 5GB, 1x Disk 27 Jan '15 11 April '15 Run2 Operations - StR LHCb Run2 Readiness - StR 8

Offline Data Processing Changes What is reconstructed offline is supposed to be the final reconstruction pass Calibration / Alignment from HLT used also offline No reprocessing (reco) foreseen before end of Run 2 Expecting a higher stripping retention because of calibration and alignment done ONLINE Partly damped by moving most physics streams to M(icro)DST format (Note: MDST O(10kB/Event), DST O(120kB/Event)) All files from one “LHCb Run” are forced to reside on the same storage A run is the smallest granularity for physics analysis files E.g. would reduce impact in case a disk breaks Workflow execution is now also possible on Tier 2 sites Needed because of increase of collected data 11 April '15 LHCb Run2 Readiness - StR

Workflow Execution Location GRIDKA RAW Reco Strppg Merge FULL.DST unm. DST DST CNAF RAW Reco FULL.DST Manchester RAW Reco FULL.DST Manchester RAW Reco FULL.DST CNAF RAW Reco Strippg Merge FULL.DST unm. DST DST X X Data Processing workflow executed by default at Tier 0/1 sites (stays the same as in Run 1) For Run 2 in addition we allow A Tier 2 site to participate for a certain Job Type remotely (most useful would be Reconstruction) Any Tier 2 is allowed at any time to participate on any Job Type (no static 1 to 1 “attaching” anymore) In principle the system also allows for ANY site to participate on any Job Type remotely Monarc (technically) 11 April '15 LHCb Run2 Readiness - StR

LHCb Run2 Readiness - StR All Workflows Run 1 Run 2 Data Processing T 0 / 1 T 0 / 1 / 2 Monte Carlo T 2 (can also run on T 0 / 1 sites if resources available) User analysis T 0 / 1 / 2D (without input data can also run on T2) Very flexible computing model allows almost all workflows to be executed on every tier level / resource type Interested in running multicore jobs – especially on VMs – but no pressing need for it “Elastic MC” – knows the work / event, at start of the payload will calculate on the fly how many events to produce until the “end of the queue” User analysis – least amount of work but highest priority in the central task queue F. Stagni, “Jobs masonry with elastic Grid Jobs”, T4, B250, Mo 5pm 11 April '15 LHCb Run2 Readiness - StR

LHCb Run2 Readiness - StR Compute Resources Non virtualized Virtualized “Classic Grid” – CE, batch system, … BOINC – volunteer computing Non pledged – commercial, HPC, … Vac – self managed cloud resources HLT farm – little use during Run 2 Vcycle – interaction via IaaS A. McNab, “Managing virtual machines with Vac and Vcycle”, T7, C210, Mo 5pm Expect rampup of virtualized infrastructures during Run 2 All environments served by the same pilot infrastructure talking to one LHCb/DIRAC central task queue F. Stagni, “Pilots 2.0: DIRAC pilots for all the skies”, T4, B250, Mo 2pm 11 April '15 LHCb Run2 Readiness - StR

Data Management

LHCb Run2 Readiness - StR Data Storage Introduced concept of Tier 2D(isk) sites i.e. Tier 2 sites with disk areas >= 300 TB No more direct processing from “tape caches” foreseen Interact with disk buffer via FTS3 and process from there E.g. pre-staging “Legacy Run 1 Stripping” data Should lead to reduction of disk cache size in front of tape 11 April '15 LHCb Run2 Readiness - StR

LHCb Run2 Readiness - StR Data Storage (ctd) C. Haen, “Federating LHCb datasets using the Dirac File Catalog”, T3, C209, Mo 4.45pm Catalogs File Catalog: provides replica information, recently migrated from LCG File Catalog to the Dirac File Catalog Bookkeeping (unchanged) provides data provenance information Data Popularity Data collected since 2012 M. Hushchyn, “Disk storage mgmt for LHCb on Data Popularity”, T3, C209, Tue 6.15pm 11 April '15 LHCb Run2 Readiness - StR

Data Access Operations Gaudi Federation In use since last fall. LHCb analysis jobs create a local replica catalog for their input data. If the local copy is not available -> fall back to next remote replica. Data access protocols SRM Shall be in use for tape interactions … and for writing to storage (job output upload, data replication) Xroot LHCb will be constructing turls for input data on the fly without SRM interaction for disk resident data access Need single and stable xroot endpoint per storage element HTTP/WEBDAV All storage sites are equipped with Http/Webdav access Could be used as second access protocol 11 April '15 LHCb Run2 Readiness - StR

Underlying Services

LHCb Run2 Readiness - StR Services CVMFS Building block for LHCb distributed computing, distributes all software and conditions data CernVM Vac, vcycle, BOINC are using CernVM 3 FTS 3 Vital for LHCb WAN transfers and tape interaction (pre-staging of input data) Several WLCG monitoring services in use SAM 3, dashboards, network monitoring Working on perfSonar data extraction into LHCbDIRAC HTTP Federation Builds on top of http/webdav access, provides easy access to LHCb data namespace Development on top ongoing for data consistency checks F. Furano, “Seamless access to LHCb HTTP/WebDAV storage”, Mo/Tue, Poster Sess. A 11 April '15 LHCb Run2 Readiness - StR

LHCb Run2 Readiness - StR Summary LHCb is ready for Run 2 Several changes introduced in Run 2 Calibration/Alignment in the HLT farm Closer integration of Tier 2 sites in data processing New Dirac file replica catalog deployed Disk resident data access via direct xroot CVMFS and FTS3 are key “external” services 11 April '15 LHCb Run2 Readiness - StR

LHCb Run2 Readiness - StR Goodie page http://lhcb-web-dirac.cern.ch/DIRAC/LHCb-Production/undefined/grid/SiteStatus/display?name=LCG.RAL.uk 11 April '15 LHCb Run2 Readiness - StR