ALICE Online Upgrade P. VANDE VYVRE – CERN/PH ALICE meeting in Budapest – March 2012

A quick look to the past: the DDL and the D-RORC Project started in 1995 Strong push from the management: George Vesztergombi and now Peter Levai Work of a solid and competent team: Ervin Denes, Tivadar Kiss, Gyorgy Rubin, Csaba Soos, and many others contributed to this successful project The Budapest team delivered to the ALICE experiment a common and reliable solution for the data transfer from the detectors to the online systems : the DDL and the D-RORC ALICE Mar.20122P. VANDE VYVRE CERN-PH

Present Online Architecture ALICE Mar.2012P. Vande Vyvre – CERN/PH3 GDC TDSM CTP LTU TTC FERO LTU TTC RCU LDC BUSY Rare/All Event Fragment Sub-event Event File Storage Network PDS L0, L1a, L2 360 DDLs D-RORC EDM LDC D-RORC Load Bal. LDC D-RORC HLT Farm FEP H-RORC DDL H-RORC 10 DDLs 10 D-RORC 10 HLT LDC 120 DDLs DA DQM DSS Event Building Network 430 D-RORC 125 Detector LDC 75 GDC 30 TDSM 18 DSS 60 DA/DQM 75 TDS Archiving on Tape in the Computing Centre (Meyrin)

DDL and D-RORC ALICE Mar.2012P. Vande Vyvre – CERN/PH4 ~ 500 DDLs and ~450 D-RORCs Collect all the data of ALICE (~2 PB/year) Configure the electronics of several detectors, in particular the biggest one (TPC) Goal for the future: Repeat this success story !

ALICE Upgrade Strategy Input from Luciano and PBM, Thorsten and Volker, DAQ team Strategy recently approved by ALICE presents a global and coherent plan to upgrade the experiment for 2018 (Long Shutdown 2 LS2) “Upgrade Strategy for the ALICE Central Barrel” Key concepts for running the experiment at high rate Pipelined electronics Triggering the TPC would limit this rate → Continuous readout Reduce the data volume by topological trigger and online reconstruction Major upgrade of the detector electronics and of the online systems Online upgrade design: Data buffer and processing off detector Fast Trigger Processor (FTP): 2 hw trigger levels to accommodate various detector latencies and max. readout rate HLT: 2 sw trigger levels: ITS, TRD, TOF, EMC (reduce the rate before building TPC event) Final decision using the data from all detectors Common DAQ /HLT farm to minimize cost while preserving the present flexibility of running modes ALICE Mar.20125P. VANDE VYVRE CERN-PH

Trigger Rates ALICE Mar.20126P. VANDE VYVRE CERN-PH Key concepts for running ALICE at high rate Hardware trigger-less architecture whenever needed and affordable Pipelined electronics and continuous readout Reduction of the data volume by topological trigger and 2 steps online reconstruction Trigger Levels Detectors Trigger with pp beams Trigger with Pb-Pb beams Frequency (kHz) LatencyFrequency (kHz) Latency No TriggerITS, TPC, TRD, EMCal, PHOS Continuous read-out at 10 MHz Level 0 (hw)TOF (Pb-Pb)  s  s Level 1 (hw)TOF (p-p), Muon  s  s Level 2 (sw) s Level 3 (sw) s

Present TPC Readout ALICE Mar.20127P. VANDE VYVRE CERN-PH RCU CTP FEC DAQ HLT MB/s 216 DDL 2.0 Gb/s 216 DDL 2.0 Gb/s FEC TTC FEC MB/s FEC 1 13 Present readout : Links: DDL at 2 Gb/s. 216 DDLs for the TPC used at 1.6 Gb/s PC adapters: D-RORC and H-RORC Up to 6 RORCs/PC DRORCDRORC DRORCDRORC HRORCHRORC HRORCHRORC

Future evolution of DDL & RORC ALICE Mar.2012P. Vande Vyvre – CERN/PH8 New common DAQ/HLT DDL2 and RORC2 Prototype under design by Heiko Engel (Udo Kebschull’s team/Frankfurt) and Tivadar Kiss (Budapest) (Ready in 2012) It will address the upgrade needs with the present architecture for the period ( ) between Long Shutdown 1 (LS1) and LS2 Includes 12 DDL2 links at 6 Gb/s. 6 links to DAQ LDC  36 Gb/s. PCIe V2 8 lanes (500 MB/s/lane)  32 Gb/s of I/O capacity. Data processing in the FPGA (e.g cluster finding) Currently: 5 links at 2Gb/s per PC  10 Gb/s of I/O capacity Prototype under development 12 links at 6 Gb/s 6 links to DAQ LDC  36 Gb/s. PCIe Gen2 8 lanes (500 MB/s/lane)  32 Gb/s of I/O capacity Final system 12 links at 10 Gb/s per PC PCIe Gen3 16 lanes  I/O 128 Gb/s

Readout Upgrade ALICE Mar.20129P. VANDE VYVRE CERN-PH FEC2 EPN or 40 Gb/s FLP 10 Gb/s ~7000 Network DAQ and HLT systems or 40 Gb/s Upgrade readout: Non-zero suppressed TPC data 57 Tb/s (570 kchannels x 10 bits x 10 MHz) ~7000 links at 10 Gb/s for TPC. ~7800 links for the whole experiment. TBD: FEC2 characteristics (GEM readout, very simple FEC vs more links) DDL3 and RORC3 for LS2 upgrade (ALICE common solution) Address the needs of the new architecture for the period after LS2 (2018) DDL3 links at 10 Gb/s Exact requirements to be addressed (radiation tolerance/hardness) Different physical layers possible (CERN GBT, Eth, FCS, wavelength multiplex.) First-Level Processors (FLPs) ~650 FLPs needed: 12 detector links 10 Gb/s, 1 network link at 10 or 40 Gb/s Data readout and first-level data processing: ZS, cluster finder and compression could be performed by the RORC FPGA Regional online reconstruction

Continuous Readout ALICE Mar.2012P. Vande Vyvre – CERN/PH10 Timet+3xt+2xt+xtt-xt-2x TRD EMC TPC ITS TOF Events n+2n+1n FLP Event Building & Processing

Online reconstruction ALICE Mar.2012P. Vande Vyvre – CERN/PH11 50 kHz Pb–Pb collisions inspected with the least possible bias using topological triggers and on-line particle identification HI run 2011: online cluster finding data compression factor of 4 for TPC Two HLT scenarios for the upgrade: 1. Partial event reconstruction: Further factor of 5 → overall reduction factor of 20. Close to what could be achieved and tested now. Rate to tape: 5 kHz. 2. Full event reconstruction: Overall data reduction by a factor of 100. Rate to tape: 25 kHz. HI run 2018: min. bias event size ~75 MB  ~4-1 MB after data volume reduction. Throughput to mass storage: 20 GB/s. Target rate after LS2 can only be reached with online reconstruction Build on success of last year with TPC cluster finding Extremely ambitious: online calibration and data processing Essential to gain experience with using online results as basis of offline reconstruction, e.g. the cluster finding. Mandatory to reach the necessary data reduction factor after LS2.

Processing Power ALICE Mar.2012P. Vande Vyvre – CERN/PH12 Current HLT Currently HLT processing rates with full TPC reconstruction: Either ~200 Hz central events (65 MB) or ~800Hz min bias events 216 front-end nodes (equivalent to the FLPs) TPC cluster finder on H-RORC FPGA and coordinates transformation by the CPU 60 tracking nodes with GPU (equivalent of the EPNs) HLT Assuming linear scaling of rates and using current HLT technology: Level 2 (reconstruction using ITS, TRD, EMC) 50 kHz min bias events (15 MB, ~4 times less CPU) → 50 kHz/800Hz*60 nodes*1/4= 1250 nodes (CPU+GPU) Level 3 (full reconstruction incl. TPC) 25 kHz central events → 25 kHz/200Hz*60 nodes = 7500 nodes (CPU+GPU) 8750 nodes of 2011 → 1250 nodes of 2018 Event Processing nodes ~1250 EPNs Current HLT processing nodes: Intel Xeon E5520, 2.2 GHz 4 cores Global track merging + reconstruction, some trigger algorithms Nvidia GTX480/580 ~90% of the tracking Performance gain 33%/yr during 7 years Factor 7 from 2011 to 2018 Overall chip performance gain → Significant R&D on computing and algorithms concepts

Network Routers and Switches for the Event Building ALICE Mar.2012P. Vande Vyvre – CERN/PH13 Present DAQ network (Force 10 now DELL) Exascale E1200i 3.5 Tb/s Capacity of present DAQ router not adequate Technology 1: Ethernet (Brocade MLX Series) MLX32e up to 15.3 Tb/s, 32 line card slots Up to 256x10GbE ports, 32x100GbE ports Line cards: 2x100 Gb/s or 8x10 Gb/s Technology 2: Infiniband (Mellanox) MIS6500 up to 51.8 Tb/s, 648 ports at 40 Gb/s

Network Layout Infiniband ALICE Mar.2012P. Vande Vyvre – CERN/PH14 Event building and HLT network - 3 data flows - Total network throughput: 17.2 Tb/s - Input to HLT L2: 10 MB/evt at 50 kHz → 500 GB/s → 4.0 Tb/s All detectors but TPC - Input of TPC to HLT L3: 65 MB/evt at 25 kHz → 1625 GB/s → 13.0 Tb/s - Output of HLT L3 (data recording):20 GB/s → 0.2 Tb/s Two possible technologies (Ethernet and Infiniband) 120 x 40 Gb/s … Infiniband (Mellanox) MIS6500 up to 51.8 Tb/s, 648 ports at 40 Gb/s Data traffic: ~650 FLPs and ~1250 EPN Total ~1900 ports 40 GbE. Satellite routers: 500 ports: data traffic to/from the nodes 120 ports: data traffic to/from the main router Total throughput: 4x120x40 Gb/s = 19.2 Tb/s

Upgrade Online (after LS2) ALICE Mar P. VANDE VYVRE CERN-PH 10 Gb/s or 40 Gb/s FLP 10 Gb/s L2 Network DAQ and HLT DAQ and HLT 10 Gb/s or 40 Gb/s FLP EPN FLP TPC TRD Muon FTP L0 L1 FLP TOF EPN FLP ITS L3 FLP PHOS FLP EMC Trigger Detectors ~ 650 FLPs~1250 EPNs ~7800 DDL3

Next Steps ALICE Mar.2012P. Vande Vyvre – CERN/PH16 Major upgrade foreseen for the online systems during LS2 Improve the Strategy Note (Mar ‘12) More detailed and optimized cost estimate (Mar ‘12) Prototyping, some using the present online systems (‘12) Continue to gain experience with using HLT online results as basis of offline reconstruction, e.g. the cluster finding Essential to reach the necessary data reduction factors after LS2 R&D program (‘12-’16) Fast Trigger Processor Readout links Event-building and HLT dataflow network Online reconstruction: online use of “offline software” Efficient use of new computing platforms (many-cores and GPUs) Online Conceptual Design Report (TBD) Requirements from detectors Key results from R&D