Filippo Costa ALICE DAQ ALICE DAQ future detector readout October 29, 2012 CERN

Current status and evolution LS2 upgrade Architectural issues Development process Present R&D activities 29/10/2012Filippo Costa, CERN 2 Outline

29/10/2012Filippo Costa, CERN 3 Trigger – DAQ – HLT GDC TDSM CTP LTU TTC FERO LTU TTC FERO LDC BUSY Rare/All Event Fragment Sub-event Event File Storage Network (8GB/s) PDS L0, L1a, L2 360 DDLs D-RORC EDM LDC D-RORC Load Bal. LDC D-RORC HLT Farm FEP DDL H-RORC 10 DDLs 10 D-RORC 10 HLT LDC 120 DDLs DA DQM DSS Event Building Network (20 GB/s) 430 D-RORC 175 Detector LDC 75 GDC 30 TDSM 18 DSS 60 DA/DQM 75 TDS Archiving on Tape in the Computing Centre (Meyrin)

29/10/2012Filippo Costa, CERN 4 DDL current status (Oct ‘12) DETECTOR# DAQ DDL# HLT DDL ACORDE10 EMCAL24 FMD33 HMPID140 MTRK20 MUON-TRG22 PHOS12 PMD60 SDD24 SPD20 SSD16 T011 TOF720 TPC216 TRD13 V011 ZDC11 TOT DETECTORs 17 TOT # DAQ DDLs 485 TOT # HLT DDLs 353

29/10/2012Filippo Costa, CERN 5 LS2 Online Upgrade 2x10 or 40 Gb/s FLP DAQ and HLT 10 or 40 Gb/s FLP EPN FLP ITS TRD Muon FTP L0 L1 FLP EMCal EPN FLP TPC Data Storage FLP TOF FLP Farm Network Farm Network PHOS Trigger Detectors ~ 2500 DDL3s 10 Gb/s L0 Data Storage EPN Storage Network Storage Network RORC3 CLK

09/11/2012Filippo Costa, CERN 6 LS2 upgrade requirements (LoI) DetectorMAX R/O rate (kHz) (pp and Pb-Pb) Event Size Pb-Pb after ZS (MB) ITSContinuous0.2 TPCContinuous1.0 TOF200 – 400 TRD27 – EMCal50 Muon5 The rate for heavy-ion events handled by the online systems up to permanent data storage should be increased up to 50 kHz (with a safety factor of 2) corresponding to roughly two orders of magnitude, compared to the present system. Data compression will reduce the input peak data throughput of 1 TByte/s to an average recorded data output of 80 GB/s to a local data storage and 12 GB/s to the computing center.

09/11/2012Filippo Costa, CERN 7 Data compression 1 Cluster finder (RORC3) Trigger Level 0,1 Trigger Level 2 Event-Building Network Detector Digitizers Front-end Pipeline/Buffer Readout Buffer Sub-event Buffer First-Level Processor (FLP) Data compression 2 Event-Building Event-building and Processing Node (EPN) Decision Detector Data Link (DDL3) Trigger & DAQ logical model Decision LHC Clock Distribution of functions as presented in the LoI Fast Trigger Processor Detector Electronics Online System LHC Clock

09/11/2012Filippo Costa, CERN 8 Trigger Level 0,1 Trigger Level 2 Event-Building Network Detector Digitizers Front-end Pipeline/Buffer Readout Buffer Sub-event Buffer First-Level Processor (FLP) Event-Building Data compression 2 (EPN) Decision Detector Data Link (DDL3) Trigger & DAQ logical model Data compression 1 Cluster finder Decision Alternatives scenarios exist: e.g. cluster finder as part of the detector readout LHC Clock

 Initial requirements in the LoI.  Carry on with functional requirements and R&D in parallel  Refine detectors functional requirements for DCS, TRG, DAQ in view of the detector TDRs (2013)  Online R&D to develop prototypes in view of Online/Offline TDR (2014) o DDL2: Prototype characterization Production for the TRD and the HLT o DDL3 Technology selection o Online dataflow demonstrator Detector readout, FLP, network, EPN 9 Development process 09/11/2012Filippo Costa, CERN

Higher reliability Reduced Support Issues well defined Common hardware and protocol 29/10/2012Filippo Costa, CERN 10 Benefit of the commonality So far we used a common protocol and hardware ( SIU – DDL – RORC ) for the read-out of all detectors. Lots of benefits: having the same data transmission protocol for all the detectors reduces specific debugging sessions. It encourages knowledge sharing between the detectors. It creates the “standard” in a custom protocol. We are planning to follow the same idea in the future upgrades, using a common protocol for sending data to the DAQ system.

29/10/2012Filippo Costa, CERN 11 DDL SIU evolution SIU1SIU2SIU3 1 2 Gb/s ACTEL FPGA (CORE cost 560 CHF) 6 Gb/s XILINX / ALTERA /ACTEL FPGA (CORE cost 0 CHF) 10 Gb/s XILINX / ALTERA / ACTEL FPGA (CORE cost 0 CHF) Custom DDL protocol (same protocol but faster) Custom DDL 10 Gb/s 10 Gb/s PCIe over fibre RUN1 LS1LS1 RUN2 LS2LS2 RUN3 Det. Read-Out FPGA SIU IP CORE Det. Read-Out FPGA SIU IP CORE

29/10/2012Filippo Costa, CERN 12 RORC evolution RORC1RORC2 (aka C-RORC)RORC3 TBD 2 2 Gb/s PCIe gen.1 x4 (1 GB/s) ALTERA FPGA 12 6 Gb/s PCIe gen.2 x8 (4 GB/s) XILINX FPGA Gb/s PCIe gen.3 ALTERA / XILINX Custom DDL protocol (same protocol but faster) Custom DDL 10 Gb/s 10 Gb/s PCIe over fibre RUN1 LS1LS1 RUN2 LS2LS2 RUN3

Run 2 - DDL1 & DDL2 Run 1 - DDL1 29/10/2012Filippo Costa, CERN 13 Speed of the link backward compatible Same DDL protocol No need to change the readout hardware if not needed. Transition DDL1 to DDL2 +

29/10/2012Filippo Costa, CERN 14 R&D DDL3 Evaluation process DDL custom protocol UDP Ethernet 10 Gb/s PCIe over fibre Other good options DDL3 Data transmission protocols are under evaluation, for the time being no final decision has been taken yet. Each protocol has different pros and cons, tests started already now, soon to come a reasonable decision.

29/10/2012Filippo Costa, CERN 15 UDP Ethernet 10Gb The evaluation for UDP Ethernet data transmission protocol has already started. Preliminary tests have been performed together with RD51 collaboration ( Hans MULLER, Alfonso TARAZONA MARTINEZ ). A test system has been prepared: 1 SRU board with a VIRTEX6, 10 Gb IP OPENCORE. 1 Machine (DELL server power Edge r 720) with 10 Gb/s port embedded in the motherboard. Continuous readout, no external trigger system, no timeout between 2 packets.

29/10/2012Filippo Costa, CERN 16 UDP Ethernet 10 Gb/s embedded in the motherboard XILINX VIRTEX 6 DDL optical fibre SFP+ 10 Gb/s

29/10/2012Filippo Costa, CERN 17 UDP Ethernet 100 kHz

29/10/2012Filippo Costa, CERN 18 UDP Ethernet (pro/con) Easy to implement in FPGA (IP cores available for 10 Gb/s). Fast and light protocol (no overhead for handshaking). Allow different configurations, point to point or network with routers. Reduce the hardware needed by the detector team to build test system, they can test their readout system using a standard Ethernet port of the PC. Is it rad tool ? We (ALICE DAQ+RD51) are evaluating different solutions. SmartFusion2: XGXS/XAUI Extension (to implement a 10 Gbps (XGMII) Ethernet PHY interface) Not reliable (but software checks can increase the reliability, backpressure algorithm implemented in UDP DATE). High CPU consuming, moving data from the Ethernet port to the memory, but different companies are already addressing the issue, PLDA and Solarflare. IP core license costs for 10 Gb/s can be expensive and not portable outside CERN, but OPENCORE can be a solution for that.

Det. readout electronics 29/10/2012Filippo Costa, CERN 19 TTCrq/rx All the detectors in ALICE receive the trigger messages through the TTCrx chip or the TTCrq board. Some of them will integrate the functionalities in the readout electronics. The code will be implemented in the FPGA, so there is no need anymore of the TTCrq board or TTCrx chip to be installed on each readout card, removing the problem of spare components. TTC VHDL FPGA

29/10/2012Filippo Costa, CERN 20 Radiation, how to beat it Use of radiation tolerant FPGA: ACTEL ( Using partial reconfiguration of FPGA reloading part of the firmware when: SEU is detected, periodically CSABA SOOS “ SEU effects in FPGA How to deal with them?” CEoQFjAD&url=http%3A%2F%2Findico.cern.ch%2FgetFile.py%2Faccess%3FcontribId%3D8%26r esId%3D2%26materialId%3Dslides%26confId%3D56796&ei=zeeIUOzUFtCQswaRyIFQ&usg=AFQj CNFjtJxDqKSGVzYv6pD-x-2b4yfD4w

Thanks!Thanks!