Kostas KORDAS INFN – Frascati XI Bruno Touschek spring school, Frascati,19 May 2006 Higgs → 2e+2  O (1/hr) Higgs → 2e+2  O (1/hr) ~25 min bias events ( >2k particles ) every 25 ns ~25 min bias events ( >2k particles ) every 25 ns ATLAS Trigger & Data Acquisition system: concept & architecture

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS2 LHC TeVatron Process  (pb) N/sN/year Total collected before start of LHC W  l 3  LEP / 10 7 FNAL Z  ee1.5  LEP t Tevatron b 5  Belle/BaBar ? Low lumi = 10 fb -1 /y ATLAS Trigger & DAQ: the need (1) Total cross section is at ~100 mb, While the very interesting physics is at ~1 nb to ~1 pb, i.e., a ratio of 1: 10 8 to 1:10 11

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS3 ATLAS Trigger & DAQ: the need (2) 40 MHz ~ 200 Hz ~ 300 MB/s Full info / event: ~ 1.6 MB/25ns ~60k TB/s p p Need high luminosity to get to observe the very interesting events Need on-line selection to write to disk mostly interesting events

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS4 Dataflow EB High Level Trigger L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) SFO L1 accept (100 kHz) 40 MHz EF EFP ~ sec EF accept (~0.2 kHz) ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 ATLAS Trigger & DAQ: architecture 40 MHz ~ 200 Hz ~ 300 MB/s 100 kHz ~ 3.5 kHz Full info / event: ~ 1.6 MB/25ns ~3+6 GB/s 160 GB/s

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS5 From the detector into the Level-1 Trigger Interactions every 25 ns: …in 25 ns particles travel 7.5 m Cable length ~100 meters: …in 25 ns signals travel 5 m Total Level-1 latency = 2.5  sec (TOF + cables + processing + distribution) For 2.5  sec, all signals must be stored in electronic pipelines Weight: 7000 t 44 m 22m Level 1 Trigger DAQ 2.5  s Calo MuTrCh Other detectors FE Pipelines 40 MHz

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS6 Upon LVL1 accept: buffer data & get RoIs ROS Level 1 Det. R/O Trigger DAQ 2.5  s Calo MuTrCh Other detectors Read-Out Systems RoI L1 accept (100 kHz) 40 MHz 160 GB/s ROD ROB ROIB Read-Out Drivers Region of Interest Builder Read-Out Buffers Read-Out Links (S-LINK) 100 kHz

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS7 LVL1 finds Regions of Interest for next levels 4 RoI  addresses In this example: 4 Regions of Interest: 2 muons, 2 electrons

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS8 Upon LVL1 accept: buffer data & get RoIs ROS Level 1 Det. R/O Trigger DAQ 2.5  s Calo MuTrCh Other detectors Read-Out Systems RoI L1 accept (100 kHz) 40 MHz 160 GB/s ROD ROB ROIB Read-Out Drivers Region of Interest Builder Read-Out Buffers Read-Out Links (S-LINK) 100 kHz On average, LVL1 finds ~2 Regions of Interest (in  ) per event Data in RoIs is a few % of the Level-1 throughput

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS9 LVL2: work with “interesting” ROSs/ROBs For each detector there is a simple correspondence  Region Of Interest ROB(s)  LVL2 Proccessing Units: for each RoI, the list of ROBs with the corresponding data from each detector is quickly identified L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L1 accept (100 kHz) 40 MHz 100 kHz 160 GB/s ~3 GB/s ROD ROB L2SVROIB Level 2 LVL2 Supervisor LVL2 Network LVL2 Processing Units Read-Out Buffers RoI-based Level-2 trigger: A much smaller ReadOut network … at the cost of a higher control traffic

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS10 Trigger DAQ Calo MuTrCh EB L2 ROS Level 1 Det. R/O 2.5  s ~10 ms Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz 160 GB/s ~3+6 GB/s ROD ROB SFI EBN Event Builder DFM L2SVROIB Level 2 Sub-Farm Input Dataflow Manager Event Building Network After LVL2: Build full events

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS11 LVL3: Event Filter deals with Full Event info Trigger DAQ EB L2 ROS Level 1 Det. R/O 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz 160 GB/s ~3+6 GB/s EF EFP ~ sec ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 Farm of Event Filter Processors Event Filter Network Full Event Sub-Farm Input ~ 200 Hz

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS12 EB L2 ROS Level 1 Det. R/O 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz 160 GB/s ~3+6 GB/s EF EFP ~ sec ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 Event Filter Processors Event Filter Network SFO EF accept (~0.2 kHz) ~ 200 Hz ~ 300 MB/s Sub-Farm Output From Event Filter to Local (TDAQ) storage

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS13 Dataflow EB High Level Trigger L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) SFO L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz ~ 200 Hz 160 GB/s ~ 300 MB/s ~3+6 GB/s EF EFP ~ sec EF accept (~0.2 kHz) ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 TDAQ, High Level Trigger & DataFlow

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS14 Dataflow EB High Level Trigger L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) SFO L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz ~ 200 Hz 160 GB/s ~ 300 MB/s ~3+6 GB/s EF EFP ~ sec EF accept (~0.2 kHz) ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 TDAQ, High Level Trigger & DataFlow High Level Trigger (HLT) Algorithms developed offline (with HLT in mind) HLT Infrastructure (TDAQ job): –“steer” the order of algorithm execution –Alternate steps of “feature extraction” & “hypothesis testing”)  fast rejection (min. CPU) –Reconstruction in Regions of Interest  min. processing time & network resources

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS15 Dataflow EB High Level Trigger L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors Read-Out Systems L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) SFO L1 accept (100 kHz) 40 MHz 100 kHz ~3.5 kHz ~ 200 Hz 160 GB/s ~ 300 MB/s ~3+6 GB/s EF EFP ~ sec EF accept (~0.2 kHz) ROD ROB SFI EBN Event Builder EFN DFM L2SVROIB Event Filter Level 2 TDAQ, High Level Trigger & DataFlow DataFlow Buffer & serve data to HLT Act according to HLT result, but otherwise HLT is a “black box” which gives answers Software framework based on C++ code and the STL

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS16 Dataflow EB High Level Trigger L2 ROS Level 1 Det. R/O Trigger DAQ 2.5  s ~10 ms Calo MuTrCh Other detectors L2P L2N RoI RoI data (~2%) RoI requests L2 accept (~3.5 kHz) SFO L1 accept (100 kHz) 40 MHz EF EFP ~ sec EF accept (~0.2 kHz) ROD ROB SFI EBN EFN DFM L2SVROIB 500 nodes 100 nodes 150 nodes 1600 nodes Infrastructure ControlCommunicationDatabases High Level Trigger & DataFlow: PCs (Linux)

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS17 TDAQ at the ATLAS site SDX1 USA15 UX15 ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links Gigabit Ethernet RoI Builder Regions Of Interest VME ~150 PCs Data of events accepted by first-level trigger Event data requests Delete commands Requested event data Event data ≤ 100 kHz, 1600 fragments of ~ 1 kByte each LVL2 Super- visor SDX1 CERN computer centre DataFlow Manager Event Filter (EF) pROS ~ 500~1600 stores LVL2 output dual-CPU nodes ~100~30 Network switches Event data pulled: partial ≤ 100 kHz, full ~ 3 kHz Event rate ~ 200 Hz Data storage Local Storage SubFarm Outputs (SFOs) LVL2 farm Network switches Event Builder SubFarm Inputs (SFIs) Second- level trigger “pre-series” system: ~10% of final TDAQ in place

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS18 Example of worries in such a system: CPU power At Technical Design Report we assumed: –100 kHz LVL1 accept rate –500 dual-CPU PCs for LVL2 –8 GHz per CPU at LVL2 So: –each L2PU does 100Hz –10ms average latency per event in each L2PU 8 GHz per CPU will not come –But, dual-core dual-CPU PCs show scaling! Preloaded ROS w/ muon events, run LVL2 Test with AMD dual-core, dual 1.8 GHz, 4 GB total We should reach necessary performance per PC at cost of higher memory needs & latency (shared memory model would be better here) 18

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS19 Last Sept: cosmics in the Tile hadronic calorimeter, brought via the pre-series (monitoring algorithms) Cosmics in ATLAS in the pit This July: Cosmic run with LAr EM + Tile Had Cal (+Muon detectors?)

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS20 ATLAS TDAQ: –3-level trigger hierarchy –Use Regions of Interest from previous level: small data movement –Feature extraction + hypothesis testing: fast rejection  min. CPU power Summary We are in the installation phase of system Cosmic run with Central Calorimeters (+muon system?) this summer TDAQ will be ready in time for LHC data taking Triggering at Hadron Colliders: –Need high luminosity to get rare events –Can not write all data to disk No sense otherwise: offline, we’ll be wasting our time looking for a needle in the hay!

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS21 Thank you

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS22 ROS units contain 12 R/O Buffers  150 units needed for ATLAS (~1600 ROBs) A ROS unit is implemented with a 3.4 GHz PC housing 4 custom PCI-x cards (ROBIN) ReadOut Systems: 150 PCs w/ special cards 12 ROS in place, more arriving Performance of final ROS (PC+ROBIN) is above requirements Note: we have also ability to access individual ROBs if wanted/needed “ Hottest” ROS from paper model 2. Measurements on real ROS H/W Low Lumi. operating region High Lumi. operating region LVL1 accept rate (kHZ) LVL2 accept rate (% of input) Not all ROSs are equal in rate of data requests ROD  ROS re-mapping can reduce requirements on busiest (hottest) ROS

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS23 So, we need: 5600 MB/s into EB system / (70MB/s in each EB node)  need ~80 SFIs for full ATLAS When SFI serves EF, throughput decreases by ~20%  actually need 80/0.80 = 100 SFIs Event Building needs Throughput requirements: 100 KHz LVL1 accept rate 3.5% LVL2 accept rate  3.5 KHz EB 1.6 MB event size  3.5 x 1.6 = 5600 MB/s total input Network limited (fast CPUs): Event building using 60-70% of Gbit network  ~70 MB/s into each Event Building node (SFI) 6 prototypes in place, evaluation of PCs now,  expect big Event Building needs from day 1: > 50 PCs till end of year

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS24 Data File LVL2 Ltcy Process Time RoI Coll Time RoI Coll Size # Req /Evt (ms) (bytes)  di-jet e Tests of LVL2 algorithms & RoI collection 2) Processing takes ~all latency: small RoI data collection time Note: Neither Trigger menu, nor data files representative mix of ATLAS (this is the aim for a late 2006 milestone) 3) Small RoI data request per event Electron sample is pre-selected 1) Majority of events rejected fast Di-jet,  & e simulated events preloaded on ROSs; RoI info on L2SV L2SV L2PU pROS Emulated ROS pROS 1 DFM 1 Plus: 1 Online Server 1 MySQL data base server

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS25 L2SV gets RoI info from RoIB Assigns a L2PU to work on event Load-balances its’ L2PU sub-farm Can scheme cope with LVL1 rate? Test with preloaded RoI info into RoIB, which triggers TDAQ chain, emulating LVL1 LVL2 system is able to sustain the LVL1 input rate: – 1 L2SV system for LVL1 rate ~ 35 kHz – 2 L2SV system for LVL1 rate ~ 70 kHz (50%-50% sharing) Scalability of LVL2 system Rate per L2SV stable within 1.5% ATLAS will have a handful of L2SVs  can easily manage 100 kHz LVL1 rate

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS26 Previous Event Filter I/O protocol limited rate for small event sizes (e.g., partially built)  changed in current TDAQ software release EF performance scales farm size Dummy algorithm: always accept, but with fixed delay Event size 1 MB Initially CPU limited, but eventually bandwidth limited Test e/  &  selection algorithms –HLT algorithms seeded by L2Result –pre-loaded (e &  ) simulated events on 1 SFI Emulator serving EF farm –Results here are for muons: Running muon algorithms:  scaling with EF farm size (still CPU limited with 9 nodes)

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS27 ATLAS Trigger & DAQ: philosophy 40 MHz ~100 kHz 2.5  s ~3 kHz ~ 10 ms ~ 1 s ~200 Hz Muon LVL1 CaloInner Pipeline Memories Read-Out Drivers RatesLatency RoI LVL2 Event builder cluster Local Storage: ~ 300 MB/s Read-Out Subsystems hosting Read-Out Buffers Event Filter farm EF ROB ROB ROB ROB ROB ROB ROD ROB ROB ROB ROB ROB ROB Hardware based (FPGA, ASIC) Calo/Muon (coarse granularity) Software (specialised algs) Uses LVL1 Regions of Interest All sub-dets, full granularity Emphasis on early rejection Offline algorithms Seeded by LVL2 result Work with full event Full calibration/alignment info High Level Trigger

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS28 Data Flow and Message Passing

XI Bruno Touschek school, Frascati, 19 May '06ATLAS TDAQ concept & architecture - Kostas KORDAS29 A Data Collection application example: the Event Builder Event Assembly Activity Input Activity Request Activity Event Handler Activity Assignment *Event Fragments *Event Event Sampler Activity ROS & pROS Event Fragments Data Requests Data Flow Manager Assignment Event Trigger (Event Filter) Event Monitoring SFI: Event Builder