KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 A Scalable 1 MHz Trigger Farm Prototype with Event-Coherent DMA Input V. Lindenstruth, D. Atanasov, I. Kisel, A. Walsch ( KIP, Uni-Heidelberg, Germany ) H. Muller, D. Altmann, A. Guirao, F. Vinci dos Santos ( CERN, Geneva, Switzerland ) LHCb Level-1 Trigger Trigger Concept Trigger Prototype Trigger Simulation Trigger Algorithm

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 e h   Level-1 Trigger for LHCb 1.Find VELO 2D tracks and reconstruct 3D primary vertex 2.Reconstruct high-impact parameter tracks in 3D 3.Extrapolate to TT through small magnetic field  PT 4.Match tracks to L0 muon objects  PT and PID 5.Select B–events using impact parameter and PT information 6.Use T1—3 data to improve further selection (5—10% of events) VELO TT (T1—3)

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Trigger Concept Reduce rate from 1 MHz to 40 kHz  Send data to RU (8 kB/evt  8 GB/s): VELO + TT + L0DU T1—3 (~5—10% on CPU demand)  Traffic shaping --> use Scheduler !  NIC with Remote Direct Memory Access !  Prototype : 2D torus with 32 dual nodes at 1.24 MHz  Trigger farm : 3D torus with up to 1200 CPUs TagNet Data PC farm (2D torus) Scheduler - RU - CN X->Y routing x y

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Scheduler - Basic Block Diagram The supervisor of the system. Handle a coherent data transfer between RUs and CNs. Feed the TagNet with tags for synchronous data transfer in the RUs. Scheduler Core Tag Output Stage TagNet Feed TagNet Feedback User Control Event entries Free CN IDs entries List of free computing nodes Control Register Status Register Tag Input Stage

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Tag InTag Out Data In Subevent Buffer DMA PCI Bus NIC Interface NIC Out NIC In C/M ? Command Execution Message Execution MUX Tag Buffer Readout Unit (RU) Command TagMessage Tag

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Trigger Farm Prototype in Heidelberg >1 MHz 64 CPUs 2D torus 6 Gbit/s NIC 1 year 480 MB/s p-p 450 MB/s x-y

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Automatic setup of the compute farm Configure and control processes on every CN GUI of Prototype

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Scheduler TagNet 3D Core1D Cover Data TagNet – schedule and send small data packets Core network – distribute data to the target compute nodes Cover network – increase number of compute nodes X->Y->Z routing path - RU - CN 3D Torus Topology 4x4x(1+2+1) x y z

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Ptolemy II Simulation of the Trigger 3D Torus (6x6x8) 275 CNs

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Simulation of the Trigger --- Results 128 B/RU 2.1 MHz measured ! 1200 CPU +5% T1--3 Compact Scalable Fast Fast Response VELO Z VELO events VELOT1-3 New VELO Scheduler

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Tracking Efficiency and PV Resolution Track subsets Reference B long Reference prim. long Reference B Reference primary Reference set All set Extra set Clone Ghost D % 3D Z core  46  m X/Y core  17  m

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Trigger Performance  time (ms)  Events 17 ms 15  s Mean: 15  s Max: ~130  s CPU 4.8 ms 1) Tracking efficiency 97—99% 2) PV resolution 46  m 3) Timing 4.8 ms Expect a factor 7—8 in CPU power in 2007 (PASTA report) => we are already within 1 ms ! Cellular Automaton algorithm FPGA co-processor at 50 MHz 8 processing units running in parallel => 15  s ! FPGA co-processor  Events  time (  s)

KIP Ivan Kisel, Uni-Heidelberg, RT May 2003 Summary: Demonstrated Architecture with 3D torus and TagNet Prototype of 64 CPUs has shown stable work at > 1 MHz The Simulation is based on the prototype measurements The Algorithm has high performance on tracks and vertices The Cost is 1300 kCHF (500 CPU) / 2300 kCHF (1000 CPU) The Team --- Heidelberg, CERN and Dubna