Wing-LDA Timing & performance Beam Interface (BIF) DAQ Hardware: Year 2015 Wing-LDA Timing & performance Beam Interface (BIF) Jiri Kvasnicka AHCAL meeting in Hamburg DESY, 10.12.2015

Introduction: DAQ chain Chain of devices with a tree hierarchy: Clock and Control Card (CCC) Provides master clock Starts and stops the acquisition according to the spill level and readiness of all ASICs Distributes trigger validation Link Data Aggregator (LDA) Packet collecting Packet processing (decoding, merging) Send the packets over TCP to DAQ PC Detector InterFace (DIF) Controls the ASICs (voltages, acq. state) Collects data from all ASICs on HBU Sends the data to LDA ASIC (SPIROC 2b, Omega group) Reads out 36 SiPMs Has 16 memory channels for ADC and TDC Trigger Spill CCC … Up to 8 LDAs LDA PC ... Up to 96 DIFs ... Up to 96 DIFs ... Up to 96 DIFs DIF Data … Up to 72 SPIROCs ASIC 36 SiPMs

Wing-LDA Designed @Uni Mainz MicroHDMI connectors (96), AHCAL Geometry Uses commercial Mars ZX3 module (Enclustra) Started with first, pre-production version Extra 4 slave FPGAs (Xilinx Kintex-7) Robust dual 400MBit/s links Mars↔Kintex, 8b10b + CRC, automatic packet retransmission Commissioned April 2015 (with 1 FPGA) Fixes: Gigabit ethernet operation 1 bug (rare corruption of small packets) 1 bug in Linux part (caused rare stream de-synchronisation under specific conditions) Heat-sinks added HW packet generator (80 MByte/s, for benchmarking) MicroHDMI full HDMI

Wing-LDA mounted on the steel absorber, CERN SPS 2015 © Oskar CCC Wing-LDA

Beamtest data rates: PS 2014, SPS 2015 TB Dec 2014 (CERN PS) TB Aug 2015 (CERN SPS) Oct 2014 (USB) Dec 2014 (HDMI1) Aug 2015 (HDMI2) Future (3) Sustained LED calib (16 mem. cells) 1-2 3-4 6.2 12 Sustained TB (no spill) RO/s 2-3 ~6-7 (est.) ~17 35 Readouts per 400 ms spill 2 7 (Est) 14 Theoretical events per 400ms spill 30 45-60 105 210 Theoretical sustained phys.events/s ~30 ~100 ~255 ~525 1 Explicit RO command 2 Automatic data send 3 Foreseen DIF speedup, we are there already since Oct 2015!

DAQ Timing (Testbeam mode, Nov 2015) Conversion phase shortened (checks now the signal from SPIROC rather than constant time) Readout speed increased by factor 2 (0.83 Mbit/s → 1.666 Mbit/s) “Easy” possibility to go to 2.5 MHz (without change of the firmware architecture) Data transfer 10 Mbits/s Easy possibility to go to 40 Mbits/s => 30-45 Roudout cycles/s in DESY Beam test (single HBU, Spiroc 2D)

BIF concept: Timestamping external signals Firmware of the AIDA mini-TLU modified Receives AHCAL clock Knows AHCAL fast commands from HDMI Records timestamps from 4 inputs (lemo) and start&stop of acquisition Separate DAQ, acquisition is gated (=records only when AHCAL active) Implemented in the “slave mode” - acts like another DIF Different modes are in discussion for the final implementation DESY TB: Nov 2015 Trigger Spill CCC … Up to 8 LDAs LDA PC beam triggers BIF DIF

BIF performance 0.78125 ns intrinsic timestamp HW resolution DESY beam structure clearly visible Time between 2 events: 1.8 ns jitter => single timestamp jitter: 1.3 ns Includes jitter from scintillator, PMT, discriminator and coincidence units <100k recored triggers/s sustainable AHCAL: in the order of 500 evt/s Raw data: 16 bytes per event Generated ~400 MB for all 4 days Running in debugging mode Covers only part (~60%) of SPIROC data Cannot handle 2 consecutive events within ~25ns at the moment.

BIF data correlation: Ultimate TDC insight tool First look at the data started Addressed some issues with the data quality Spiroc 2D, Chip #129, ch4, memory cell 15

Summary 2 stable LDA form factors: Wing-LDA and mini-LDA Good experience from summer beam test DIF speedup ~2x improved recently (tested only with Spiroc 2d) BIF: We have external time reference for beam events! First prototype in slave mode (works as another DIF) Jitter better than 1.3ns Final implementation under discussion Spiroc 2d TDC data insight thanks to BIF timestamp correlation We see some issues we would not be able to see otherwise => better feedback to SPIROC developpers Plans 2016: Final BIF (might be merged with Master CCC project) New DIF firmware DIF modification towards the power-pulsing and ILC mode in beam test LDA Maintanance: Update LDA design to newer compiler (Vivado 2015) Improve speed to full 1GBit Ethernet utilization

Backup

BIF: position in DAQ chain Chain of devices with a tree hierarchy: Clock and Control Card (CCC) Provides master clock Starts and stops the acquisition according to the spill level and readiness of all ASICs Distributes trigger validation Link Data Aggregator (LDA) Packet collecting Packet processing (decoding, merging) Send the packets over TCP to DAQ PC Detector InterFace (DIF) Controls the ASICs (voltages, acq. state) Collects data from all ASICs on HBU Sends the data to LDA ASIC (SPIROC 2b, Omega group) Reads out 36 SiPMs Has 16 memory channels for ADC and TDC Beam InterFace (BIF) Record trigger timing with <1 ns precision Based on mini-TLU hardware Trigger Spill CCC … Up to 8 LDAs LDA PC ... Up to 96 DIFs BIF DIF Data … Up to 72 SPIROCs ASIC 36 SiPMs

Common running with SiW ECAL @ CERN PS 2014 Eudaq (Labview) AHCAL + SiW ECAL layer SiW ECAL System differences 50 MHz clock (vs. 40 MHz) No busy / Memory full (re-enabled in the Si ECAL DIF by Remi) Only Spill input 2.5 MHz BXID (vs. 250 kHz) No TDC EUDAQ as common DAQ SW Common properly timed events found! AHCAL SW SiEcal SW Scint. CCC Spill, clk CCC LDA xLDA xLDA 1 DIF 8 DIFs 7 DIF SiECAL Sci CAL

Simplified LDA architecture mini-LDA Wing-LDA

Correlation in time Depends on the configuration and cabling Some interesting corner effects DIF sometimes assigns previous BxID Gap around the BxID change Where are the real limits of BxID? Start acq Relative timestamp 13448 BIF 1 2 10.506 us 1 2 DIF DIF starts recording time

Correlation (chip 129, ch 4), TB mode with validation Green: BIF_bxid == DIF_bxid Blue: BIF_bxid == (DIF_bxid – 1) Red: BIF_bxid == (DIF_bxid + 1)

Spiroc 2D on-time Without particle, the dead time is low! No dummy trigger, no T0 cable connected! It gets worse with T0 dead time acq Conversion +readout

BIF connection HDMI cable gnd 3 pairs in RJ45 Clock (diff)

Considered BIF implementations beam triggers Master CCC beam triggers mTLU Spill Spill ? CCC ScCCC IP ? … Up to 8 LDAs LDA PC Up to 2 LDAs LDA PC ... Up to 96 DIFs ... Up to 96 DIFs DIF DIF

TDC, chip 131, ch 7, memcel 14, zoomed

T0 TDC, chip 131, ch 7, memcel 5, zoomed Odd BxID noisy

T0: chip 129, chan 29, memcell 1