1. Wing-LDA Timing & performance Beam Interface (BIF)
DAQ Hardware: Year 2015
Wing-LDA
Timing & performance
Beam Interface (BIF)
Jiri Kvasnicka
AHCAL meeting in Hamburg
DESY,

2. Introduction: DAQ chain
Chain of devices with a tree hierarchy:
Clock and Control Card (CCC)
Provides master clock
Starts and stops the acquisition according to the spill level and readiness of all ASICs
Distributes trigger validation
Link Data Aggregator (LDA)
Packet collecting
Packet processing (decoding, merging)
Send the packets over TCP to DAQ PC
Detector InterFace (DIF)
Controls the ASICs (voltages, acq. state)
Collects data from all ASICs on HBU
Sends the data to LDA
ASIC (SPIROC 2b, Omega group)
Reads out 36 SiPMs
Has 16 memory channels for ADC and TDC
Trigger
Spill
CCC …
Up to 8 LDAs
LDA PC
...
Up to 96 DIFs
...
Up to 96 DIFs
...
Up to 96 DIFs
DIF
Data …
Up to 72 SPIROCs
ASIC 36
SiPMs

3. Wing-LDA Designed @Uni Mainz MicroHDMI connectors (96), AHCAL Geometry
Uses commercial Mars ZX3 module (Enclustra)
Started with first, pre-production version
Extra 4 slave FPGAs (Xilinx Kintex-7)
Robust dual 400MBit/s links Mars↔Kintex,
8b10b + CRC, automatic packet retransmission
Commissioned April 2015 (with 1 FPGA)
Fixes:
Gigabit ethernet operation
1 bug (rare corruption of small packets)
1 bug in Linux part (caused rare stream de-synchronisation under specific conditions)
Heat-sinks added
HW packet generator (80 MByte/s, for benchmarking)
MicroHDMI full HDMI

4. Wing-LDA mounted on the steel absorber, CERN SPS 2015
© Oskar
CCC
Wing-LDA

5. Beamtest data rates: PS 2014, SPS 2015
TB Dec 2014 (CERN PS)
TB Aug 2015 (CERN SPS)
Oct 2014 (USB)
Dec 2014 (HDMI1)
Aug 2015 (HDMI2)
Future
(3)
Sustained LED calib (16 mem. cells)
1-2
3-4
6.2 12
Sustained TB (no spill) RO/s
2-3
~6-7 (est.)
~17 35
Readouts per 400 ms spill 2
7 (Est) 14
Theoretical events per 400ms spill 30
45-60
105
210
Theoretical sustained phys.events/s
~30
~100
~255
~525
1 Explicit RO command
2 Automatic data send
3 Foreseen DIF speedup, we are there already since Oct 2015!

6. DAQ Timing (Testbeam mode, Nov 2015)
Conversion phase shortened (checks now the signal from SPIROC rather than constant time)
Readout speed increased by factor 2 (0.83 Mbit/s → Mbit/s)
“Easy” possibility to go to 2.5 MHz (without change of the firmware architecture)
Data transfer 10 Mbits/s
Easy possibility to go to 40 Mbits/s
=> Roudout cycles/s in DESY Beam test (single HBU, Spiroc 2D)

7. BIF concept: Timestamping external signals
Firmware of the AIDA mini-TLU modified
Receives AHCAL clock
Knows AHCAL fast commands from HDMI
Records timestamps from 4 inputs (lemo) and start&stop of acquisition
Separate DAQ, acquisition is gated (=records only when AHCAL active)
Implemented in the “slave mode” - acts like another DIF
Different modes are in discussion for the final implementation
DESY TB: Nov 2015
Trigger
Spill
CCC …
Up to 8 LDAs
LDA PC
beam
triggers
BIF
DIF

8. BIF performance 0.78125 ns intrinsic timestamp HW resolution
DESY beam structure clearly visible
Time between 2 events: 1.8 ns jitter
=> single timestamp jitter: 1.3 ns
Includes jitter from scintillator, PMT, discriminator and coincidence units
<100k recored triggers/s sustainable
AHCAL: in the order of 500 evt/s
Raw data: 16 bytes per event
Generated ~400 MB for all 4 days
Running in debugging mode
Covers only part (~60%) of SPIROC data
Cannot handle 2 consecutive events within ~25ns at the moment.

9. BIF data correlation: Ultimate TDC insight tool
First look at the data started
Addressed some issues with the data quality
Spiroc 2D, Chip #129, ch4, memory cell 15

10. Summary 2 stable LDA form factors: Wing-LDA and mini-LDA
Good experience from summer beam test
DIF speedup
~2x improved recently (tested only with Spiroc 2d)
BIF: We have external time reference for beam events!
First prototype in slave mode (works as another DIF)
Jitter better than 1.3ns
Final implementation under discussion
Spiroc 2d TDC data insight thanks to BIF timestamp correlation
We see some issues we would not be able to see otherwise => better feedback to SPIROC developpers
Plans 2016:
Final BIF (might be merged with Master CCC project)
New DIF firmware
DIF modification towards the power-pulsing and ILC mode in beam test
LDA Maintanance:
Update LDA design to newer compiler (Vivado 2015)
Improve speed to full 1GBit Ethernet utilization

11. Backup

12. BIF: position in DAQ chain
Chain of devices with a tree hierarchy:
Clock and Control Card (CCC)
Provides master clock
Starts and stops the acquisition according to the spill level and readiness of all ASICs
Distributes trigger validation
Link Data Aggregator (LDA)
Packet collecting
Packet processing (decoding, merging)
Send the packets over TCP to DAQ PC
Detector InterFace (DIF)
Controls the ASICs (voltages, acq. state)
Collects data from all ASICs on HBU
Sends the data to LDA
ASIC (SPIROC 2b, Omega group)
Reads out 36 SiPMs
Has 16 memory channels for ADC and TDC
Beam InterFace (BIF)
Record trigger timing with <1 ns precision
Based on mini-TLU hardware
Trigger
Spill
CCC …
Up to 8 LDAs
LDA PC
...
Up to 96 DIFs
BIF
DIF
Data …
Up to 72 SPIROCs
ASIC 36
SiPMs

13. Common running with SiW ECAL @ CERN PS 2014
Eudaq
(Labview)
AHCAL + SiW ECAL layer
SiW ECAL System differences
50 MHz clock (vs. 40 MHz)
No busy / Memory full (re-enabled in the Si ECAL DIF by Remi)
Only Spill input
2.5 MHz BXID (vs. 250 kHz)
No TDC
EUDAQ as common DAQ SW
Common properly timed events found!
AHCAL SW
SiEcal SW
Scint. CCC
Spill,
clk
CCC
LDA
xLDA
xLDA
1 DIF
8 DIFs
7 DIF
SiECAL
Sci CAL

14. Simplified LDA architecture
mini-LDA
Wing-LDA

15. Correlation in time Depends on the configuration and cabling
Some interesting corner effects
DIF sometimes assigns previous BxID
Gap around the BxID change
Where are the real limits of BxID?
Start acq
Relative timestamp 13448
BIF 1 2 us 1 2
DIF
DIF starts recording
time

16. Correlation (chip 129, ch 4), TB mode with validation
Green: BIF_bxid == DIF_bxid
Blue: BIF_bxid == (DIF_bxid – 1)
Red: BIF_bxid == (DIF_bxid + 1)

17. Spiroc 2D on-time Without particle, the dead time is low!
No dummy trigger, no T0 cable connected! It gets worse with T0
dead time
acq
Conversion +readout

18. BIF connection
HDMI cable
gnd
3 pairs
in RJ45
Clock (diff)

19. Considered BIF implementations
beam
triggers
Master
CCC
beam
triggers
mTLU
Spill
Spill ?
CCC
ScCCC IP ? …
Up to 8 LDAs
LDA PC
Up to 2 LDAs
LDA PC
...
Up to 96 DIFs
...
Up to 96 DIFs
DIF
DIF

20. TDC, chip 131, ch 7, memcel 14, zoomed

21. T0 TDC, chip 131, ch 7, memcel 5, zoomed
Odd BxID noisy

22. T0: chip 129, chan 29, memcell 1