Monday, June TIPP Amsterdam1 A scalable gigabit data acquisition system for calorimeters for linear collider GASTALDI Franck Grant ANR On behalf of the electronic & software team

Introduction: ILC detectors Method: Imaging calorimetry ~ channels/detectors Issues: –Integration –Power consumption Ideas: –Detectors prototypes Power pulsing (1% duty cycle ~25µW/ch) allowed due to the beam structure (5 Hz spills) –Switched on during > ~1ms of ILC bunch train and data acquisition –Bias currents shut down between bunch trains –Data acquisition and control A single cable for everything Scalable architecture Reliable protocols & simplicity Monday, June TIPP Amsterdam2

Introduction: ‘generic’ DAQ In most cases, detectors and associated readout systems are designed, tested and approved before DAQ effort is undertaken Our idea for this project is to design as a ‘generic’, scalable, and a self contained system, build around commercial components where possible. This DAQ is then configured towards multiple ‘use-cases’. ILC calorimetry might not be the only customer Remark : This work follows a R&D from Univ.College London, Manchester Univ and Cambridge Univ that continued at LLR-Ecole Polytechnique / IN2P3-CNRS Monday, June TIPP Amsterdam3

Calorimeter DAQ: overview Monday, June TIPP Amsterdam4 Machine clock DIFs Slabs DAQ2 PC DCC (optional) Clock & Control Digital (Config, Control, Data) Clock & Sync GDCC Network card GDCC ×7 ⋮ ×8 Optic GigE or copper Debug USB ×n layers : DCC (optional) ⋮ ×8 Slabs = detector unit : detector with integrated front-end electronics and sensors DIFs: Detector InterFace, servicing the detector unit GDCC: Giga-Data-Concentrator-Card: Concentrates data, fanin/fanout for clock and control data CCC: Clock & Control card: Fanout of clock and fast controls DCC: Data concentrator Card: optionnal extra level of data concentration 50 Mb/s 1 Gb/s

Calorimeter DAQ: Serial Link (cont’d) HDMI connectors between DIF-DCC-GDCC-CCC - Commercial standard for consumer electronics - High-bandwith connection at low cost 3 twisted pairs + 2 optional Reference clock (50 MHz), fan-out from CCC Data in (fast control, slow-control) Data out (slow control, data readout) Monday, June TIPP Amsterdam5

DAQ: The DIF card The DIF concept is generic in firmware, running on detector specific hardware –Based on low cost FPGA –Compact (73mm x 50 mm) –Control up to 10K channels Functionalities are simple –VFE chip management (power pulsing, SC, DAQ) with a common interface –Local storage of SC data (Flash Ram) Monday, June TIPP Amsterdam6 Architecture of the DIF FPGA

Format : VME 6U (chassis with only J1 connector used for power distribution) Format shared in 2 part (1/3 – 2/3) 1/3 is the mezzanine with the HDMI connections Reliability of mezzanine by a specific Samtec connector (SEAM and SEAF series: 160 pins) Until 28 differential signals and 19 single ended 2/3 is the GDCC “heart” with the main functionalities Based around a Xilinx Spartan XC6SLX75 + Marvell component USB is used to an extra access to the GDCC (debug for example) DAQ: The GDCC card Monday, June TIPP Amsterdam7 7 x DIFs HDMI CCC HDMI RJ45 & sfp fiber VME USB Main part Mezzanine part

DAQ: the GDCC card (cont’d) Functionalities: Aggregate data from many DIF links and send it to the PC over Gigabit Ethernet link –The PHY layers is made by a specific component MARVELL88E1111 Signaling between the DIF and the GDCC is made by 5 differential LVDS pairs in HDMI cable Extract packet from the PC and execute the command sent (R/W register, DIF configuration packet, fast command) Encapsulate data from DIF in Ethernet frame and send them to the PC Monday, June TIPP Amsterdam8 Gemac lithe Homemade (from xilinx reference design) Totally free Main Interface (based on several FSM And few Xilinx reference design) DIFs Links (Protocol fsm ser-des 8b/10b) ETHERNETETHERNET TODIFTODIF CCC interface MARVELL component FPGA mclk trig Single architecture of GDCC card

DAQ: The GDCC card (cont’d) Dst MACSrc MACEthernet Type GDCC_typeGDCC_modifierGDCC_pktIDGDCC_dataLengthGDCC_DataPADCRC32 6 Bytes 2 Bytes VariableMin size Eth4 Bytes Monday, June TIPP Amsterdam9 GDCC frame to the PC is based on standard Ethernet format 3 kinds of frame Fast-command with a special Ethernet type 0x809 (GDCC  DIF) Control data with a special Ethernet type 0x810 (GDCC  DIF) Read-out data with the Ethernet type 0x811 (DIF  GDCC) GDCC Header Content of the DIF structure Data Example of sending data from GDCC to DIF Data are sampled on rising-edge of clock CLK DIF SOFDIF EOF Register packet from DIF

DAQ: GDCC improvement Put in place an UDP interface –Simple and fast protocol –Easy to be implemented in hardware –Does not require big resources Monday, June TIPP Amsterdam10 Ethernet frame structure with UDP Header Architecture of UDP bloc Currently under the first tests and after 3 days of sending a command to the DIF to read some registers (~ times), there is no error.

DAQ: The DCC card (optional) VME format VME only used for the card power supply 1 HDMI connection for the GDCC Until 8 connections for the DIFs Identical data rate at the input and output (50 Mb/s) Advantage: This card can be connected or disconnected in DAQ chain without modification of behavior. Monday, June TIPP Amsterdam11 Architecture of DCC

Calicoes Software The Acquisition chain Ecal dedicated software suite Based on the Pyrame framework (LLR) –Based on XML language –Allow to prototype rapidly a on-line system Multi-media distribution(files, sockets and shared memories) Online event-building Monday, June TIPP Amsterdam12 Acquisition chain: software architecture

Calicoes Software The Control-Command Highly modular and distributed Control the Ecal electronics but also the peripheral devices (Power supply, pulse generator,…) Provides a high level state machine for final user Scripting language (Python) Good stability Monday, June TIPP Amsterdam13 Global control-command architecture

The system: beam test This DAQ has been used on the SiW-Ecal technical prototype for two years It has been used successfully for 4 test beams at DESY Typical setup is : (~2.5K Channels) 10 layers of detection, 10 DIFs, 2 GDCC Monday, June TIPP Amsterdam14 DAQ chassisSLAB structure S/N > GBytes of data have been generated This system has been validated for 10 Hz of spill frequency (ILC requirement is 5 Hz) Exemple of event display 1e- (5GeV) 5 W plates between layers

Conclusion The aim who was to develop a DAQ system generic in nature, using commercial components where possible has been in most part attained The tests had shown the ability of the DAQ to take a lot of data (~250 GB) During the last beam test, configurations have been injected in the system in three weeks. It remained stable during all this time. Currently, we improve our system with the implantation of a UDP block on GDCC. Next step Connect the DAQ to a real calorimeter system –16 ASICs per ASU (under test today), will be up to 160 ASICs per layer Perspectives for ECAL(ILC) With this actual configuration and for 100M channels ECAL for example the setup will be: DCC, 2000 GDCC and 200 PC For reducing the number of card, the main work must be done on front end modules for easiness of integration Monday, June TIPP Amsterdam15

Monday, June TIPP Amsterdam16

Back up Monday, June TIPP Amsterdam17

Time line Monday, June TIPP Amsterdam18 Physics prototype Technological prototype ILD ? Proof of concept Linearity Resolution Sensors Very front-end Feasibility of design options Compactness Granularity Front-end Power pulsing Long SLAB Construction Integration Environement Services Industrialization Tooling Project org. S/N ~ 7.5 ~24 X0, 20 cm thick ~2500 m 2 active detectors ~100M readout channels layers 4000 channels 1500 channels/dm channels/dm channels/dm 3 S/N ~ 15

Slab details Monday, June TIPP Amsterdam19 ASUS with 16 Asics (180 x 180 mm) 1 Si Wafer with 256 pixels of 5X5 mm2 and thickness of 325 µm Battery charger application AVX BestCap BZ01 After regulator 360 mm 190 mm 180 mm 70 mm Slab overview

DIF card Slow control and read-out Sent from the DAQ/control PC as a raw Ethernet frame Passed to/from the DIF via GDCC/DCC with the following structure (protocol) Monday, June TIPP Amsterdam20 Internally decoded frame (test pin) DIF input:Standard packet DIF output: Here: read out of 13x16b status registers (Reshaped into GDCC frame) IDLE SOFdataheaderEOF Exemple of a decoding frame at the DIF level Exemple of a fast decodind command at the DIF level

GDCC: some plots Monday, June TIPP Amsterdam21 Example of data readout Example of Result of eyes diagram and jitter on data readout Example of readout packet spied by wireshark Trigger = start spill DIF SOF RJ ~23 ps DJ ~166ps eye width ~19.75ns 0xfcff = start spill 0xfdff = start chip data Data from DIF

CCC card Supplied by University of Cambridge in 2009 Synchronize all sub-systems upon pre-spill warning Until 8 HDMI connection Distribute asynchronous fast trigger and/or busy signals Capable to run stand-alone for distribute clock (50 MHz) and spill from an external trigger Monday, June TIPP Amsterdam22