1. Monday, June 2 2014TIPP 2014 @ Amsterdam1 A scalable gigabit data acquisition system for calorimeters for linear collider GASTALDI Franck Grant ANR-2010-0429-01 On behalf of the electronic & software team

2. Introduction: ILC detectors Method: Imaging calorimetry ~100 10 6 channels/detectors Issues: –Integration –Power consumption Ideas: –Detectors prototypes Power pulsing (1% duty cycle ~25µW/ch) allowed due to the beam structure (5 Hz spills) –Switched on during > ~1ms of ILC bunch train and data acquisition –Bias currents shut down between bunch trains –Data acquisition and control A single cable for everything Scalable architecture Reliable protocols & simplicity Monday, June 2 2014TIPP 2014 @ Amsterdam2

3. Introduction: ‘generic’ DAQ In most cases, detectors and associated readout systems are designed, tested and approved before DAQ effort is undertaken Our idea for this project is to design as a ‘generic’, scalable, and a self contained system, build around commercial components where possible. This DAQ is then configured towards multiple ‘use-cases’. ILC calorimetry might not be the only customer Remark : This work follows a R&D from Univ.College London, Manchester Univ and Cambridge Univ that continued at LLR-Ecole Polytechnique / IN2P3-CNRS Monday, June 2 2014TIPP 2014 @ Amsterdam3

4. Calorimeter DAQ: overview Monday, June 2 2014TIPP 2014 @ Amsterdam4 Machine clock DIFs Slabs DAQ2 PC DCC (optional) Clock & Control Digital (Config, Control, Data) Clock & Sync GDCC Network card GDCC ×7 ⋮ ×8 Optic GigE or copper Debug USB ×n layers : DCC (optional) ⋮ ×8 Slabs = detector unit : detector with integrated front-end electronics and sensors DIFs: Detector InterFace, servicing the detector unit GDCC: Giga-Data-Concentrator-Card: Concentrates data, fanin/fanout for clock and control data CCC: Clock & Control card: Fanout of clock and fast controls DCC: Data concentrator Card: optionnal extra level of data concentration 50 Mb/s 1 Gb/s

5. Calorimeter DAQ: Serial Link (cont’d) HDMI connectors between DIF-DCC-GDCC-CCC - Commercial standard for consumer electronics - High-bandwith connection at low cost 3 twisted pairs + 2 optional Reference clock (50 MHz), fan-out from CCC Data in (fast control, slow-control) Data out (slow control, data readout) Monday, June 2 2014TIPP 2014 @ Amsterdam5

6. DAQ: The DIF card The DIF concept is generic in firmware, running on detector specific hardware –Based on low cost FPGA –Compact (73mm x 50 mm) –Control up to 10K channels Functionalities are simple –VFE chip management (power pulsing, SC, DAQ) with a common interface –Local storage of SC data (Flash Ram) Monday, June 2 2014TIPP 2014 @ Amsterdam6 Architecture of the DIF FPGA

7. Format : VME 6U (chassis with only J1 connector used for power distribution) Format shared in 2 part (1/3 – 2/3) 1/3 is the mezzanine with the HDMI connections Reliability of mezzanine by a specific Samtec connector (SEAM and SEAF series: 160 pins) Until 28 differential signals and 19 single ended 2/3 is the GDCC “heart” with the main functionalities Based around a Xilinx Spartan XC6SLX75 + Marvell component USB is used to an extra access to the GDCC (debug for example) DAQ: The GDCC card Monday, June 2 2014TIPP 2014 @ Amsterdam7 7 x DIFs HDMI CCC HDMI RJ45 & sfp fiber VME USB Main part Mezzanine part

8. DAQ: the GDCC card (cont’d) Functionalities: Aggregate data from many DIF links and send it to the PC over Gigabit Ethernet link –The PHY layers is made by a specific component MARVELL88E1111 Signaling between the DIF and the GDCC is made by 5 differential LVDS pairs in HDMI cable Extract packet from the PC and execute the command sent (R/W register, DIF configuration packet, fast command) Encapsulate data from DIF in Ethernet frame and send them to the PC Monday, June 2 2014TIPP 2014 @ Amsterdam8 Gemac lithe Homemade (from xilinx reference design) Totally free Main Interface (based on several FSM And few Xilinx reference design) DIFs Links (Protocol fsm ser-des 8b/10b) ETHERNETETHERNET TODIFTODIF CCC interface MARVELL component FPGA mclk trig Single architecture of GDCC card

9. DAQ: The GDCC card (cont’d) Dst MACSrc MACEthernet Type GDCC_typeGDCC_modifierGDCC_pktIDGDCC_dataLengthGDCC_DataPADCRC32 6 Bytes 2 Bytes VariableMin size Eth4 Bytes Monday, June 2 2014 TIPP 2014 @ Amsterdam9 GDCC frame to the PC is based on standard Ethernet format 3 kinds of frame Fast-command with a special Ethernet type 0x809 (GDCC  DIF) Control data with a special Ethernet type 0x810 (GDCC  DIF) Read-out data with the Ethernet type 0x811 (DIF  GDCC) GDCC Header Content of the DIF structure Data Example of sending data from GDCC to DIF Data are sampled on rising-edge of clock CLK DIF SOFDIF EOF Register packet from DIF

10. DAQ: GDCC improvement Put in place an UDP interface –Simple and fast protocol –Easy to be implemented in hardware –Does not require big resources Monday, June 2 2014TIPP 2014 @ Amsterdam10 Ethernet frame structure with UDP Header Architecture of UDP bloc Currently under the first tests and after 3 days of sending a command to the DIF to read some registers (~ 3.106 times), there is no error.

11. DAQ: The DCC card (optional) VME format VME only used for the card power supply 1 HDMI connection for the GDCC Until 8 connections for the DIFs Identical data rate at the input and output (50 Mb/s) Advantage: This card can be connected or disconnected in DAQ chain without modification of behavior. Monday, June 2 2014TIPP 2014 @ Amsterdam11 Architecture of DCC

12. Calicoes Software The Acquisition chain Ecal dedicated software suite Based on the Pyrame framework (LLR) –Based on XML language –Allow to prototype rapidly a on-line system Multi-media distribution(files, sockets and shared memories) Online event-building Monday, June 2 2014TIPP 2014 @ Amsterdam12 Acquisition chain: software architecture

13. Calicoes Software The Control-Command Highly modular and distributed Control the Ecal electronics but also the peripheral devices (Power supply, pulse generator,…) Provides a high level state machine for final user Scripting language (Python) Good stability Monday, June 2 2014TIPP 2014 @ Amsterdam13 Global control-command architecture

14. The system: beam test This DAQ has been used on the SiW-Ecal technical prototype for two years It has been used successfully for 4 test beams at DESY Typical setup is : (~2.5K Channels) 10 layers of detection, 10 DIFs, 2 GDCC Monday, June 2 2014TIPP 2014 @ Amsterdam14 DAQ chassisSLAB structure S/N > 14 250 GBytes of data have been generated This system has been validated for 10 Hz of spill frequency (ILC requirement is 5 Hz) Exemple of event display 1e- (5GeV) 5 W plates between layers

15. Conclusion The aim who was to develop a DAQ system generic in nature, using commercial components where possible has been in most part attained The tests had shown the ability of the DAQ to take a lot of data (~250 GB) During the last beam test, 120000 configurations have been injected in the system in three weeks. It remained stable during all this time. Currently, we improve our system with the implantation of a UDP block on GDCC. Next step Connect the DAQ to a real calorimeter system –16 ASICs per ASU (under test today), will be up to 160 ASICs per layer Perspectives for ECAL(ILC) With this actual configuration and for 100M channels ECAL for example the setup will be: 12500 DCC, 2000 GDCC and 200 PC For reducing the number of card, the main work must be done on front end modules for easiness of integration Monday, June 2 2014TIPP 2014 @ Amsterdam15

16. Monday, June 2 2014TIPP 2014 @ Amsterdam16

17. Back up Monday, June 2 2014TIPP 2014 @ Amsterdam17

18. Time line Monday, June 2 2014TIPP 2014 @ Amsterdam18 Physics prototype Technological prototype ILD ? Proof of concept Linearity Resolution Sensors Very front-end Feasibility of design options Compactness Granularity Front-end Power pulsing Long SLAB Construction Integration Environement Services Industrialization Tooling Project org. S/N ~ 7.5 ~24 X0, 20 cm thick ~2500 m 2 active detectors ~100M readout channels 2004-2008 30 layers 4000 channels 1500 channels/dm 3 4000 channels/dm 3 4000..10000 channels/dm 3 S/N ~ 15

19. Slab details Monday, June 2 2014TIPP 2014 @ Amsterdam19 ASUS with 16 Asics (180 x 180 mm) 1 Si Wafer with 256 pixels of 5X5 mm2 and thickness of 325 µm Battery charger application AVX BestCap BZ01 After regulator 360 mm 190 mm 180 mm 70 mm Slab overview

20. DIF card Slow control and read-out Sent from the DAQ/control PC as a raw Ethernet frame Passed to/from the DIF via GDCC/DCC with the following structure (protocol) Monday, June 2 2014TIPP 2014 @ Amsterdam20 Internally decoded frame (test pin) DIF input:Standard packet DIF output: Here: read out of 13x16b status registers (Reshaped into GDCC frame) IDLE SOFdataheaderEOF Exemple of a decoding frame at the DIF level Exemple of a fast decodind command at the DIF level

21. GDCC: some plots Monday, June 2 2014TIPP 2014 @ Amsterdam21 Example of data readout Example of Result of eyes diagram and jitter on data readout Example of readout packet spied by wireshark Trigger = start spill DIF SOF RJ ~23 ps DJ ~166ps eye width ~19.75ns 0xfcff = start spill 0xfdff = start chip data Data from DIF

22. CCC card Supplied by University of Cambridge in 2009 Synchronize all sub-systems upon pre-spill warning Until 8 HDMI connection Distribute asynchronous fast trigger and/or busy signals Capable to run stand-alone for distribute clock (50 MHz) and spill from an external trigger Monday, June 2 2014TIPP 2014 @ Amsterdam22