The Back-End Electronics of the Time Projection Chambers in the T2K Experiment D. Calvet 1, I. Mandjavidze 1, B. Andrieu 2, O. Le Dortz 2, D. Terront 2, A. Vallereau 2, C. Gutjahr 3, K. Mizouchi 3, C. Ohlmann 3, F. Sanchez 4 1 CEA/DSM/IRFU/SEDI, Saclay, France 2 CNRS/IN2P3/LPNHE, Paris, France, 3 TRIUMF, Vancouver, Canada 4 IFAE, Barcelona, Spain

RT2010 – Lisboa May Tokai to Kamioka (T2K) experiment Main Physics Goal: neutrino oscillation μ disappearance for improved accuracy on  23 e appearance to improve sensitivity to  kT water 2 Experiment now taking data

RT2010 – Lisboa May 2010 Summary of features Amplification: MicroMegas modules segmented in 7 x 10 mm pads Over pads to instrument ~9 m 2 sensitive area T2K Time-Projection Chambers 1 of 3 TPCs shown 1 m 2 m 3

RT2010 – Lisboa May T2K TPC-FGD readout: AFTER chip 4 pole zero cancellation Cs 16 values + - Cp Rp Vdc CSA in Cl Cdet Cf Rf 4 Rf*Cf (100us) ADC in VdcinG-2 Cg *Cg2 Gain-2 x511 Cm Memory Cell ckwckr r1 vicm + - r1 r2 Buffer Cs Sallen&Key Filter 16 values Cg Cs 2*Cg + - Q max.MIPNoise < 120fC e750 e 240fC e1500 e 360fC e2250 e 600fC e3750 e Tp(5%-100%) ns ckw: 1MHz-50MHz ckr: 20MHz Noise < 2mV ADCin 2V/10MIPs 72 Analog Channels72 Analog Channels Switched Capacitor Array: 511 cellsSwitched Capacitor Array: 511 cells 4 Gains; 16 Peaking Time values (100ns to 2µs)4 Gains; 16 Peaking Time values (100ns to 2µs) Input Current Polarity: positive or negativeInput Current Polarity: positive or negative Readout: 76*511 cells at 20MHz [external 12bits ADC]Readout: 76*511 cells at 20MHz [external 12bits ADC] P. Baron et al.

RT2010 – Lisboa May TPC Read-Out Diagram 72 x 2 Gbps fibres detector pads 72 Front-End Electronic Modules 72 Front-End Electronic Modules 18 Quad-optical link Data Concentrator Cards 18 Quad-optical link Data Concentrator Cards 24-ports Gbit Ethernet Switch TPC DAQ PC On-line database Global event builder Run control Mass storage Nd 280 network Slave Clock Module Slave Clock Module 18 Master Clock Module Master Clock Module On-detector in magnet Off-detector back-end electronics ~20 m Clock/Trigger Control/Data 5 Back-end elements Data Concentrator Cards (DCCs) to aggregate data from front-end Gbit Ethernet private network and PC for TPC local event building

RT2010 – Lisboa May Data Concentrator Cards Main requirements Fanout global 100 MHz clock, synchronous trigger to front-end with low skew (< 40 ns) Aggregate data from multiple front-ends (typ. 4 modules per DCC, i.e channels). Target DAQ rate: 20 Hz Configure front-end and read-back run-time settings Interfaces to TPC DAQ PC through standard Gbit Ethernet switch to front-end via 72 duplex optical links – 2 x 144 Gbps aggregate to Slave Clock Module via dedicated LVDS links 6 Design strategy Development of dedicated board too slow to meet project deadlines → Extend platform used for single module readout; scale it up to full size system by trivial duplication → Common hardware design with T2K Fine-Grain Detector (FGD) to share development effort

RT2010 – Lisboa May Doing more with an evaluation board 3+1 optical ports Gbit Ethernet RJ MHz ref. Clock Trigger input PLL 7

RT2010 – Lisboa May DCC Close-up Design risks and benefits Optical link board easily adds 3 SFP transceivers to ML405 External clock distribution requires hardware modification of ML405 → Lowest cost, minimal design effort and shortest deployment time solution 1 of 18 DCCs Optical link extension card Clock board External Clock Trigger input Xilinx ML405 Evaluation Platform 8

RT2010 – Lisboa May DCC Transmitters Eye Diagram 9 On-board SFPLink via SMA Link via SATA#1 Link via SATA#2 Prototype: Local 25 MHz local oscillator + LMK03000 PLL multiplication for 200 MHz RocketIO reference Final design: Silicon Laboratories Si5326 PLL

RT2010 – Lisboa May DCC Optical Transmitter Jitter 10 Optical link on SATA#2 connector Remarks Clock quality not significantly degraded by add-on cards and modifications on ML405 board In real experiment 100 MHz ref. clock have to use cat. 6 cable (~1 m) → All 72 links in final system running stably – Estimated BER <

RT2010 – Lisboa May Clock/Trigger fanout capability 11 Remarks Phase of recovered clock on Virtex 2 Pro / 4 RocketIO w.r.t. phase of sender not predictable from one system configuration to the next – but remains stable (<~2 ns) after link synchronized Limited control of skew of synchronization signals sufficient for this TPC → R&D on Virtex 5: predictable latency achievable with GTP transceivers DCC Clock FEM#1 Clock FEM#2 Clock DCC to FEM path delay variations over 100 FPGA reconfigurations (16 ns primary reference clock) Recovered clock on 2 FEMs compared to DCC reference clock (16 ns primary reference clock)

RT2010 – Lisboa May TPC Backend Electronics Crate Principles Each DCC mounted on an aluminum plate 6 DCCs per 6U crate (i.e. 1 crate reads-out 1 TPC or 1 FGD) All cables and fibers at the back; CompactFlash (firmware) at the front → Most practical way found to rack-mount non-standard form factor cards 1 of 3 DCC crates 12

RT2010 – Lisboa May 2010 Complete TPC Back-end System Front-end Power & Slow Control Rack Back-end Electronics Rack Front-end slow control PC Front-end power supplies (2 Wiener PL508) Crates of DCCs Slave clock module Ethernet switch Power supplies Power supply

RT2010 – Lisboa May DCC Embedded Firmware/Software 14 Principles Use hard IP blocks of Virtex 4: transceivers, processor, Ethernet MAC… Finite state machine in user logic pulls data from 4 front-ends in parallel Standalone server program in PowerPC 405 unloads front-end data, elementary checks, uncapsulates in UDP/IP frames sent by MAC

RT2010 – Lisboa May 2010 Hard-wired data collection FSM Next Channel ZS F (0 to 5) ZS, FEC, ASIC, Channel Finite State Machine Send Enable Sender entry LUT counters A (0 to 3) C (0 to 78) 1111 ZS0000 ZS0011 Rx FIFO 0 Free > 2 KB Sender 1 Sender 2 Sender 3 Rx FIFO 1 Free > 2 KB Rx FIFO 2 Free > 2 KB Rx FIFO 3 Free > 2 KB & request Suspend PowerPC 405 (300 MHz) PLB 32-bit 100 MHz Start event Abort Operation Requests data channel by channel from 4 front-end modules in parallel Suspends if any FIFO cannot store data of next request Next request posted automatically when data of pending request entirely received or pre-defined timeout has elapsed To RocketIO TX From RocketIO RX Fifo’s

RT2010 – Lisboa May 2010 T2K Nd280 DAQ system TPC DAQ PC … fetpcdcc 0 fetpcdcc 0 fetpcdcc 17 fetpcdcc 17 Local event builder Local event builder Cascade to/from DCCs via private Gbit Ethernet switch nd280 network nd280 network Global event builder Global event builder FGD DAQ PC PODD AQ PC xxx DAQ PC Global DAQ PC On-line database, Mass storage, etc. On-line database, Mass storage, etc. Architecture Local event building in TPC DAQ PC Bridge to MIDAS global event builder via Cascade Private nd280 network provides access to services shared by other detectors in the experiment See: M. Thorpe et al., “The T2K near Detector Data Acquisition Systems”, this conference See: R. Poutissou et al., “Cascading MIDAS DAQ Systems and Using MySQL Database for Storing History Information”, this conference

RT2010 – Lisboa May 2010 TPC DAQ System Performance Global system performance in running conditions of T2K 33 Hz event taking rate in zero-suppressed mode (required: 20 Hz) 50 ms latency for laser calibration events (1 DCC in full readout mode) Max. DCC throughput: 16 MB/s (limitation: PPC405 I/O over DS-OCM) Global throughput linear with # of DCCs until GbE link saturation → All requirements met – system in exploitation Event acquisition time for 1 DCC in full readout mode Event acquisition rate for 1 front-end module in zero-suppressed mode T2K laser events T2K beam, cosmic events

RT2010 – Lisboa May 2010 Experience Return System robustness and stability All optical links very stable despite sub-optimal clocking; BER < Clock skew stable in operation but variation from one power-up to next No hardware failure after ~6 months of operation Need improve handling of oversized events to minimize deadtime → Successful assembly of production system from evaluation hardware Beam spill + Cosmic events Typical TPC event size: ~65 kB (spill or cosmic) 750 kB on 1 DCC for laser calibration events Mean: 3.5 kB A cosmic event seen at T2K Nd280

RT2010 – Lisboa May Summary Purpose of the development A 1-to-72 clock-trigger fanout tree plus a 72-to-1 data aggregation system. Bridges TPC front-end 144 Gbps optical path to 1 Gbit Ethernet on DAQ side Key elements 18 Xilinx Virtex 4 evaluation boards with minimal add-ons + off-the-shelf PC and Ethernet networking products Performance and limitations Clock distribution stable during operation but each link settle to a different skew (~40 ns p.p.) at system power-up due to Virtex 2Pro / 4 RocketIO limitations – other R&D show this can be much better controlled with Virtex 5 Limited I/O capability of PPC405, DOCM faster than PLB. Our application: 130 Mbit/s usage of Gbe link – by-pass processor if one needs to fully exploit Gbe System pros and cons Low-cost and shortest deployment time solution. Evaluation kits not meant for production systems…but it works! 19 See poster: Y. Moudden et al., “The level 2 trigger of the H.E.S.S 28 meter Cerenkov telescope”

RT2010 – Lisboa May Outlook FPGA board design is getting more and more complex Manufacturer has to provide many guidelines, notes, reference design schematics…but still need skilled designers / layout engineers to build your own board. Time consuming, technical risks at design and production Many applications require same common blocks beyond FPGA itself De-coupling capacitors, external memory (DDR2, 3), configuration Flash, several high speed serial links, many customizable user I/O, flexible clocking Evaluation boards Originally to help designers make their own board design. Brings very latest FPGA technology instantly to anyone at low cost and no risk. An eval board has a lot in common with the core of a typical end product – why re-invent? Think evaluation boards (have them made) as macro-components? Worked for us. Improvements: lower-profile (2 cm stacking), smaller form factor, access to more user I/O and high speed serial links, flexible clocking → Towards a better eval. board paradigm?: FPGA + SDRAM + Flash, I/O connectors on a general purpose module directly re-usable by customer; the element for demo is the carrier board (largest diversity of interface standards) 20