The STAR Heavy Flavor Tracker PXL detector readout electronics J. Schambach (University of Texas at Austin) G. Contin, L. Greiner, T. Stezelberger, C. Vu (LBNL) X. Sun (CCNU) M.Szelezniak (IPHC)

Talk Outline Introduction: STAR, HFT, PXL Electronics: Hardware Electronics: Firmware Interfaces: DAQ, USB, Slow Controls Summary J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Heavy Flavor Tracker @ STAR STAR @ RHIC Magnet HFT PXL IST SSD TPC TPC – Time Projection Chamber: main STAR tracking detector HFT – Heavy Flavor Tracker SSD – Silicon Strip Detector IST – Intermediate Silicon Tracker PXL – Pixel Detector Tracking inwards with graded resolution:  = ~1 mm R (cm) SSD r = 22 IST r = 14 PXL r2 = 8 r1 = 2.8  = ~300 m  = ~250 m  = <30 m J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

First MAPS based vertex detector at a collider experiment HFT Subsystems Silicon Strip Detector (SSD) Double sided silicon strip modules with 95 µm pitch Existing detector with new faster electronics Radius: 22 cm Intermediate Silicon Tracker (ITS) Single sided double-metal silicon pad with 600 µm x 6 mm pitch Radius: 14 cm PiXeL detector (PXL) MAPS sensors with 20.7 µm pitch pixels Radius: 2.8 and 8 cm First MAPS based vertex detector at a collider experiment J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Ladder with 10 MAPS sensors (~ 2×2 cm each) PXL Detector Design 2 layers 10 sectors 2 detector halves 5 sectors / half 4 ladders / sector MAPS Sensor “Ultimte-2” 20 cm RDO Buffers / Drivers MAPS 2-layer kapton flex cable with Al traces Ladder with 10 MAPS sensors (~ 2×2 cm each) PXL Sensor 928 rows * 960 columns =~ 1M pixel Rolling shutter, column-parallel readout through end-of-column discriminators 185.6 us integration time MIP Signal ~ 1000 e-, S/N ~ 30 Configuration via JTAG In-pixel Correlated Double Sampling (CDS) On-chip zero-suppression and run-length encoding on rows (up to 9 hits / row) 2 memory banks of 1500 words each for frame readout in ping-pong configuration 2 LVDS data outputs @ 160 MHz J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Sector Readout Electronics Chain 11 m (24 AWG TP) Clock, config, data, power 2 m (42 AWG TP) Clk, config, data PXL Sector Mass Termination Board signal buffering + latch-up protected power RDO board with Xilinx Virtex-6 FPGA DAQ PC with fiber link interface board (ALICE “RORC”) 100 m (fiber optic) PXL built events Highly parallel system 4 ladders per sector 1 Mass Termination Board (MTB) per sector 1 sector per RDO board 10 RDO boards total in the PXL system Trigger, Slow control, Configuration, etc. Existing STAR infrastructure J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

PXL Readout Power Supplies Networking SIU JTAG USB Daughter Card TCD Interface USB Hub Fiber-Optics Hub Power Supplies 356 M pixel readout in a single 9U size crate J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Detailed Sector Readout Architecture Ladder x 4 RDO board x 1 DAQ RDO PC FPGA fiber SIU x 4 TCD I/F Trigger LU prot. power SRAM USB Black – cfg, ctl, clk. path Blue – data path Red – power / gnd path Power Supplies Control PC MTB x 1 J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Mass Termination Board (MTB) J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Readout Board Block Diagram SIU Fiber module Events, control control, events trigger Trigger interface trigger FTDI USB module Events, control Xilinx Virtex-6 FPGA BUSY VME P1 BUSY configuration x4 JTAG Flash config drivers VHDCI 68 pin Clock, control, data To MTB J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Readout Board (RDO) SIU Virtex-6 LX240T-FF1759: 15 Mb RAM Power 720 User I/O high-performance SelectIO technology FPGA Configuration via either Platform Flash XL (DS617) or front panel JTAG header USB Interface: FTDI FT2232H daughter card DAQ I/F: ALICE DDL “SIU” daughter card Trigger (TCD) interface via daughter card Ladder interfaces on the back of the board via VHDCI connectors: Data (LVDS) from sensors Clock to sensors JTAG for sensor config I2C for MTB monitoring & control Power TCD I/F Busy Status LEDs V6 USB Ladder I/Fs Power Clk Config J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Firmware: Readout Requirements 40 sensors per RDO = 80 data LVDS pairs at 160 MHz Data consists of bit-serially transmitted run-length encoded hit addresses Average trigger rate in excess of 1kHz Data generation speed (40 sensors x 160MHz x 2 = 12.8 Gb/s) must be reduced to fiber speed (2Gb/s) RDO burst (buffer) capability must match TPC RDO capability (1 event every 50 μs in bursts up to 8 events) in order to not increase the DAQ dead time Each trigger will result in a separate event containing a full frame to DAQ: frame1 frame2 frame3 Trigger time Event data J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Firmware Architecture Sensor output Clock, JTAG & Control Trigger Temperature, Voltage, Current & Latch-up Monitoring IOdelay Ladder Interface Event Readout Configuration Slow Controls Fiber Interface USB Interface J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Firmware: Event Readout Module Sensor Output Sensor Output Sensor Output Sensor Output 160 Mhz Serial To Parallel Converter Serial To Parallel Converter Serial To Parallel Converter Serial To Parallel Converter ...20... ...20... 10 Mhz valid(1:8) & data(32) valid(1:8) & data(32) Trigger & Readout Controller Mux (40X20) : 40 Mux (40X20) : 40 Trigger valid(1) valid(8) valid(1) valid(8) 200 Mhz data(32) Event Buffer Event Buffer Event Buffer Event Buffer ...8... ...8... Busy feedback Mux (32X8) : 32 Mux (32X8) : 32 Mux (32X2) : 32 200 Mhz Event FIFO 50 Mhz Fiber (SIU) J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

PXL DAQ Data Format Header Event Sensor Hit Data Header token Trigger word RHIC counter temperature Firmware version Hardware ID ……. reserved system status Event Header Hit Block Length Hit Addresses Separator (0xCCCCCCCC) High Level Trigger Info CRC of all preceding words Ender (0xBBBBBBBB) Sensor Hit Data J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

USB Protocol FT2232H chip’s FIFO interfaces to FPGA Firmware Two levels of protocol: USB-FPGA low level, FPGA High Level Protocol Low Level Protocol: Separate “WRITE” and “READ” transaction to 2 Firmware FIFOs WRITE Transaction Start of READ Transaction High Level Protocol: 32bit “Command words” interface to “memory-mapped” 16-bit registers “WRITE” command word: set configurations or start actions “READ” command word: results in Data in “READ” FIFO, which can then be read with a READ low- level transaction Command Word Format: 31-28 27-16 15-0 4bit control 31 READ/NOT-WRITE 30-28 not defined 12bit address 16bit data J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

USB Software Interface Lowest level interface via libD2xx or lib_ftdi on either Linux or Windows Four types of memory mapped registers: “Configuration” (R/W) “Status” (read-only) “Action” (write-only) – start FPGA transactions like JTAG or I2C “Indirect” (R/W) – memory or FIFO interfaces (“Count”, “Address”, “Data”) PXL library provides convenience functions in C++ and Python open_ftdi, close_ftdi readReg, writeReg readMem, writeMem Python used for scripting and control GUIs Interface to STAR Slow controls (EPICS) via EPICS “soft-IOCs” that C++ and Python can read/write J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

PXL Slow Controls (EPICS and Python) Power Supply and Cooling Monitor PXL control GUI Power up, configure and check for errors Controlled shut down Fast reset Status indicator Error indicator Process monitoring terminal J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015

Thank you for your attention Concluding Remarks Readout Performance: Readout through DAQ tested up to ~3 kHz (limit of trigger system) Typical event rate during 200 GeV Au-Au ~ 1kHz with <5% PXL dead time Typical Au-Au PXL data rate about 120 MB/s PXL operation highly scripted, very little shift operator intervention needed (mostly just turning PXL on and off) Latch-up events cleared automatically by RDO firmware Electronics was used successfully during STAR Run 14 and Run 15 More than 1B events taken each run, which allowed STAR to perform direct topological reconstruction of charmed hadrons Thank you for your attention J. Schambach University of Texas at Austin TWEPP 2015, Sep 28 – Oct 2, 2015