1. The STAR Heavy Flavor Tracker PXL detector readout electronics
J. Schambach
(University of Texas at Austin)
G. Contin, L. Greiner, T. Stezelberger, C. Vu (LBNL)
X. Sun (CCNU)
M.Szelezniak (IPHC)

2. Talk Outline Introduction: STAR, HFT, PXL Electronics: Hardware
Electronics: Firmware
Interfaces: DAQ, USB, Slow Controls
Summary
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

3. Heavy Flavor Tracker @ STAR
RHIC
Magnet
HFT
PXL
IST
SSD
TPC
TPC – Time Projection Chamber: main STAR tracking detector
HFT – Heavy Flavor Tracker
SSD – Silicon Strip Detector
IST – Intermediate Silicon Tracker
PXL – Pixel Detector
Tracking inwards with graded resolution:
 = ~1 mm
R (cm)
SSD r = 22
IST r = 14
PXL r2 = 8
r1 = 2.8
 = ~300 m
 = ~250 m
 = <30 m
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

4. First MAPS based vertex detector at a collider experiment
HFT Subsystems
Silicon Strip Detector (SSD)
Double sided silicon strip modules with 95 µm pitch
Existing detector with new faster electronics
Radius: 22 cm
Intermediate Silicon Tracker (ITS)
Single sided double-metal silicon pad with 600 µm x 6 mm pitch
Radius: 14 cm
PiXeL detector (PXL)
MAPS sensors with 20.7 µm pitch pixels
Radius: 2.8 and 8 cm
First MAPS based vertex detector at a collider experiment
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

5. Ladder with 10 MAPS sensors (~ 2×2 cm each)
PXL Detector Design
2 layers
10 sectors
2 detector halves
5 sectors / half
4 ladders / sector
MAPS Sensor
“Ultimte-2”
20 cm
RDO Buffers / Drivers
MAPS
2-layer kapton flex cable with Al traces
Ladder with 10 MAPS sensors (~ 2×2 cm each)
PXL Sensor
928 rows * 960 columns =~ 1M pixel
Rolling shutter, column-parallel readout through end-of-column discriminators
185.6 us integration time
MIP Signal ~ 1000 e-, S/N ~ 30
Configuration via JTAG
In-pixel Correlated Double Sampling (CDS)
On-chip zero-suppression and run-length encoding on rows (up to 9 hits / row)
2 memory banks of 1500 words each for frame readout in ping-pong configuration
2 LVDS data 160 MHz
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

6. Sector Readout Electronics Chain
11 m
(24 AWG TP)
Clock, config, data, power
2 m (42 AWG TP)
Clk, config, data
PXL Sector
Mass Termination Board signal buffering + latch-up protected power
RDO board
with Xilinx Virtex-6 FPGA
DAQ PC with fiber link interface board
(ALICE “RORC”)
100 m (fiber optic)
PXL built events
Highly parallel system
4 ladders per sector
1 Mass Termination Board (MTB) per sector
1 sector per RDO board
10 RDO boards total in the PXL system
Trigger,
Slow control,
Configuration,
etc.
Existing STAR
infrastructure
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

7. PXL Readout Power Supplies
Networking
SIU
JTAG
USB
Daughter Card
TCD Interface
USB Hub
Fiber-Optics Hub
Power Supplies
356 M pixel readout in a single 9U size crate
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

8. Detailed Sector Readout Architecture
Ladder x 4
RDO board x 1
DAQ
RDO PC
FPGA
fiber
SIU
x 4
TCD
I/F
Trigger
LU prot. power
SRAM
USB
Black – cfg, ctl, clk. path
Blue – data path
Red – power / gnd path
Power
Supplies
Control PC
MTB x 1
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

9. Mass Termination Board (MTB)
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

10. Readout Board Block Diagram
SIU
Fiber
module
Events, control
control, events
trigger
Trigger
interface
trigger
FTDI
USB
module
Events, control
Xilinx
Virtex-6
FPGA
BUSY
VME P1
BUSY
configuration x4
JTAG
Flash
config
drivers
VHDCI
68 pin
Clock, control, data To
MTB
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

11. Readout Board (RDO) SIU Virtex-6 LX240T-FF1759: 15 Mb RAM Power
720 User I/O
high-performance SelectIO technology
FPGA Configuration via either Platform Flash XL (DS617) or front panel JTAG header
USB Interface: FTDI FT2232H daughter card
DAQ I/F: ALICE DDL “SIU” daughter card
Trigger (TCD) interface via daughter card
Ladder interfaces on the back of the board via VHDCI connectors:
Data (LVDS) from sensors
Clock to sensors
JTAG for sensor config
I2C for MTB monitoring & control
Power
TCD I/F
Busy
Status
LEDs V6
USB
Ladder I/Fs
Power
Clk
Config
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

12. Firmware: Readout Requirements
40 sensors per RDO = 80 data LVDS pairs at 160 MHz
Data consists of bit-serially transmitted run-length encoded hit addresses
Average trigger rate in excess of 1kHz
Data generation speed (40 sensors x 160MHz x 2 = 12.8 Gb/s) must be reduced to fiber speed (2Gb/s)
RDO burst (buffer) capability must match TPC RDO capability (1 event every 50 μs in bursts up to 8 events) in order to not increase the DAQ dead time
Each trigger will result in a separate event containing a full frame to DAQ:
frame1
frame2
frame3
Trigger time
Event data
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

13. Firmware Architecture
Sensor output
Clock, JTAG & Control
Trigger
Temperature, Voltage, Current & Latch-up Monitoring
IOdelay
Ladder Interface
Event Readout
Configuration
Slow Controls
Fiber
Interface
USB
Interface
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

14. Firmware: Event Readout Module
Sensor Output
Sensor Output
Sensor Output
Sensor Output
160 Mhz
Serial To
Parallel
Converter
Serial To
Parallel
Converter
Serial To
Parallel
Converter
Serial To
Parallel
Converter
10 Mhz
valid(1:8) & data(32)
valid(1:8) & data(32)
Trigger
&
Readout
Controller
Mux (40X20) : 40
Mux (40X20) : 40
Trigger
valid(1)
valid(8)
valid(1)
valid(8)
200 Mhz
data(32)
Event
Buffer
Event
Buffer
Event
Buffer
Event
Buffer
Busy feedback
Mux (32X8) : 32
Mux (32X8) : 32
Mux (32X2) : 32
200 Mhz
Event FIFO
50 Mhz
Fiber
(SIU)
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

15. PXL DAQ Data Format Header Event Sensor Hit Data
Header token
Trigger word
RHIC counter
temperature
Firmware version
Hardware ID
…….
reserved
system status
Event
Header
Hit Block Length
Hit Addresses
Separator (0xCCCCCCCC)
High Level Trigger Info
CRC of all preceding words
Ender (0xBBBBBBBB)
Sensor Hit Data
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

16. USB Protocol FT2232H chip’s FIFO interfaces to FPGA Firmware
Two levels of protocol: USB-FPGA low level, FPGA High Level Protocol
Low Level Protocol: Separate “WRITE” and “READ” transaction to 2 Firmware FIFOs
WRITE Transaction
Start of READ Transaction
High Level Protocol:
32bit “Command words” interface to “memory-mapped” 16-bit registers
“WRITE” command word: set configurations or start actions
“READ” command word: results in Data in “READ” FIFO, which can then be read with a READ low- level transaction
Command Word Format:
31-28
27-16
15-0
4bit control
31 READ/NOT-WRITE
not defined
12bit address
16bit data
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

17. USB Software Interface
Lowest level interface via libD2xx or lib_ftdi on either Linux or Windows
Four types of memory mapped registers:
“Configuration” (R/W)
“Status” (read-only)
“Action” (write-only) – start FPGA transactions like JTAG or I2C
“Indirect” (R/W) – memory or FIFO interfaces (“Count”, “Address”, “Data”)
PXL library provides convenience functions in C++ and Python
open_ftdi, close_ftdi
readReg, writeReg
readMem, writeMem
Python used for scripting and control GUIs
Interface to STAR Slow controls (EPICS) via EPICS “soft-IOCs” that C++ and Python can read/write
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

18. PXL Slow Controls (EPICS and Python)
Power Supply and Cooling Monitor
PXL control GUI
Power up, configure and check for errors
Controlled shut down
Fast reset
Status indicator
Error indicator
Process monitoring terminal
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015

19. Thank you for your attention
Concluding Remarks
Readout Performance:
Readout through DAQ tested up to ~3 kHz (limit of trigger system)
Typical event rate during 200 GeV Au-Au ~ 1kHz with <5% PXL dead time
Typical Au-Au PXL data rate about 120 MB/s
PXL operation highly scripted, very little shift operator intervention needed (mostly just turning PXL on and off)
Latch-up events cleared automatically by RDO firmware
Electronics was used successfully during STAR Run 14 and Run 15
More than 1B events taken each run, which allowed STAR to perform direct topological reconstruction of charmed hadrons
Thank you for your attention
J. Schambach University of Texas at Austin
TWEPP 2015, Sep 28 – Oct 2, 2015