DUAL-OUTPUT HOLA FIRMWARE AND TESTS Anton Kapliy Mel Shochet Fukun Tang Daping Weng

Summary New dual-output HOLA board design with independent flow control from DAQ and FTK channels See Fukun Tangs slides for a detailed walkthrough New firmware to interface the S-LINK core to inexpensive Altera Cyclone IV GX transceivers 90% of original HOLA code preserved. Fitting & assembling performed in Quartus 10.1 SP1 We also have an experimental port to Quartus 11.0 (released in May 2011) Simulation using a comprehensive Modelsim testbench Manufactured, stuffed, and tested two prototype boards: Firmware loaded on power-up from a 4 Mbit EPROM (EPCS4) Data sent through both channels Ensured that flow control can be asserted from either channel BER tests using 1.35E15 bits sent through both channels 2

What is a HOLA? 3 A device that implements the S-LINK protocol for point-to-point data transfer. Pixel and SCT data in a ROD is organized into 32-bit chunks and sent to a HOLA card, which serializes and sends it through optical fibers. Another HOLA card receives this data stream in a ROB (and FTK DF) and de-serializes it. S-LINK implements data framing, error detection, and flow-control.

4 FIFO 40 MHz 50 MHz Interface to TLK TLK2501 chip (sends data) FPGA TLK2501 chip (receives data) FPGA provides a parallel interface to an outside SERDES device (TLK-2501), which feeds serial signal to an optical transmitter. However, TLK-2501: Is deprecated and not marketed by TI anymore Costs more than a Cyclone IV FPGA Consumes four times as much power 100 MHz Big picture: original HOLA LSC (only forward channel shown) Mezzanine data port TX_EN TX_ER Serial connection (via optical transceiver)

Big picture: new HOLA LSC 5 FIFO 40 MHz 50 MHz Interface to TLK ALTGX wrapper (emulates TLK2501) FPGA Serial connections (via optical transceivers) (only forward channel shown) Xilinx FPGA (FTK DF) TLK2501 chip (DAQ) OLD LSC CORE (Altera SERDES megafunction) TX_CLK 90% of original LSC code is unchanged. ALTGX wrapper emulates TLK2501: Plugs in into the original LSC core code Implements link startup and synchronization TLK receiver on DAQ side is oblivious to the change 100 MHz Mezzanine data port TX_EN TX_ER

Modelsim testbench 6 Stefan Haas gave us a Modelsim testbench implementing the above configuration. We updated some features of this testbench to fit our purposes: Serial LSC LDC communication is simulated using Altera transceivers LSC gains another channel (thus, dual-output HOLA) Another LDC+ROMB block was added to receive data from the second channel. FEMB, LDC, and ROMB were simulated functionally, while LSC (including Altera transceiver interface) - using a compiled & fitted (gate-level) Altera Quartus design. RODDAQ/FTK ROB

PC setup: ROD side S32PCI64 SOLAR mezzanine card: Provides access to S-LINK via PCI bus The first prototype of dual-output HOLA Mezzanine connector to transfer 32-bit data 2 x optical transceivers FEMB S-LINK LSC 7

PC setup: ROB and FTK side FILAR card (based on S32PCI64): Implements S-LINK protocol Provides access to S-LINK via PCI bus 8 S-LINK LDC and ROMB (in one package) FTK DAQ Identical data is received from the two fibers running from the dual- output HOLA. Note that each channel can assert flow control. The two channels are read out independently. E.g., we can read out all pending buffers from DAQ, so that only FTK will be asserting flow control. Multi-mode fibersOptical attenuators (7 850 nm)* * We also ran BER tests to ~2E14 bits using attenuators rated for nm

BER test with 1.35E15 bits We use the so-called SLIDAS mode in SOLAR PCI card For the main BER test, we used the following configuration: Continuous frames with 254 words x 32 bits are fed into HOLA LSC These words follow a pseudo-random sequence reproducible in C++ On reception, we check word-by-word that all received patterns are correct PC presents a bottleneck: need to generate these pseudo- random patterns and compare them with received data words. We can achieve about 65 MB/s (21 days to test BER to 1E15). Under this scheme, we continuously exercise flow control: FEMB wants to send data at nearly 200 MB/s, but ROMB & LDC buffers quickly fill up and assert flow control until they are read out and recharged by the slow PC Effectively, flow control is asserted / de-asserted continuously 1.35E15 bits were received by both FILAR channels without any error (~4 weeks of non-stop transmission) 9

Additional tests After the BER test was completed, Daping spent a couple of weeks running additional tests: Generating data on the PC (rather than using SLIDAS patterns) Send data to the SOLAR card via the first PCI port Read it back on reception from the FILAR via the second PCI port. SLIDAS random data with words per frame Theoretically, the frames can be even larger, but our FILAR device drivers are hardcoded to allocate enough memory only for 1024 words. ROD mode Specially-designed frames that look like LHC data Each frame contains a header, payload, and a footer TEST mode (a walking-bit pattern) All tests have finished without any errors 10

BACKUP SLIDES 11

SLIDAS ROD mode Data pattern: There is one control word at the beginning of each frame Next is a header of length 9 words, containing information indicating that this is the Nth frame sending The header is followed by an increasing arithmetic progression starting from 0 and ending at (frame length – 15) Next is a footer of length 3 words Ending control word Length options are the same as in the random data mode We tested to the order of 1E9 words, 256 words per frame and no error was observed 12 A Typical Frame: Control word N (mark of the Nth frame) 3N-2 mod Frame length – 15 1 Frame length – 14 0 Control word