1. Use of FPGA for dataflow Filippo Costa ALICE O2 CERN

2. Filippo Costa FILIPPO COSTA
I am working in the ALICE DAQ GROUP, as software and firmware developer, since 2007.
As software developer my responsibilities are:
readout:
the program that collects data from the detectors and stores them in the PC’s memory.
TPCC:
software used to send configuration commands to the TPC electronics (the biggest detector in ALICE) over 216 optical fibers.
As firmware developer my responsibilities are:
DAQ readout card firmware:
the PCIe card hosting FPGA that collects data from the detectors through different optical connections and stores the information in the memory of the hosting PC, performing DMA transaction.
DDG:
a data generator able to send data with different communication protocols used in the ALICE experiment. It is used to simulate the FEE behavior and test the readout chain.
CRU readout card firmware:
the new PCIe card that will be used in the future runs starting from 2018.
FILIPPO COSTA
Nickname
PiPPo
Job
Software and firmware developer in the ALICE DAQ GROUP since 2007

3. MOTIVATION GBT PCIe DMA engine Outline
why do we use FPGA in HEP experiments?
GBT
a radiation tolerant custom protocol
PCIe DMA engine
data is stored in the PC memory

4. MOTIVATION We work in extreme environment condition:
experiments are located
50 – 100 m
under ground
(Earth provides shielding from external activity (cosmic rays) and prevents the particles, generated during collisions, to escape.)

5. MOTIVATION
CUSTOM PROTOCOL: the FEE (Front End Electronics) is installed in area where the radiation level is high. For this reason we use custom communication protocol, developed to be radiation tolerant, to transfer data from the detector to the O2 system.
SPECIFIC DETECTOR SLOW CONTROL CONFIGURATION: the detector electronics is not accessible at any time, during beams activity no one is allowed in the experiment area.
THE READOUT CARD HAS TO INTERFACE WITH DIFFERENT FEE: different detector groups have different readout requirements in terms of data size and acquisition rate.

6. MOTIVATION
It is very important to move data from the Experiment to the O2 farm, in a fast and reliable way
Working in a radiation environment reduces considerably the choices of data transmission protocol we can use.
Ethernet protocol would be a natural choice to move data from point A to B, however the implementation in FPGA requires high number of resources and specific features (high speed SERDES).
RADIATION TOLERANT FPGAs have limited resources, it is not possible to implement fast protocol like 10 Gb/s Ethernet. B A
DATA FLOW
RADIATION ENVIRONMENT

7. GBT (GigaBit Transceiver)
The GBTx is a radiation tolerant chip, developed at CERN, that can be used to implement multipurpose high speed (3.2, Gb/s bandwidth) bidirectional optical links for high-energy physics experiments.
reference (GBT_FPGA user guide v1_21 pdf)

8. GBT (GigaBit Transceiver)
The main features of the chip are:
A single bi-directional link is used simultaneously for:
trigger and timing control distribution,
data readout,
experiment slow control and monitoring
The link is a p2p, optical, bidirectional, constant latency connection that can work with very high reliability in the harsh radiation environment of the LHC experiments. 1 1 2 2 3 3

9. GBTx chip The different operation: receiving data from the FEE.
On the readout cards, installed in the O2 system, it is mandatory to be able to decode the GBT protocol and extract the data and monitoring information, before storing them in the memory of the PC.
The different operation:
receiving data from the FEE.
data processing.
delivering of the clock and trigger to the FEE.
configuration and monitoring of the FEE.
must all be performed in parallel and as fast as possible.

10. Custom protocol + parallel processing = FPGA
General features of latest FPGA:
transceiver speed: 28 Gbps
PCIe: Gen.3 x 8
Logic elements: 1M
GFLOPS: 1366
FPGA is a chip able to perform parallel tasks at the same time. It is configurable allowing the developer to implement basically any type of communication protocol.

11. FPGA & GBT
The GBT-FPGA core targets FPGAs from ALTERA and XILINX, allowing the implementation of 2 different types:
STANDARD
LATENCY OPTIMIZED
And 3 different encoding mode :
GBT-Frame
8b10b
Wide-bus

12. GBT bank configuration
Using a single text file it is possible to implement the different options for the
GBT LINK
constant GBT_BANKS_USER_SETUP : gbt_bank_user_setup_R_A(1 to NUM_GBT_BANKS) := (
-- GBT Bank 1:
1 => (NUM_LINKS => 4, Comment: * 1 to 4
TX_OPTIMIZATION => STANDARD,
RX_OPTIMIZATION => STANDARD,
TX_ENCODING => WIDE_BUS,
RX_ENCODING => WIDE_BUS,
STD_MGT_REFCLK_FREQ => FREQ_240MHz,
SPEEDUP_FOR_SIMULATION => false)--,

13. (bi-dir optical fiber)
The DATA-FLOW
GBT-HDL O2
FIRMWARE +
SOFTWARE
FEE
DATA
GBT-HDL
GBT-HDL
GBT DATA
(bi-dir optical fiber)
~13000 links p2p
~6 TB/s sustained
PCIe DMA is used to transfer data from the readout card to the O2 system
radiation
zone

14. PCIe DMA engine, DATA stored in the PC

15. (Torsten Alt’s presentation)
The DATA-FLOW (FEE – O2)
CRU
GBT DATA
DATA PROCESSING
(Torsten Alt’s presentation)
PCIE

16. The CONTROL-FLOW (O2 – FEE)
CRU
slow control data
SC DATA
PCIE

17. The TRIGGER-FLOW (CTP– FEE)
Central
Trigger
Processor
CRU
TRIGGER
interface
Trigger
Clock
PCIE

18. The FLOW CRU Central Trigger Processor TRIGGER Trigger interface Clock
slow control data
SC DATA
GBT DATA
DATA PROCESSING
PCIE

19. As fast as possible … how fast is it?

20. Same firmware, different hardware
The GBT code can be implemented on different FPGA families.
Using different hardware it is possible to use the same features.

21. Same functionalities, different detectors
All the detectors have a main goal in common, to detect particles and send the information to the O2.
However due to the different features of each detector the requirements to send data are different.
Example of data-flow CRU firmware:
TPC: cluster finder
TOF: GBT RAW data, load balance
TRD: custom LINK (no GBT) coming from the FEE
Given the complexity of the operation to be performed during data taking we will have different firmware to implement all the different features but using the same readout card hardware.

22. Questions?