1. Grzegorz Korcyl - Jagiellonian University, Kraków Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

2. Outline 1. Motivation 2. General structure a) GbE Buffer b) Packet Constructor c) Frame Constructor d) Frame Transmitter e) Resources utilization 3. Results a) Setup 1 b) Setup 2 4. Summary 5. Future plans Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

3. 1. Motivation The way to transport large amounts of data at high speed Protocol with routing capabilites needed UDP over GbE is going to be implemented on HUB2, MDC, Shower addons Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010 Addon + GbE 1 Evt builder 2 Evt builder 3 Evt builder GbE Switch

4. 2. General Structure TrbNet – Hades trigger, slow control and data transport network system GbE Buffer gathers subevents from TrbNet and prepares packet construction Packet Constructor creates a HadesTransportUnitQueue Frame Constructor fragments data and creates Ethernet frames Frame Transmitter transmits each frame Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

5. 2. General Structure Message Subevent 1Subevent 2Subevent 3Subevent 4 Packet 1Packet 2Packet 3 Frame 1Frame 2Frame 3Frame 1Frame 2Frame 3Frame 1Frame 2 Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010 GbE Buffer Packet Constructor Frame Constructor TrbNet ETHERNET Frame Transmitter

6. 2.a GbE Buffer Interface between TrbNet and GbE logic Buffers incoming data Responds back to readout request Allows configuration via slow control Pepares the data for Packet Constructor Decides about the start/close a packet Selects event builder address basing on the readout parameteres Adds additional internal subevent headers Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

7. 2.b Packet Constructor Constructs HadesTransportUnitQueue: Several subevents in one queue Consistency check by termination bytes Right now up to 64kB in total Minimum overhead (one subevent case) is 56B per queue Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

8. 2.c Frame Constructor Constructs IP frames: Fragments the whole queue into 1.5kB (MTU) parts and encapsulates with network headers 42B of headers overhead per frame Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

9. 2.d Frame Transmitter Uses two IP Cores to transmit bytes over fiber: 1. Tri-Speed MAC Transmit state machine Insertion of pause frames Addition of Ethernet preamble and CRC 2. SGMII/GbE PCS Interface between TS MAC and SERDES Link state machine Autonegotiation Conversion to 8b/10b Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

10. 2.e Resources utilization Built on Lattice ECP2M LFE2M100E Mainly consumes RAM blocks inside FPGA Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010 ResourceUtilizationPercentage Slices4256 out of 473768% LUTs5984 out of 950006% Blocks RAM76 out of 288 (~150kB)26%

11. 3.a Results – setup 1 One data source – dummy front-end HUBv2 Netgear optical GbE switch One event builder Triggering by TRB Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

12. 3.a Results – setup 1 No pipelining, single packet construction at a time limits throughput to half of nominal link bandwidth Event rate limited by single trigger in the system at a time Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

13. 3.b Results – setup 2 5x ADCM data sources HUBv2 Longshine optical GbE switch One event builder Triggering by generator Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

14. 3.b Results – setup 2 Packet size 38.8kB Accepted event rate 1.25kHz and 49MB/s throughput Long-lasting and stable connection Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010 Old DAQNew DAQFactor Bandwith to Event builder6 MB/s150 MB/s25x faster Event rate / sector50 Hz1.25 kHz25x faster Throughput / sector1 MB/s49 MB/s50x faster

15. 4. Summary 1. Construction of HadesTransportUnitQueues of the size up to 64kB 2. Transmission of packets at satisfying speed 3. Multiple subevents combined into one queue 4. Configuration via slow control 5. Autonegotiation 6. Dynamic event builder selection Basic, ready to use GbE component is ready Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010

16. 5. Future Plans 1. Stabilize fast packet construction Pipeline packet construction in order to reach the full Gb 2. Large queues fragmentation into multiple UDP packets Ability to transport queues (subevents) larger than one UDP packet (64kB) Test different MTU sizes (Jumbo packets) in order to find the optimal configuration 3. Control of the transmission status and statistics Generate status information and create systems to handle unexpected situations Full featured GbE component planed to be ready in May Grzegorz Korcyl – PANDA TDAQ Workshop, Giessen April 2010