1. FELIX Design FELIX Design Upgrades of detector readout meeting 9 June 2014 Lorne Levinson, for the FELIX group Upgrades of detector readout meeting, 9 July 20141L. Levinson

2. Guiding principles Goal: a GBT to LAN “switch” with as little awareness of detector specifics as possible – No user code in FELIX (so far): configuration params for options Separate data movement from data processing Configurable data paths from E-link to network endpoint Scalable at Front end, Back end, and switch levels Do as much as possible in software COTS FPGA PCIe board – But… see talk on TTC clock recovery by Jinlong Zhang Readout, so latency is not important, but provide low latency GBT-to-GBT links Future proof by following COTS market, separately upgradeable components: GBT, FPGA PCIe board, serve PC, network cards Routing is for E-links; they are the logical connections L. LevinsonUpgrades of detector readout meeting, 9 July 20142

3. Virtex-7 PCIe board L. Levinson Upgrades of detector readout meeting, 9 July 2014 3 From Global HiTech, PCIe gen 3, 8 lanes Virtex X690T 2x12 bidir CXP connectors, FMC

4. Off- detector architecture with FELIX Functions can now be separated, physically, or just logically. “ROD” = Event fragment builder/processor for ROS L. LevinsonUpgrades of detector readout meeting, 9 July 20144 “ROD”

5. L. LevinsonUpgrades of detector readout meeting, 9 July 20145

6. Routing Configured via control port on main or separate network Simple/fixed: FELIX port # / E-link-range → IPaddress / port or → Infiniband local identifier / queue pair E-link packet protocol: – Allows several logical streams to use the same E-link  header with: stream-id + packet len (if not fixed length packets) FELIX port # / stream-id → IPaddress / port… Cloning, broadcasts / multicasts possible, in either direction Reliable transport from FELIX to end-point via TCPIP Configurable Quality of Service per logical connection – Less-than-best-effort for event data for monitoring L. LevinsonUpgrades of detector readout meeting, 9 July 20146

7. Use cases – 1 1.Receive data from an E-link or bonded group of E-links (a channel) and route it to one or more network endpoints via TCP or UDP. Bits must be arranged into correct order. – Optional 8b/10b or HDLC checking and/or decoding – Optionally verify checksum – Packet boundary identification either by SOP/EOP or length, optionally there can be no packet boundary, just a byte stream. If a maximum packet length is configured, data arriving after the maximum is discarded and this error is marked in the packet status word. – If streams are defined for this channel packets are routed according to the stream. The stream ID may contain an extension filled by the detector with part of the Level-1 ID or the Trigger Type for advanced event routing. – Sampling can be requested L. LevinsonUpgrades of detector readout meeting, 9 July 20147

8. L. Levinson Upgrades of detector readout meeting, 9 July 2014 8

9. Use cases – 2 2.Receive BUSY from an E-link command, internally from FELIX itself, or from the network. Then route it to a network endpoint or to a standard ATLAS RODBUSY signal (open collector Lemo). In Phase II it may be sent to the TTC system. 3.Receive TTC information (BC, L1A, ECR, BCR, user bits) and broadcast them to the requested E-links – The BC clock is supplied to the GBT because the FELIX's GBT link clock is the BC clock supplied by the TTC system to FELIX. – Three options: 2-bit, 4-bit, 8-bit as described in the datasheet. 4.Receive data from a TCP port and route it to one or more E-links – Optional 8b/10b or HDLC encoding – A specific multicast channel can broadcast packets to all out-going E- links with the "MC" attribute. L. LevinsonUpgrades of detector readout meeting, 9 July 20149

10. Use case – 3 8.Scheduled data transfer lists: setting and activating (list is per GBT, but E- links are enabled by a bit map.) Open Issue: Or, several lists per FELIX and each E-link configures to use a specific list. Each item in the list is qualified by a delay to wait from the previous item in the list. – Activation by TTC user bit – Activation by command received via TCP 9.Receive data from an E-link or bonded group of E-links (a channel) and route it to a Direct Output Low Latency link (probably a GBT). Typically only a particular stream in the channel would be so routed. L. LevinsonUpgrades of detector readout meeting, 9 July 201410

11. L. Levinson Upgrades of detector readout meeting, 9 July 2014 11 Low latency path is between predefined GBT pairs. Uses “cut-through” Routing.

12. FELIX: FW/SW interface Data interface PCIe low level packet format (fixed length) PCIe high level packets, spans low level packets Push/pull? Hints from RobinNP architecture Configuration parameter interface Need description of parameters that can generate: XML for OKS, VHDL for FPGA See (but needs to be updated): https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf https://twiki.cern.ch/twiki/pub/Atlas/GBT2LAN/MappingGBT_DataToNetworkEndpoints.pdf Monitoring and statistics interface FPGA configuration interface IP core from HitechGlobal ~4k$ L. LevinsonUpgrades of detector readout meeting, 9 July 201412

13. L. Levinson Upgrades of detector readout meeting, 9 July 2014 13

14. L. Levinson Upgrades of detector readout meeting, 9 July 2014 14

15. Backup L. Levinson Upgrades of detector readout meeting, 9 July 2014 15

16. Functionality of E-lane handlers From GBT: need to decode 8b/10 or HDLC data Collect 2, 4, 8, 16 bits per 40MHz clock Align using comma symbols Decode 8b/10b, discard comma, Start-of-Packet, End-of-Packet Add 1KB block headers and “chunk” trailers Optionally verify checksum To GBT: need to encode 8b/10b or HDLC data Read from FIFO Encode (8b/10b(?) or HDLC) 2 bytes at a time from FIFO Chop into 2, 4, 8, 16 bits per 40MHz clock Probably run at 80MHz to get extra clocks for headers/trailers L. LevinsonUpgrades of detector readout meeting, 9 July 201416

17. Clock recovery The 2x40MHz clock comes from the TTC link, i.e. a recovered clock. Probably OK for GTH’s in the same GTH block of 4 transceivers ??? – Can it be distributed to the other blocks? – Does the recovered clock have sufficiently low jitter for the other GTH’s? We should probably consider a TTC receiver and jitter remover on an FMC card. – rework the GLIB version so that the FMC is on top of the PCIe mother board instead of extending beyond the PCIe board boundary L. LevinsonUpgrades of detector readout meeting, 9 July 201417

18. Error handling Just beginning the considerable thought that needs to be done here. Two issues: – Discovery of corrupted data – Surviving errors without out loosing event boundaries This requires some effort on the front ends Corruption within a GBT 80-bit word is detected by the GBT’s FEC Hard to believe E-links can be mixed into each other, so expect errors within E-link channels: repeated clocks, lost clocks If E-links use 8b/10b with packet framing codes, fixed length, or variable length with counts, then errors (not all) can be detected. FELIX can flag these errors in the header it sends with a stream’s data L. LevinsonUpgrades of detector readout meeting, 9 July 201418