May 23 rd, 2003Andreas Kugel, Mannheim University1 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn ATLAS Trigger/DAQ Read-Out-Buffer (RobIn) Prototype Outline Read-Out Environment RobIn (HW, SW) Status + Future Work on behalf of the ATLAS TDAQ Dataflow Group

May 23 rd, 2003Andreas Kugel, Mannheim University2 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn DATAFLOWDATAFLOW EVBEVB R/O S Y S T E M HLTHLT LV L1 D E T R/O ROD LVL2 TriggerDAQ 2.5  s ~ 10 ms 40 MHz 75 kHz ~2 kHz ~ 200 Hz Calo MuTrCh Other detectors ROB IOM SFI SFO RRC RR EBN EFN FE Pipelines Read-Out Drivers ROD-ROB Connection Read-Out Buffers ROD-ROS Merger I/O Manager Dataflow Manager Sub-Farm Input Sub-Farm Output Event Filter N/work ROIB L2P L2SV L2 N Event Filter DFM EFP RoI Builder L2 Supervisor L2 N/work L2 Proc Unit RoI RoI data = 2% RoI requests Lvl2 acc = ~2 kHz Event Building N/work ~ sec Lvl1 acc = 75 kHz 40 MHz 120 GB/s ~ 300 MB/s ~3+3 GB/s Event Filter Processors 120 GB/s ~3 GB/s EFacc = ~0.2 kHz T/DAQ Architecture ROS

May 23 rd, 2003Andreas Kugel, Mannheim University3 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Total of ~1600 Read-Out-Links (RRC/ROLs) ROS Baseline Gb E Concentrator switch ROLs … … Gb E ROBi n … RoBin Gb E PCI Prototype RobIn accepts 2 ROLs per Board Target Implementation accepts 4 ROLs Gb E I/O Optimisation still to be done

May 23 rd, 2003Andreas Kugel, Mannheim University4 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn ROS IOM (PC based) = Linux Scheduler = Thread = Process on PC Requests (L2, EB, Delete) Request Queue RobIn’s Request Handlers Control, error Trigger Fragments PCI GbE

May 23 rd, 2003Andreas Kugel, Mannheim University5 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Direct Connection (No IOM) Requests (L2, EB, Delete) RobIn’s Fragments GbE...

May 23 rd, 2003Andreas Kugel, Mannheim University6 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn RobIn Requirements Summary Functions: RECEIVE – BUFFER – DELIVER – RELEASE Multiple S-LINK Input, max. 160MB/s per link, XOFF Capability Embedded version due to mechanical constraints Local buffer to compensate trigger latency Efficient buffer management PCI and GbE Interfaces for control and data requests In-situ firmware upgrades, Test facilities, JTAG Optional stand-alone mode (PCI disabled) Standard Parameters: 1kB fragment size (avg.), 75 kHz input event rate ~ 3 kHz L2 request rate, ~ 3 kHz EB rate

May 23 rd, 2003Andreas Kugel, Mannheim University7 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Mezzanine Connector SD-RAM Buffer Virtex2 FPGA PLX 64Bit PCI Interface GbE Interface PowerPC Module Not shown: ROL Interface (TLK2501) Rapid Prototyping: MPRACE

May 23 rd, 2003Andreas Kugel, Mannheim University8 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn RobIn Block Diagram, Component View Hardware assistance for input provided by FPGA (DF-CORE) Management task suitable for CPU (AUX-CORE) Main data-path doesn't touch CPU

May 23 rd, 2003Andreas Kugel, Mannheim University9 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn RobIn Block Diagram, Functional View CPU controls common functions and serialises access to shared components (MAC, PCI) FPGA provides parallel implementation of ROB-”Slices” “Slice“ Part (1..n) Common Part 100MHz 125MHz 66MHz Logical pages ~1kB Manages 64k pages FIFO comms

May 23 rd, 2003Andreas Kugel, Mannheim University10 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn FPGA PCICPU Network Buffers ROL1 ROL2 CTL RobIn 64Bit/66Mhz PCI, 3.3V supply (3V+5V signalling level) 12 Layer PCB, Size 220*107mm, Test Connectors Unit Cost: ~1400€

May 23 rd, 2003Andreas Kugel, Mannheim University11 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Board Control Board supervised by CPLD XC95288XL Programmable Reset Generation Flash-Eeprom (for FPGA) programming via serial (JTAG-like) protocol External access via connector Access for one of PLX, PPC, FPGA FPGA power-on configuration (slave SelectMap) Path to external JTAG connector for factory programming and test Few LEDs Global System Clock: 66MHz for CPU, MAC, PCI, FPGA

May 23 rd, 2003Andreas Kugel, Mannheim University12 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Embedded S-LINK ● Unidirectional data transfer, with XOFF on return channel ● Pluggable Optical Transceiver ● TLK Gbit/s SerDes, 16 Bit MHz Link protocol engine in FPGA (CERN core) 2.5GBit/s => 160MB/s forward, XOFF backward, links status ROS Format Start-of-Packet Control word Header Data, with L1ID End-of-Packet Control Word Extra: Link can be re-configured as LSC => Loopback testing ROL

May 23 rd, 2003Andreas Kugel, Mannheim University13 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Cover Trigger Latency of ~ 10ms => 1.6MB required 64MB SD-RAM => 400ms Good SD-RAM performance requires bursts: – Block oriented Dual-Port Emulation provides sufficient BW Buffer

May 23 rd, 2003Andreas Kugel, Mannheim University14 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn PLX PCI9656 PCI Bridge 64 bit PCI, up to 66MHz 32 bit local 66MHz, Multiplexed A/D FIFO-buffered PCI-DMA, controlled from PLX or local master (FPGA) for Request Messages Fifo-buffered “Direct Master Mode” (FPGA writes into host memory) for Response Messages 1. Fragment buffers are pre-allocated in host memory 2. Physical buffer address provided to RobIn with request 3. Data transferred via “Direct Master” into host memory, first data word (offset 0) set to “1” to indicate activity 4. End-of-transfer signalled via start-of-header marker (offset 0) written after data block completed PCI

May 23 rd, 2003Andreas Kugel, Mannheim University15 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Current: With Master Mode DMA, Old: No Master Mode DMA Required for event data per ROL: ~ 10MB/s PCI Performance

May 23 rd, 2003Andreas Kugel, Mannheim University16 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Intel IXF1002 GbE MAC Separate 32 bit RX, TX 66MHz 16 bit control interface VLAN support 2kB TX-, 4kB RX-FIFO Marvell Alaska 88E1011S GbE PHY Dual Media Interface for Copper and Fibre RJ45 Connector Pluggable Optical transceiver Network I/F

May 23 rd, 2003Andreas Kugel, Mannheim University17 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Xilinx Virtex-2 XC2V1500-FF896 (XC2V2000 possible) Recent 0.15µ FPGA technology, 1.5V core voltage Logic: LUTs 48 blocks of embedded memory (18kbit), 16 used for FIFOs Hardware Multipliers (not used) 8 Clock managers for DLL, Frequency Synthesis, 6 used Up to 8 global on-chip clocks, 5 used On-chip driver termination (DCI) for high-speed signals (ROL) 528 I/O pins, all in use (including TEST connectors) Easily reconfigurable for different scenarios FPGA

May 23 rd, 2003Andreas Kugel, Mannheim University18 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn IBM PowerPC PPC405CR ( MHz) I+D Cache (16/8 kB) 133MHz External 66MHz, DMA 64 MB SD-RAM Memory (same as for Buffer) Boot ROM: 8MB Flash Serial Interface (debug terminal) I2C Interface (EEPROM for local configuration) FPGA resources memory mapped to external bus < 1W Power Consumption Code compatible to PPC405GPr (400MHz), PPC405 in Virtex-2Pro (400MHZ) Processor

May 23 rd, 2003Andreas Kugel, Mannheim University19 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Tools GCC, GDB + JTAG Debugger OS possible (Linux, VxWOrks) ROM resident Monitor + Loader (128kB): Peek(), Poke(), Printf(), Getc(), Malloc(), Free(), LoadApp() Application program Page-level buffer manager for 64k Pages (75 kHz): Used Page FIFO, Free Page Fifo + Free Page Stack L1ID Hashing, Page Management List Request Handling (~7kHz): Input Queues, Output DMA Network Message Translation: Raw Socket Format Operational monitoring Local Configuration (e.g. number of ROLs, Media selection) Application Coding in “C” => simplify code migration into FPGA Small Code Footprint, < 16kB for core functions expected Software

May 23 rd, 2003Andreas Kugel, Mannheim University20 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Processor: Access to Resources Management Memory FPGA

May 23 rd, 2003Andreas Kugel, Mannheim University21 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Hardware 10 RobIn Boards available since March 03 PCI Master Mode: √ Buffer Memories: √ Processor: √ Network RX, TX/DMA: √ ROL LDC Core: √ ROL Handling and Page Manager: Simulation OK Very close to put everything together Status

May 23 rd, 2003Andreas Kugel, Mannheim University22 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Software ROM Monitor + Terminal: √ Application Code Page Management: √ Request Handling: 80% PCI Request / Response Scheme: √ Network Translation: 60% Monitoring: 60% ROS Software Device Library + Driver √ Fragment Processing API √ => ROS Baseline Direct Connect: No special SW required Status cont’

May 23 rd, 2003Andreas Kugel, Mannheim University23 Mannheim University – FPGA Group Real Time Conference 2003, Montréal ATLAS RobIn Future Work June 03:RobIn Application and VHDL complete Sept 03:Performance Testing and Optimisation, incl. Bus/Network I/O Optimisation Dec 03:Design evolution: ROL media selection (fiber / copper), number of ROLs, component modifications, Packaging May 04:Final prototypes ready