Firmware Overview and Status Erno DAVID Wigner Research Center for Physics (HU) 26 January, 2016

FLP Server CRU RAM CPU GBT GBTx, GBT-SCA TTS PCIe User (detector specific) Logic FLP DAQ Software FLP DCS Software Detector Data Frames FEE LTU CTP CRU Ctrl. Detector Control Frames NIC NIC/IB to EPN Servers to DCS Servers Common CRU FW/SW Components provided by the CRU team Developed by the Detector and DCS teams Developed by the Detector and O2 teams Extensible by the Detector teams CRU Block Diagram

Main transactions: Delivering the TTS information or Readout Control from CRU to FE Receiving the detector data packets in serial and parallel form from the FE Delivering custom packets from CRU to FE FPGA in parallel form Sending and receiving packets to and from GBTx internal register block Sending and receiving packets to and from GBT-SCA ASIC Different Communication Forms Between the CRU and the FEs SFP+ LTU 40 MHz LHC Clock bit data (Orbit, BC, …) Busy GBT LatOpt. Tx (24/36/48 link) FEE GBTx (ASIC) ASIC or FPGA GBT-SCA (ASIC) I2C, SPI, GPIO, JTAG 80 Mb/s 40 MHz x 2 bit Internal SC 40 MHz x 2 bit External SC 40 MHz LHC Clock + 80 bit trigger (Orbit, BC, …) Trigger or DCS (up to 40 e-links) FEE Data (up to 40 e-links) 40 MHz x 2 bit Internal SC 40 MHz x 2 bit External SC 40 MHz x 80 bit FEE Data 40 MHz LHC Clock... CTP CRU Trigger and Timing Data DCS User (detector specific) Logic TTS Busy PCIe, x16, Gen3, 128 Gbps 40 MHz x 80 bit DCS or GBT Tx GBT Rx GBT Std. Rx (24/36/48 link) DCS DATA

GBT-Frame vs Different Communication Forms 4 bit Header 2 bit IC 2 bit80 bit User Data 32 bit FECEC GBTx reg. access (e.g. TPC) GBT-SCA comm. GBT-SCA comm. (e.g. TPC) Custom packet for FE FPGA (e.g. ITS) GBT Downlink (CRU -> FE) 80 bit TTS information (e.g. TOF) Detector specific readout control (e.g. TPC) 4 bit Header 2 bit IC 2 bit80 bit User Data 32 bit FECEC GBTx reg. access (e.g. TPC) GBT-SCA comm. GBT-SCA comm. (e.g. TPC) Single well formed CDH/SDH packet (e.g. TOF) GBT Uplink (FE -> CRU) Multiple detector data packets in serial form (e.g. MCH) MHz Raw front- end data in Wide Bus Mode (e.g. TPC)

GBT Tx 4.8 Gbps N x MHz User Logic PCIe BAR Avalon- MM Master N x 2 bit PCIe Endpoint 0 PCIe 10G PON 9.6 Gbps TTS EMU GBT Payload Generator GBTx Packet Sender GBT-SCA Packet Sender Interconnect 10G PON GBT CRU Downstream Path (FLP -> FE) 1. Common CRU Firmware 80 bit trigger forwarding Packet sender 2. TPC, MCH, MID FE readout-control Can be upgraded for DMA Downlink

GBT Rx 4.8 Gbps N x MHz PCIe BAR Avalon- MM Master N x 2 bit PCIe Endpoint 0, 1 PCIe 10G PON 9.6 Gbps TTS EMU Raw GBT Payload Recorder GBTx Packet Receiver GBT-SCA Packet Receiver Interconnect PCIe DMA Uplink 0, 1 User Logic 10G PON GBT CRU Upstream Path (FE -> FLP) 1. Common CRU Firmware Packet receiver 2. TPC SAMPA raw data proc., CF 3. MID Custom serial packet decoder

cru_top_24.vhd pcie_wrapper.vhd QSYS PCIe Endpoint 0 QSYS PCIe Endpoint 1 7 Current CRU Firmware SoC Architecture user_logic.vhd tts_wrapper.vhd dcs_wrapper.vhd Avalon-ST Source (xxx MHz, 256 bit) Avalon-MM Slave (xxx MHz, 32 bit) gbt_24_top.vhd Avalon-MM Slave (xxx MHz, 32 bit) GBT-SCAGBTx 40 MHz, 24 x 80 bit 40 MHz 24 x 4 bit QSYS design with 24 x GBT + Pattern generator and Checker QSYS design with 1 x 10G PON + Pattern Generator and Checker User Logic 40 MHz 24 x 80 bit 40 MHz 24 x 4 bit 40 MHz 184 bit 40 MHz 16 bit Avalon-MM Master Bridge (250 MHz, 32 bit) Avalon-ST Sink (250 MHz, 256 bit) Avalon-MM Master Bridge (250 MHz, 32 bit) Avalon-ST Sink (250 MHz, 256 bit) Avalon-MM Master Bridge (250 MHz, 32 bit) 10G PON GBT 0 GBT 23 PCIe 0 PCIe MHz 256 bit 250 MHz 32/64 bit

FLP Server LinuxPCIe40 Board Arria 10 FPGA QSYS PCIe Endpoint 0 bar_x_master Avalon-MM Master 250 MHz x 32/64 bit 8 CRU Firmware Slow Control SoC Architecture QSYS Interconnect (250 MHz x 32/64 bit) GBT-SCA [0..47] x 2 40 MHz I2C CRU API O2 Software DCS Software ??? Software [0..47] x 2 40 MHz GBTx ASIC [0..47] x 2 40 MHz Si 5338 MiniPOD 0 MiniPOD 7 ? Arria 10 Firmware Flash... SFP+ User Logic TTS, GBT, … (core modules) Custom Interface

9 PCIe40 CRU Firmware Git Repository Status Aim: Provide the core features in a well separated way (everything is under /core directory) The detector specific User Logic interacts with core features through a well defined interface Heavy emphasis on command line scripting for easy integration with automated build system Status: Git repository: You can check out and build the core modules + TPC user logic placeholder git clone mkdir syn; cd syn quartus_sh --script../pcie40cru/tpc/firmware/scripts/quartus/build.tcl -g../pcie40cru The missing features implemented as dummy logic to avoid dropping by optimization. Ongoing development in the /core directory, but we should be able to work in parallel with the TPC and any other detector team through the User Logic Interface

10 Status of the Firmware Modules TTS – the 10G PON downlink development just started, the emulator is developed by CTP GBT – ongoing x24/x36/x48 development, the x6 design is tested with VLDB over single GBT links PCIe – the two x8 endpoint is visible from Linux, the PCIe DMA controller (not yet started), PCIe Uplink packet aggregator (not yet started) GBTx – not yet started GBT-SCA – a basic SCA IP (with polling) is in the early phase of development CRU Control – not yet started Raw GBT Payload Recorder – not yet started Packet Based Communication for Common CRU Firmware – not yet started Data Generator – not yet started Multiplexers, Demultiplexers, Proxies, Arbitrators - not yet started