The CMS Modular Track Finder (MTF7) Trigger Board D. Acosta, G. Brown, A. Carnes, M. Carver, D. Curry, G.P. Di Giovanni, I. Furic, A. Kropivnitskaya, A. Madorsky, D. Rank, C. Reeves, B. Scurlock, S. Wang University of Florida/Physics, Gainesville, FL, USA M. Matveev, P. Padley Rice University, Houston, TX, USA

CMS Endcap Muon System TAMU meeting A. Madorsky φ θ, η η coverage: 1.2 to 2.4

Muon Trigger structure rework TAMU meeting A. Madorsky Endcap TF Overlap TFBarrel TF - Overlap TF is now separated from Endcap and Barrel

CMS Endcap Muon Trigger Each of two Endcaps is split into 6 sectors, 60° each Each sector is served by one Sector Processor (SP) Total 12 SPs in the entire system CMS trigger requires us to identify distinct muons Each SP can build up to 3 muon tracks per BX Trigger sector 60˚ TAMU meeting A. Madorsky

Endcap Muon Trigger upgrade Trigger information  Wiregroup patterns  Strip hits Muon Endcap Trigger sector (60°) Upgraded Port Cards One per station 1/6 filtering Trigger primitives 2 per chamber 18 per station 90 total 18 primitives per station 90 total Fibers (~100 m) Upgraded Sector Processor  Complete 3-D tracks assembled from primitives  Up to 3 tracks per BX TAMU meeting A. Madorsky

Motivations for upgrade  Current Endcap Trigger is adequate for LHC luminosity before the upgrade  The following improvements are needed for the upgrade:  Transverse Momentum (Pt) assignment  Final Pt assignment is currently done with 2MB LUT  Address space is already over-saturated  Need bigger Pt assignment memory  Trigger primitives bandwidth  With luminosity upgrade, we expect ~7 Trigger primitives per Sector per BX (average).  Current system selects only 3 best primitives in each sector  Inadequate for the upgrade  Need to import preferably all primitives on each BX  18 per sector  Also need to import other data (Resistive Plate Chambers)  The above means:  Higher input bandwidth  Bigger FPGA TAMU meeting A. Madorsky

Block diagram Optical plant (fanouts and splitters) From MPCs core fibers, 8 cores used in each. 90 trig. primitives per 60° sector 3.2 Gbps From RPC Up to 216 fibers at 1.6 Gbps (may be concentrated to higher bandwidth and fewer fibers) Sector Processors 12 units 60° sector each Muon Sorter 8 Best Muons to Global Muon Trigger To Overlap Track Finder Best 3 Muons in each sector TAMU meeting A. Madorsky

uTCA chassis SP MSMS  Sector Processor (SP)  occupies 2 uTCA slots  12 units in system  Muon Sorter (MS)  hardware identical to SP  1 unit in system  All chassis use AMC13 (designed by Boston University)  clocking and DAQ  3 units  Plan to control boards via PCI-express  Will make compatible with IPbus as well Chassis #1 Chassis #2 Chassis #3 TAMU meeting A. Madorsky

Hardware prototype 2012 prototype  based on Virtex-6 FPGA  Modular design  Makes future partial upgrades easier TAMU meeting A. Madorsky Optical module Custom backplane Core logic module Optical module Core logic module Pt LUT module Custom backplane

Virtex-6 Core logic module TAMU meeting A. Madorsky Custom backplane connector Core logic FPGA Control FPGA uTCA connector Control FPGA JTAG PT LUT module connector FMM connector SD card connector MMC USB console MMC JTAG MMC CPU Estimated power consumption: ~50 W (assuming FPGAs nearly full) PT LUT mezzanine not included 1Gb FLASH Main FPGA firmware storage MMC = Module Management Controller

Virtex-6 Core logic module: Features  Serial I/O:  53 GTX receivers (up to 4.8 Gbps)  8 GTH receivers (10 Gbps)  12 GTX transmitters (up to 4.8 Gbps)  2 GTH transmitters (10 Gbps)  MMC – Wisconsin design  See this link for detailslink  Configuration memory for Core FPGA:  PC28F00AP30EFA  1 Gb parallel FLASH  Can be used to store any other information (in addition to firmware)  Permanent configuration settings  Multiple firmware versions  SD card slot  Can also be used to store Core FPGA firmware and settings  Fast Monitoring (FMM) connector  Compatible with the current FMM system  Control interfaces:  PCI express  IPbus TAMU meeting A. Madorsky

Optical module TAMU meeting A. Madorsky uTCA connector Custom backplane connector Backplane redrivers Optical receivers (2 out of 7 installed) MMC Optical transmitters (2 out of 3 installed)

Optical module  Receivers:  7 12-channel RX  Avago’s AFBR-820BEZ  84 RX channels  Transmitters:  3 12-channel TX  Avago’s AFBR-810BEZ  28 TX channels ( )  All of them 10 Gbps parts  Not enough space on front panel to accommodate all  TX parts located inside  connect with short fibers to MPO fiber couplers on front panel  Tight but enough space to fit couplers on top of AFBR-820 parts.  Receivers are on front panel to minimize count of fiber-to-fiber transitions for inputs  Control:  Wisconsin MMC design, no FPGA  Compatible with future Virtex-7 design of Core logic board TAMU meeting A. Madorsky

Glue logic FPGA (Spartan-6) Base board connector RLDRAM3 memory 16 chips, 8 on each side (clamshell topology) Total size: 512M x 18 bits ≈ 1GB Upgrade possible to 2 GB with bigger RLDRAM3 chips (no board redesign) DC-DC converters Clock synthesis and distribution PT LUT module TAMU meeting A. Madorsky

Optical communication Gbps  47 input channels  Transmission from:  Loopback  Muon Port Card  Earlier VME prototype (2010)  For MPC and VME prototype clock was synchronized with VME crate  Twisted pair LVDS connection to uTCA Gbps  6 input channels  Asynchronous clock  Transmission from:  Loopback  Earlier 10Gbps prototype (2006) Results  Zero errors for hours TAMU meeting A. Madorsky Eye Gbps. GTH receiver input.

PT LUT tests Parameters:  RLDRAM clock : 200 MHz  Address & control: 200 Mbps each bit  Data: 400 Mbps each bit  RLDRAM can tolerate up to ~1GHz clock. However:  Hard to implement in FPGA  Needed for burst-oriented applications mostly  Does not change latency for random address access  Lower clk F  lower power consumption Tests performed (random data, full 1GB space):  Writing into consecutive addresses  Reading from consecutive addresses  Reading from random addresses No errors detected  Except soldering defects in one RLDRAM chip TAMU meeting A. Madorsky

PCI Express tests: Setup TAMU meeting A. Madorsky PC adapter Motherboard AMC113 (uTCA PCIE adapter) 2012 Prototype Fiber (up to 50 m) Multiple PC adapters tested Best results: HIB35-x4 Also least expensive VendorCard VadatechPCI113 SamtecPCIEA OneStopSystemsHIB35-x4

PCI express performance  Theoretical PCIe performance (generation 2, single lane):  Link bitrate: 5 Gbps  After 8b/10b encoding: 4 Gbps  After PCIe packet overhead: ~3.56 Gbps  On top of that:  Software  Hardware overhead of a particular chipset on host system  Hardware overhead of your device  Our test design has very small overhead  Results of performance tests at UF:  Sustained performance, all overheads included  5 meter fiber:  Reading: 2.4 Gbps  Writing: 2.88 Gbps  5 0 meter fiber:  Reading: 2.3 Gbps  Writing: 2.88 Gbps TAMU meeting A. Madorsky

Plans for Virtex-7 design Core logic module FPGAs:  Core logic: XC7VX690T-FFG1927 or XC7VX550T-FFG1927  Control: XC7K70-FB676 Inputs:  80 Virtex-7 GTH links (10 Gbps) directly to Core FPGA  Minimal latency  All available receivers are designated for trigger data  Maximum flexibility  4 additional Kintex GTX links (6.4 Gbps) to Control FPGA  Delivered to Core FPGA via parallel channel  Longer latency Outputs:  28 Virtex-7 GTH links (10 Gbps) Control:  PCI express Gen 2, 2 lanes  Ipbus Status: PCB layout nearly done TAMU meeting A. Madorsky

Core logic module FPGA interconnections TAMU meeting A. Madorsky XC7VX690T or XC7VX550T XC7K70 PT LUT connector 28 outputs 10 Gbps 80 inputs 10 Gbps 4 inputs 10 Gbps IPbus PCI express 2 lanes Parallel data exchange channel (control, DAQ control, 4 extra links) Custom backplane connector uTCA backplane connector DAQ control (RX) 4 TX to AMC13 DAQ TX

Conclusions  Virtex-6 based prototype built and tested  53 inputs up to 4.8 Gbps  8 inputs at 10 Gbps  RLDRAM3-based Pt LUT (1GB)  PCI express  IPbus  Virtex-7 base board prototype is in its final design phase TAMU meeting A. Madorsky

Backup

CSC trigger data sharing Endcap and Overlap processors ME1 ME2,3, Subsector 1 Subsector 2 Overlap TF Endcap TF Multiple chambers have to be shared between Endcap and Overlap TFs (shown in yellow)

TAMU meeting A. Madorsky CSC trigger data sharing Neighbor sector ME1 ME2,3, Subsector 1 Subsector 2 Neighbor sector TF Neighbor sector TF Several chambers shared with neighbor Sector Processor for better sector overlap coverage (shown in pink) Most of the chambers shared between 2 SPs Some chambers shared between 4 SPs

Optical components TAMU meeting A. Madorsky MPO Fanout 2-way splitter 4-way splitter Components from Fibertronics

Optical plant (one sector) TAMU meeting A. Madorsky MTP connectors (inputs) LC-LC adapters MTP connectors (outputs) Splitters MTP fanouts Slack spool Slack spool 19” 1U Rack-mount Enclosure * Components shown not in full quantities