01/06/09 15:26 Electronics for CMS Endcap Muon Level-1 Trigger System Phase-1 and HL LHC Upgrades D. Acosta, A. Batinkov, V. Barashko, P. Bortignon, A.

Slides:



Advertisements
Similar presentations
Phase-0 topological processor Uli Schäfer Johannes Gutenberg-Universität Mainz Uli Schäfer 1.
Advertisements

Level-1 Topology Processor for Phase 0/1 - Hardware Studies and Plans - Uli Schäfer Johannes Gutenberg-Universität Mainz Uli Schäfer 1.
An ATCA and FPGA-Based Data Processing Unit for PANDA Experiment H.XU, Z.-A. LIU,Q.WANG, D.JIN, Inst. High Energy Physics, Beijing, W. Kühn, J. Lang, S.
ZTF Interconnecting Scheme Stephen Kaye
CHEP March, B. Scurlock, University of Florida1 D. Acosta, V. Golovtsov, M. Kan, A. Madorsky, B. Scurlock, H. Stoeck, L. Uvarov, S.M. Wang University.
The Track-Finding Processor for the Level-1 Trigger of the CMS Endcap Muon System D.Acosta, A.Madorsky, B.Scurlock, S.M.Wang University of Florida A.Atamanchuk,
Emulator System for OTMB Firmware Development for Post-LS1 and Beyond Aysen Tatarinov Texas A&M University US CMS Endcap Muon Collaboration Meeting October.
Status and planning of the CMX Philippe Laurens for the MSU group Level-1 Calorimeter Trigger General Meeting, CERN May 24, 2012.
An Asynchronous Level-1 Tracking Trigger for Future LHC Detector Upgrades A. Madorsky, D. Acosta University of Florida/Physics, POB , Gainesville,
Leo Greiner IPHC testing Sensor and infrastructure testing at LBL. Capabilities and Plan.
Printed by Topical Workshop on Electronics for Particle Physics TWEPP-08, Naxos, Greece / September 2008 MEZZANINE CARDS FOR.
Preliminary Design Review: Hub Implementation Dan Edmunds, Wade Fisher, Yuri Ermoline, Philippe Laurens Michigan State University 01-Oct-2014.
Pulsar II Hardware Overview Jamieson Olsen, Fermilab 14 April 2014
CSC Endcap Muon Port Card and Muon Sorter Upgrade Status May 2013.
CPT Week, April 2001Darin Acosta1 Status of the Next Generation CSC Track-Finder D.Acosta University of Florida.
Design Considerations for an Upgraded Track-Finding Processor in the Level-1 Endcap Muon Trigger of CMS for SLHC operations Sep TWEPP09 D. Acosta,
Muon Port Card, Optical Link, Muon Sorter Upgrade Status M.Matveev Rice University December 17, 2009.
Annual Review Cern -June 13th, 2006 F. Loddo I.N.F.N. Bari RPC Electronics: Technical Trigger Flavio Loddo I.N.F.N. Bari On behalf of the RPC-Trigger group.
Upgrade of the CSC Endcap Muon Port Card Mikhail Matveev Rice University 1 November 2011.
CSC Endcap Muon Port Card and Muon Sorter Status Mikhail Matveev Rice University.
Status and planning of the CMX Wojtek Fedorko for the MSU group TDAQ Week, CERN April , 2012.
CSC ME1/1 Upgrade Status of Electronics Design Mikhail Matveev Rice University March 28, 2012.
ATLAS Trigger / current L1Calo Uli Schäfer 1 Jet/Energy module calo µ CTP L1.
MTF7 hardware status Alex Madorsky for UF group. Muon Trigger structure rework A. Madorsky2 Endcap TF Overlap TFBarrel TF - Overlap TF is now.
SRS Trigger Processor Option Status and Plans Sorin Martoiu (IFIN-HH)
Electronics Review, May 2001Darin Acosta1 OPTICAL SP 1 Muon Sorter 3  / port card 3  / sector ME1 ME2-ME3 ME4 SR DT TF SP From CSC Port Cards MS MB1.
Upgrade of the CSC Endcap Muon Port Card with Spartan-6 FPGA Mikhail Matveev Rice University 30 April 2012.
Consideration of the LAr LDPS for the MM Trigger Processor Kenneth Johns University of Arizona Block diagrams and some slides are edited from those of.
The CMS Modular Track Finder (MTF7) Trigger Board D. Acosta, G. Brown, A. Carnes, M. Carver, D. Curry, G.P. Di Giovanni, I. Furic, A. Kropivnitskaya, A.
29/05/09A. Salamon – TDAQ WG - CERN1 LKr calorimeter L0 trigger V. Bonaiuto, L. Cesaroni, A. Fucci, A. Salamon, G. Salina, F. Sargeni.
E. Hazen -- Upgrade Meetings1 AMC13 Project Development Status E. Hazen, S.X. Wu - Boston University.
E. Hazen1 MicroTCA for HCAL and CMS Review / Status E. Hazen - Boston University for the CMS Collaboration.
E. Hazen -- ACES ACES 2011 MicroTCA in CMS E. Hazen, Boston University for the CMS collaboration.
E. Hazen -- Upgrade Week1 AMC13 Project Status E. Hazen - Boston University for the CMS Collaboration.
E. Hazen -- Upgrade Week1 HCAL uTCA Readout Plan E. Hazen - Boston University for the CMS HCAL Collaboration.
E. Hazen -- Upgrade Meetings1 AMC13 Project DAQ and TTC Integration For MicroTCA in CMS Status Report E. Hazen - Boston University for the CMS.
E. Hazen -- HCAL Upgrade Workshop1 MicroTCA Common Platform For CMS Working Group E. Hazen - Boston University for the CMS Collaboration.
E. Hazen1 AMC13 Status E. Hazen - Boston University for the CMS Collaboration.
E. Hazen1 AMC13 Project Status E. Hazen - Boston University for the CMS Collaboration.
OTMB Development and Upgrade Plan for LS2
CMS Trigger Improvements towards Run II
Slow Control and Run Initialization Byte-wise Environment
Slow Control and Run Initialization Byte-wise Environment
AMC13 Project Status E. Hazen - Boston University
DAQ and TTC Integration For MicroTCA in CMS
AMC13 T1 Rev 2 Preliminary Design Review E. Hazen Boston University
ATLAS calorimeter and topological trigger upgrades for Phase 1
E. Hazen - Back-End Report
xTCA in CMS Magnus Hansen Thanks to many CMS collaborators
AMC13 Status Report AMC13 Update.
Future Hardware Development for discussion with JLU Giessen
The Ohio State University
L1Calo Phase-1 architechure
Update on CSC Endcap Muon Port Card
Large Area Endplate Prototype for the LC TPC
Upgrade of the CSC Endcap Muon Port Card and Optical Links to CSCTF
Optical data transmission for
Muon Track-Finder Trigger
LATOME LAPP Nicolas Dumont Dayot on behalf of the LAPP team
CMS EMU TRIGGER ELECTRONICS
ATLAS L1Calo Phase2 Upgrade
MicroTCA Common Platform For CMS Working Group
MicroTCA Common Platform For CMS Working Group
Read-out of High Speed S-LINK Data Via a Buffered PCI Card
A microTCA Based DAQ System for the CMS GEM Upgrade
xTCA in CMS Magnus Hansen Thanks to many CMS collaborators
Regional Cal. Trigger Milestone: Major Production Complete
Regional Cal. Trigger Milestone: Production & Testing Complete
CSC Trigger Muon Port Card & Sector Processor in production
Sector Processor Status Report
Presentation transcript:

01/06/09 15:26 Electronics for CMS Endcap Muon Level-1 Trigger System Phase-1 and HL LHC Upgrades D. Acosta, A. Batinkov, V. Barashko, P. Bortignon, A. Brinkerhoff, A. Carnes, D. Curry, I. Furic, A. Jyothishwara, J.F. Low, A. Madorsky, N. Pratap Ghanathe, Y. Xia University of Florida/Physics, Gainesville, FL, USA P. Cao, L. Chen, Z. Liu, C. Wang Chinese Academy of Sciences, Institute of High Energy Physics, Beijing, China M. Matveev, P. Padley, J. Rorie Rice University, Houston, TX, USA E. Juska Texas A&M University, College Station, TX, USA K. Bunkowski, A. Kalinowski, K. Kierzkowski, M. Konecki, W. Oklinski University of Warsaw, Warsaw, Poland A. Byszuk, K. Pozniak, W. Zabolotny University of Warsaw, Warsaw, Poland and Warsaw University of Technology, Warsaw, Poland T. Gorski, A. Svetek, J. Tikalsky, M. Vicente University of Wisconsin-Madison, Madison, WI, USA CMS OO Transition. David Stickland

CMS Endcap Muon System φ θ, η η coverage: 1.2 to 2.4 CMS cutout with Endcaps glowing θ, η η coverage: 1.2 to 2.4 INSTR 2017 A. Madorsky

CMS Muon Level-1 Trigger structure 01/06/09 15:26 CMS Muon Level-1 Trigger structure Barrel Muon Track Finder (BMTF) Overlap Muon Track Finder (OMTF) Endcap Muon Track Finder (EMTF) Cathode Strip Chambers (CSC) Resistive Plate Chambers (RPC) L1 Trigger Upgrade TDR CERN-LHCC-2013-011 CMS-TDR-012 (2013) INSTR 2017 A. Madorsky CMS OO Transition. David Stickland

CMS Endcap Muon L-1 Trigger Each of two Endcaps is split into 6 sectors, 60° each Each sector is served by one Sector Processor (SP) Total 12 SPs in the entire system CMS trigger requires us to identify distinct muons Each SP can build up to 3 muon tracks per BX Challenges unique to Endcap: Very high background near beam pipe Non-uniform magnetic field Trigger sector 60˚ Brief introduction INSTR 2017 A. Madorsky

LS1 upgrade goals LS1 = Long Shutdown 1 (2013-2015) Feature Before LS1 (legacy system) After LS1 upgrade Imported CSC trigger primitives Only up to 3 segments per station per sector per BX - bottleneck Up to 15 best from each 60˚ sector per BX All available – up to 18 per station per sector per BX, up to 90 primitives from each 60˚ sector per BX Imported RPC trigger primitives None All available – up to 72 primitives from each 60˚ sector per BX (firmware still in progress) Sector overlap processing Each sector receives trigger primitives from 10˚ of the neighboring sector CSC: up to 18 primitives RPC: up to 12 primitives Transverse momentum (Pt) assignment 2 MB LUT 1 GB LUT provides much more flexibility for physics algorithm INSTR 2017 A. Madorsky

Upgraded system block diagram Optical plant (fanouts and splitters) Endcap Muon Track Finder (EMTF) Global Muon Trigger CSC trigger primitives Best 3 Muons in each sector RPC trigger primitives To OMTF INSTR 2017 A. Madorsky

Production hardware – MTF7 Modular Track Finder 7 µTCA standard Adopted by CMS for LS1 upgrades Based on Virtex-7 FPGA Modular design Logic module (left): Main FPGA – Virtex-7 Control FPGA – Kintex-7 Control interfaces: PCI Express, Gen2, 2 lanes IPbus Optical module (right): 7 receivers 12 channels each, up to 10 Gbps 84 RX channels total (currently used: 56) 2 transmitters 12 channels each, up to 10 Gbps 24 TX channels total (currently used: 1) Logic and Optical modules connect via custom 10Gbps backplane section PT LUT module (mezzanine) Based on Reduced Latency DRAM (RLDRAM 3) 1 G x 9 bits of space Also used by OMTF INSTR 2017 A. Madorsky

CSC trigger data sharing ME1 ME2,3,4 (3 stations) Subsector 1 Subsector 2 Color Native EMTF OMTF Neighbor EMTF Neighbor OMTF Split (ways) Chamber count  None 11 2 25 4 5 INSTR 2017 A. Madorsky

Optical patch panel 12 EMTF sectors + output One EMTF sector panel 1U rack-mount box 14 fan-outs 29 2-way splitters 5 4-way splitters 12 EMTF sectors + output INSTR 2017 A. Madorsky

Firmware Firmware significantly reworked relative to legacy system up to 108 input CSC primitives / BX up to 84 input RPC primitives / BX Legacy system: up to 15 primitives / BX Nearly 13 times more than legacy! φ patterns are used to find straightest paths with CSC data only 12 best φ patterns are selected Input primitives are matched to patterns RPC data used to complement missing CSC primitives Due to longer RPC latency Θ path verified to be straight, out-of-line primitives removed Δφ and Δθ between primitives calculated Best three tracks selected and fed to Pt assignment LUT INSTR 2017 A. Madorsky

φ pattern logic Station ME2 is considered a “key” station Straightest patterns have maximum resolution Lower resolution for larger sagitta patterns Pattern quality depends on: Number of stations with hits Pattern straightness Pattern quality is used to select best patterns INSTR 2017 A. Madorsky

Performance plots INSTR 2017 A. Madorsky

Future Upgrades

New trigger data sources Challenges: More RPC chambers New GEM chambers Endcap (Proposed scope): Cathode Strip Chambers (CSC), RPC, plus GEM (ME0, GE1/1, GE2/1), and RPC (RE3/1, RE4/1) INSTR 2017 A. Madorsky

Adding new detectors Inst. luminosity after upgrade: 5*1034 cm-2s-1 Pileup: 200 More chambers offer more hits per track Better track parameter measurement Better rate reduction More chambers improve robustness if: A chamber is dead Too much pileup INSTR 2017 A. Madorsky

Input bandwidth per 60° sector Source Link count, current Link count after upgrades Bit rate, current, Gbps Bit rate after upgrades, Gbps CSC 49 3.2 RPC (existing) 7 10 RPC (upgrade) 3 GE1/1 14 6.4 GE2/1 16 ME0 (GE1/0) 4 9.6 Total 56 links 93 links 226.8 Gbps 487.2 Gbps MTF7 does not have enough input serial links Input bandwidth grows by a factor of ~2.1 FPGA in the current system is ~80% full Replacement FPGA should have at least twice the resources INSTR 2017 A. Madorsky

Advanced Processor R&D 01/06/09 15:26 Advanced Processor R&D Collaboration project with University of Wisconsin-Madison Goals: General-purpose FPGA processing board with expansion capabilities ATCA standard Evaluated by CMS as replacement for µTCA ~100 Optical I/O connections, multi-rate capable Channel count driven by FPGA packaging 1..14 Gbps base range, 25G extended range Mezzanine(s) for user-designed expansion devices, such as: External memory (LUTs) Supplemental FPGA processors Exotic I/O Associative memory Rear-Transition Module (RTM) support Embedded Linux control platform Customizable I/O capability E.g., legacy optical connection support Scalable Cost Model (FPGA-driven) INSTR 2017 A. Madorsky CMS OO Transition. David Stickland

Advanced Processor (APx) Flexible I/O Interface Optical Engines CDR SPF 25G+ Connector TTC Interface Retimers (opt.) Zone 3 Optical MPO Processing FPGA Expansion Mezzanine Digital I/O Base Ethernet (1Gbe) Fast Ethernet (10G/40G) 4-Lanes DAQ? TTC? ADF-type Z2 Optical MPO Optical Engines X- point (4 lanes) Optical MPO Power, IPMI Optical Engines Control Device (ZYNQ) ATCA Z1 4 Lanes Link Aux I/O (ZYNQ/ FPGA/QSFP via Crosspoint) Retimers QSFP RTM (Rear Transition Module) Front Board (ATCA) INSTR 2017 A. Madorsky

AP FPGAs: C2104 Package FPGAs 16G MGTs 30G MGTs Total MGTs K LUTs VU095 32 64 538 VU125 40 80 716 VU160 52 104 926 VU190 1074 VU5P 601 VU7P 788 VU9P 1182 VU11P 96 86 1296 VU13P 1728 XC7VX690T (MTF7, shown for comparison) 80 (11 Gbps) 433 Any of the compatible FPGAs can be used Users select FPGA based on their requirements Count of serial links Logic resources Price range INSTR 2017 A. Madorsky

IPMI Management Controller (IPMC) ATCA uses similar IPMI infrastructure as µTCA For the APx, the IPMC is similar to a Module Management Controller (MMC) expanded responsibilities 244 pin Low Profile MiniDIMM form factor Mounts to main ATCA blade using angled connector INSTR 2017 A. Madorsky

Embedded Processor Module COMExpress Mini Footprint EPMZ7 – Embedded Processor Module (ZYNQ 7000-based) Embedded Linux control point for the APx or similar main ATCA processor board Expansion of the COMExpress- Mini form factor COMExpress—PICMG form factor for putting a PC-style computer on mezzanine Mini—55m×84mm board with a single 220 pin connector COMExpress is a form factor proposal from CMS CERN group for the embedded Linux processor Considered as Phase 2 IPBus replacement GbE PHY DDR Memory ZYNQ 7000 ‘035 or ‘045 40-Pair High Speed Connector 220 pin COMe A/B Connector Refclk Circuitry µSD Boot Flash INSTR 2017 A. Madorsky

Mechanical 3D AP sketch Early attempt, nothing final yet All parts have real dimensions Part of serial links should be routed to RTM, for flexibility (~25%) Multiple FireFly parts can be connected to one MPO adapter to save space Two expansion mezzanines with wide I/O connection to FPGA Heat sink covers FPGA and FireFly parts, shown cut out for clarity DC-DC converters and LDOs located under mezzanines and on reverse side (except bulk DC-DC) INSTR 2017 A. Madorsky

Optical links Evaluating several options Most promising: Manufacturer: Samtec Family name: FireFly Two flavors: Separate parts: 12 × RX, 12 × TX One part: 4 × RX + 4 × TX Mechanically identical Available now: 1 .. 14 Gbps Coming soon: 28 Gbps version Evaluation adapter in production now INSTR 2017 A. Madorsky

PT LUT memory evaluation Considering using regular DDR4 memory modules Fast enough to read from random addresses 3 times in one BX Supported in UltraScale+ FPGAs: 128 GB max RDIMM modules Hardware tests in progress Using evaluation board SODIMM 8 GB INSTR 2017 A. Madorsky

PCB design challenges Accommodating 25Gbps bit rate is not easy Special materials must be used. Examples: Tachyon 100G Megtron 6 Trace geometry becomes critical Even vias can present impedance discontinuity at such bit rates Some PCB features require 3D field solver modelling to get the impedance right Differential via model in HFSS INSTR 2017 A. Madorsky

Firmware & software co-development 01/06/09 15:26 Firmware & software co-development We’ve used a home-made VPP library since 2002 for CSCTF project Single C++ source code, dual use: C++ model compatible with CMSSW Generates human-readable, synthesizable Verilog Problem: EMTF firmware too large for VPP Working on using Vivado High-Level Synthesis (HLS) for firmware development Same philosophy: Except that HLS-generated Verilog is hard to read  So far: Converted nearly all EMTF algorithm into HLS Validated that HLS is compatible with g++ and CMSSW Validated most of the generated Verilog modules in real hardware HW Outputs match HLS outputs Hardware tests ongoing, more modules being tested See TWEPP 2016 talk by N. Ghanathe: https://indico.cern.ch/event/489996/contributions/2210922/attachments/1343543/2024393/TWEPP_Presentation_PRATAP.pdf INSTR 2017 A. Madorsky CMS OO Transition. David Stickland

Conclusions Modular Track Finder 7 designed and constructed for LS1 upgrade Endcap and Overlap Muon L-1 triggers Successfully used for data taking during 2015, 2016 Advanced Processor is under development for future upgrades Phase-1 HL-LHC General-purpose FPGA processing board User expansion capabilities ATCA standard Collaboration project with U. Wisconsin-Madison INSTR 2017 A. Madorsky

Backup INSTR 2017 A. Madorsky

Virtex-7 Logic module Custom backplane connector PT LUT module connector Logic FPGA Control FPGA uTCA connector Connecors for Clock modules MMC CPU MMC = Module Management Controller INSTR 2017 A. Madorsky

Logic module FPGA interconnections PT LUT connector 24 outputs 10 Gbps XC7VX690T Custom backplane connector 80 inputs 10 Gbps 4 TX to AMC13 Parallel data exchange channel (control, DAQ control, 4 extra links) DAQ TX 4 inputs 10 Gbps PCI express 2 lanes XC7K70 IPbus uTCA backplane connector DAQ control (RX) INSTR 2017 A. Madorsky

Logic module Logic module FPGAs: Inputs: Outputs: Control: Logic: XC7VX690T-FFG1927 Control: XC7K70-FB676 Inputs: 80 Virtex-7 GTH links (10 Gbps) directly to Logic FPGA Minimal latency All available receivers are designated for trigger data Maximum flexibility 4 additional Kintex GTX links (6.4 Gbps) to Control FPGA Delivered to Logic FPGA via parallel channel Longer latency Outputs: 24 Virtex-7 GTH links (10 Gbps) Control: PCI express Gen 2, 2 lanes Ipbus INSTR 2017 A. Madorsky

Optical module Optical transmitters Custom backplane connector Optical receivers Backplane redrivers uTCA connector MMC INSTR 2017 A. Madorsky

Optical module 7 12-channel RX 2 12-channel TX Receivers: 7 12-channel RX Avago’s AFBR-820BEZ 84 RX channels Transmitters: 2 12-channel TX Avago’s AFBR-810BEZ 24 TX channels All of them 10 Gbps parts Not enough space on front panel to accommodate all TX parts located inside connect with short fibers to MPO fiber couplers on front panel Tight but enough space to fit couplers on top of AFBR-820 parts. Receivers are on front panel to minimize count of fiber-to-fiber transitions for inputs INSTR 2017 A. Madorsky

PT LUT module Base board Connector (rear side) RLDRAM3 memory 16 chips, 8 on each side (clamshell topology) Total size: 512M x 18 bits ≈ 1GB DC-DC converters INSTR 2017 A. Madorsky

PT LUT Parameters: RLDRAM clock : 200 MHz Address & control: 200 Mbps each bit Data: 400 Mbps each bit RLDRAM can tolerate up to ~1GHz clock. However: Hard to implement in FPGA Needed for burst-oriented applications mostly Does not change latency for random address access Lower clk F  lower power consumption INSTR 2017 A. Madorsky

FireFly Very small footprint 01/06/09 15:26 FireFly Very small footprint All high-speed contacts on one side – easy to route Multiple options for heat sink Footprint compatible with high-speed jumper cables Also by Samtec INSTR 2017 A. Madorsky CMS OO Transition. David Stickland