1. Wesley Smith, U. Wisconsin Project Management Group March 25, 2015
Level-1 Trigger Update
Wesley Smith, U. Wisconsin
Project Management Group
March 25, 2015
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

2. Muon Trigger Transition
New system
Old system
Install fully parallel and higher bandwidth optical path for CSC
New CSC Muon Port Cards mezzanine cards installed in cavern during LS1
Prerequisite done by Operations Program
Alleviates bottleneck & sends all segments from each CSC (robustness to PU and collimated signals)
Build up new Track Finder in 2015 and commission in parallel, ready by 2016
Cavern
Counting Room
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

3. EMU Trigger Upgrade Cavern Counting Room VME Endcap Muon Track-Finder
Trigger MotherBoard
Data MotherBoard
Clock & Control
Muon Sorter
Clock&Control
VME
Sector Processors
Endcap Muon
Track-Finder
(U. Florida)
Muon Port Card
Mezzanine* (Rice)
MPC-EMUTF
Optical Fibers*
(Rice)
(MP7)
*M&S on FY13 Ops. Prog.
MTF7
μTCA: Advanced Mezzanine Cards from
Telecommunications Computing Architecture
(commercial telecommunications hardware)
EMUTF Infrastructure
(U. Florida)
Muon Sorter (Rice)
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

4. Muon Port Card (Ops) (Rice - M&S on Operations Program)
(60 needed + 15 spares + 5 test setups = 80)
Use the existing MPC main board
Backplane interface to TMB remains unchanged
x spares
Test stands = Rice, Florida, 904, likely UCLA and TAMU. Also EMU test stands
3 original optical links are still available
New mezzanine card with new FPGA and new links
Production done, tested, installed
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

5. EMUTF: Hardware Layout
Optical
Patch Panel
MPC Mezzanine
Upgrade
MTF7
μTCA Board
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

6. EMUTF Hardware Counts
Require three μTCA chasses to process 12 sectors + sort output
MTF7 occupies 2 μTCA slots,
12 units in system, each covers 60o sector
Muon Sorter also built on MTF7,1 unit in system
use Boston U AMC13 for clocking and DAQ
Re-route and split optical signals using 12 passive optical patch panels SP MS
Chassis #1
Chassis #2
Chassis #3
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

7. EMU Track-Finder (U. Florida)
Muon Track Finder card: MTF7
13 needed + 4 spares + 3 test setups = 20
Optimized for maximum input from muon detectors (84 input links, 28 output links)
Dual card w/large capacity for RAM (~1GB) to be used for pT assignment in track finding
Back: Core FPGA card ⇒
⇐ Front: Optics card
V7 has 80 links at 10gbps
V6 has 48 links at 6.4gbps
Development and testing complete, production started
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

8. MTF7 Hardware Prototypes @ CERN
Details of board hardware in supporting slides
Base Board
[pT LUT mezzanine not shown]
Optical Board
Pass-through Backplane Board
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

9. HW Prototypes: Testing @ Pt 5
01/06/09 15:26
HW Prototypes: Pt 5
Installed equipment:
Two μTCA chassis, MCH in each
One complete MTF7 kit [Base board + Optical board]
One AMC13
Control computer
PCIe connection to both chassis
One EMU sector optical patch panel
Status:
AMC13 clocking: OK
PCIe access: OK
MPC link tests:
PRBS using IBERT
Both Endcaps – no errors
Signals going via 2-way and 4-way splitters
Confident in our ability operate the old trigger path with the new MPC mezzanine
Upgraded MPCs have participated in global runs since the last week of October, and in the extended cosmic run
Fulfilled “Do No Harm” Principle
25 Mar. 2015, Wesley Smith
PMG: Trigger Update
CMS OO Transition. David Stickland

10. Hardware Production Status
MPC Mezzanine Upgrade:
85 upgraded MPC mezzanines have been built and tested at Rice in the summer of 2014
73 Upgraded MPCs are at CERN - 60 MPCs installed at P5, 13 spares
MTF7 Base Board:
12 base PCBs received, 24 more coming soon
Initial 4 MTF7 base boards submitted for assembly, coming back next week
Full quantity assembly after tests: 20 boards will contain FPGAs, the other 16 without FPGAs at this time
MTF7 Optical Board:
36 boards received from production
4 pcs initial assembly sent out [Mar 11th], coming back next week
Full quantity assembly after tests
pT LUT module
Production prototype under tests
Read latency reduced as expected  (2 BX instead of 6 BX in first prototype)
Firmware under development now
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

11. Commissioning Status Readout of trigger primitives from MPCs:
Using simplified logic for now
One spy buffer
Records 7 BXs on each L1A
Once spy buffer full, stop recording, read out
Start recording again
Trigger primitives decoded on the fly in test software
Valid primitives dumped into text file
Example of trigger primitive dump of cosmic muons:
l1a: 0049 e: 023 m: 1 f: 5 hs: 076 wg: 004 q: 14 p: 09 lr: 1
l1a: 0053 e: 093 m: 4 f: 7 hs: 083 wg: 029 q: 13 p: 07 lr: 1
l1a: 0063 e: 072 m: 3 f: 3 hs: 048 wg: 061 q: 15 p: 10 lr: 0
l1a: 0070 e: 030 m: 3 f: 2 hs: 138 wg: 051 q: 15 p: 10 lr: 0 e: 030 m: 3 f: 2 hs: 084 wg: 049 q: 14 p: 08 lr: 0
l1a: 007a e: 100 m: 3 f: 3 hs: 011 wg: 060 q: 12 p: 04 lr: 0
l1a: 007b e: 107 m: 4 f: 2 hs: 015 wg: 021 q: 11 p: 02 lr: 0
l1a: 0086 e: 051 m: 4 f: 7 hs: 126 wg: 051 q: 11 p: 03 lr: 1
l1a: 0090 e: 121 m: 3 f: 5 hs: 098 wg: 060 q: 14 p: 08 lr: 0
l1a: 0096 e: 030 m: 3 f: 6 hs: 141 wg: 055 q: 14 p: 08 lr: 0
l1a: 009b e: 065 m: 2 f: 7 hs: 084 wg: 031 q: 12 p: 04 lr: 0
l1a: 00a0 e: 093 m: 4 f: 0 hs: 032 wg: 007 q: 14 p: 08 lr: 0
l1a: 00a6 e: 002 m: 4 f: 7 hs: 060 wg: 017 q: 11 p: 02 lr: 0
l1a: 00ad e: 051 m: 4 f: 4 hs: 026 wg: 038 q: 12 p: 04 lr: 0 e: 051 m: 4 f: 4 hs: 026 wg: 035 q: 12 p: 04 lr: 0
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

12. Algorithms Preliminary results, double-checking
Will be used to generate “low luminosity” version of LUT
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

13. Offline Software [Emulators]
Bitwise emulator(s):
Signal emulation generated from Verilog source
Human-written code following algorithm specification
Compare HW vs Signal Emulation: HW is executing FW correctly
Compare HW/Signal Emulation vs Human-written: FW is doing what we think it is doing
Human-written emulator implemented in CMSSW [development branch]
Emulator updated to have option for 30 bit pT LUT
Implementation currently in GitHub assumes full precision values using at most 6 input parameters for each mode
Including configuration parameter to select corridors
These new features are undergoing debugging
Expect to have code to generate corresponding binary files soon
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

14. Firmware Status MTF7 hardware is different from CSCTF
New firmware needed for MTF7 [similar components]
Loading 1 GB of memory content via PCI express
New track finding algorithms better suited for occupancies of up to 18 hits per station, more robust against collimated muons
Reading LUT memory content 4x per LHC clock cycle
Post-LUT corrections (“tail clipping”) only has SW implementation
Possibility of MVA evaluation – studying resource optimization
Design, definition, monitoring of spy buffers
Importing and inclusion of RPC hits in track finding / pT assignment
Readiness:
taking cosmic runs with one sector
with final pT LUT firmware [6 BX → 2 BX latency] will have full data processing chain, will be ready to produce L1 muon objects
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

15. Online Software Data format + Packer / Unpacker:
1st version exists, awaiting integration
Run Control Framework:
0th version exists: no-op hooks based on legacy endcap trigger run control framework
Online SW development is currently under-manned
Have support from USCMS to hire SW expert for this task
Hiring online SW expert was delayed [various issues]
Rebooting search with high priority
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

16. Calo. Trigger Upgrade in Parallel: Split inputs from ECAL and HCAL
2015: ECAL, Legacy RCT & associated Cal. Trigger Cards*
HCAL
Energy (HTR)
ECAL
Energy (TCC)
Regional
Calo Trigger
Global EM
candidates
Region
energies HF
Energy (μHTR)
Layer 1
Layer 2
Current L1 Trigger System
Upgrade L1 Trigger System
oSLB
oRM
HCAL Optical Splitters
2016: HCAL uHTR & assoc. Cal. Trigger Cards US US
(Cluster Finding) UK UK
(Object Finding)
ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (oRM) connect to Present and Upgrade Calorimeter Trigger (*Details of evolution in breakout).
All installed and operating in CMS Global Runs
HCAL: optical splitters drive both HTRs and μHTRs
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

17. Upgrade Calorimeter Trigger Layer 1 and 2 Hardware
US:
(Cluster Finding)
UK:
Vienna:
(Object Finding)
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

18. Stage-1 Upgrade in 2015 (ops)
Layer-1
Existing RCT
18 crates
RCT 0
RCT 1
RCT 2 …
RCT 17
18 oRSCs (new)
oRSC 0
oRSC 1
oRSC 2 …
oRSC 17
USCMS
Layer-2
MP7 (new)
Algorithm Card(s) (1-3)
Imperial HW + Core FW/SW
USCMS Algorithm FW/SW
CTP7 (new)
Single Readout Card
In Situ RCT Test / Monitor
USCMS HW/FW/SW
Global Trigger (existing)
GT Inputs remain unchanged
DAQ – AMC 13 (new)
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

19. oRSC Installation and Commissioning
Optical Regional Summary Cards are attached to each
RCT Jet Summary Card in the RCT racks and converts JSC output to optical fibers
18 cards (1 per/RCT crate). Each card outputs:
3 copies of 2 fibers at 10 Gbps  1-3 MP7s
1 copy of 2 fibers at 10 Gbps  1 CTP7
1 copy at 2Gbps  legacy GCT
Preliminary connection/BERT tests in May 2014
Final production completed summer 2014
Installation at CERN September 2014
Exercised regularly and reliably for the last six months
All 18 oRSCs mounted on RCT crates
oRSC production board – U. Wisconsin
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

20. Stage 1 Algorithm Firmware architecture “GCT in a chip” – J
Stage 1 Algorithm Firmware architecture “GCT in a chip” – J. Berryhill, FNAL team
18 10-Gbps optical fibers of RCT regions, egammas, feature bits
Products for GT and DAQ
Pipelined processing at 80 MHz with latency 10.5 bx
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

21. Stage 1 Algorithm Firmware architecture “GCT in a chip” - FNAL
All algorithm features implemented
All features bit-level emulated
All features board-tested
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

22. Stage 1 trigger commissioning – FNAL, MIT, Rice
Remaining Hardware work:
Synchronization with ECAL and DAQ link commissioning in progress (w/Wisc.)
Commissioning and validation:
MC patterns sent to RCT are sent through oRSCMP7 and compared with emulator
All objects have been bit-level validated with several types of MC samples
Now proceeding to more elaborate and automated pattern tests
Bit-level emulator performance validated against older emulator version for trigger menu development  the FW will deliver the expected physics performance
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

23. Stage-1 Online software development
Online software has all components (RCT, oRSC, MP7) communicating with
trigger supervisor for pattern testing, configuration and debugging, with appropriate
GUI functionality.
To do:
DAQ functions
Automated config of MP7 via IPBUS through a Global Tag in a DB
Further online monitoring features
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

24. MP7 Algorithm Card(s) (10+2)
Stage-2 Upgrade in 2016+
Layer-1
18 CTP7s +h
CTP+0
CTP+1
CTP+2 …
CTP+17
CTP–0
CTP–1
CTP–2 …
CTP–17
18 CTP7s –h
USCMS
DAQ – AMC 13
Layer-2
Full flexibility for traditional or time-multiplexed architecture
Highest precision location of objects with dynamic clustering
MP7 Algorithm Card(s) (10+2)
Imperial HW + FW/SW
USCMS Algorithm FW/SW
mTCA Global Trigger
DAQ – AMC13
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

25. CTP7 Card Reminder – U. Wisconsin
CTP7 Goals for Cal Trig Layer 1
Provide needed interconnect flexibility for all proposed architectures and link speeds
Maximize the power available to the Virtex-7 690T FPGA for trigger data processing
Exploit latest-available technologies
First two units delivered in Dec, 2013
Excellent results in checkout
Full Production Complete Mar ‘15, 2015
Excellent checkout results continued
24 total CTP7s shipped to CERN
True Triple-Link-Rate, Multi-Clock-Domain Card
6.4 and 4.8 Sync Rx, 10G Async Tx in same MGT quad
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

26. Production CTP7s CTP7 CTP7s Without ← in Stage 2 Rack CXPs + heatsinks
← Front
Back→
CTP7s
← in Stage 2 Rack
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

27. CTP7 Builds 1 Q4/2013 Falcon (A) 2 Q2-Q3/ 2014 Raven (B) 6 3 Q1/2015
Build Number
Time Frame
Series
(PCB Rev)
Scheduled Qty
Reject/ Fail
Yield Qty 1
Q4/2013
Falcon (A) 2
Q2-Q3/ 2014
Raven (B) 6 3
Q1/2015
Eagle (C) 8 4
Q1-Q2/ 2015 40 0†
†Incremental result on build currently in-process
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

28. CTP7 Production Steps (6)
Solder Reflow (outside task at contract manufacturer)
Incoming Inspection/Ohmmeter Test
Check resistances on power rails and critical nets
Assign barcode at completion
Initial Mechanical Assembly
Install CXP Sockets, LED light pipes, JTAG mezzanine, front panel
Power On Test Suite
Program JTAG CPLD, MMC Microcontroller
Boot card in µTCA crate—verify IPMI functionality, DC supply rails, ZYNQ booting, Ethernet connectivity, V7 load capability
Final Mechanical Assembly
Install heat sink, optical modules, PRIZM assembly, heat sink cover
Functional Test Suite
Test MGT Links, AXI bridge, QDR memory, clock logic, ancillary I/O
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

29. CTP7 Build 4 Status (As of 17-MAR-2015)40 40 40 40 40 30 28
Units Completed Through Process Step 20
Solder Reflow
Functional Suite
Ohmmeter Test
Final Assembly
Initial Assembly
Power-On Suite 1 2 3 4 5 6
Process Step
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

30. CTP7 Cards at CERN Cards at CERN from Builds 1-3: 12
Completed Shipments from Build 4:
Shipment #1: 6 Units
Shipment #2: 6 Units
Scheduled Shipments:
Shipment #3: 7 Units
Shipment #4: 7 Units
Estimate all shipments at CERN on/before 2nd Week of April, 2015
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

31. Layer 1 Installation (S2E10)
ECAL/HCAL/HF Fibers enter back side of rack as individual LC connections
3U Optical Feed-thru panels pass MPO12 fibers internally through rack to front side
Rx connections made to CTP7 cards as in-line MPO12-MPO12 connections on front side
3 CTP7 uTCA Crates, each covering 120° of φ and all of η
1U Power Panel delivers 48V from existing RCT internal power supplies to crate Power Modules
48V Power Panel
oRSC 10G
Patch Panels
CTP Crate #1
Fiber Feed-Thru 9×
ECAL/HCAL/HF
96 LC to 8 MPO12 Patch Panel
48V Power Panel
CTP Crate #2
Fiber Feed-Thru 9×
ECAL/HCAL/HF
96 LC to 8 MPO12 Patch Panel
48V Power Panel
CTP Crate #3
2 ×
Power Chassis
S2E10 Back Side
S2E10 Front Side
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

32. CTP7 Crate Installation at CERN
All 4 Final CTP7 Crates installed and operating
1 for Stage 1 (rack 9)
3 for Stage 2 (rack 10)
Stg. 2
Stg. 1
oRSC
Stg. 2
Legacy
RCT x9
Stage 2
oRSC
Stg. 2
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

33. CTP7 2016 Trigger Firmware – I Release V1.0 on March 9 - Milestone
Functionality
Brings together 3 different subsystems: ECAL, HCAL, and HF
Applies tower-level calibration in lookup tables
Builds combined trigger tower words and streams them to Layer-2 for processing
Captures inputs for DAQ readout
Captures inputs and outputs at reduced rate for readout via Ethernet (for validation)
NB: All development in Vivado from the beginning
Input Features
Extensive input diagnostics
Link layer checks (reception of valid data, link speed, …)
Protocol layer checks (CRC and Hamming code verification)
Non-invasive eye-scans continuously running
Flexible FW-based input link alignment logic
Allows link alignment on any combination of input links
BC0 latency measurement
BC0 periodicity error counters
Input capture/playback
Data for 256 BXs (both 8-bit trigger tower energies and feature/fine-grain bits)
256 BXs of data can be played out once per orbit, starting with BX=0. Or repeatedly every 256 BXs
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

34. CTP7 2016 Trigger Firmware – II Release V1.0 on March 9 - Milestone
Layer-1 Processing
Applies tower-level calibration in lookup tables
η-corrected scale
Run-time programmable over AXI bus
16-bit combined trigger tower word to Layer-2
9-bit: E+H trigger tower sums,
3-bit: Energy ratio: log(E/H), log(H/E)
2-bits:ECAL and HCAL feature/fine-grain info
1-bit: E-over-H info, 1-bit: Denominator Zero Flag
Layer-1 DAQ
Layer-1 implemented in 3 crates, each crate with one AMC13 with 5 Gbps output bandwidth to DAQ (15 Gbps using 3 cards)
Captures all input data (8-bit trigger tower energies + all feature/fine grain bits) at 100 kHz L1 rate within the maximum output bandwidth of 3 AMC13s:
In total, 3x 4.1 Gbps = 12.3 Gbps
Unpacked data + a few link diagnostic bits
DAQ chain fully integrated into CTP7 FW v1.0
AMC13 low-level driver core also integrated
The same core that we used in October 2014 to successfully transfer data from CTP7  AMC13  FEROL (tests in B14)
Core will be updated for recent BU improvements/updates/bug fixes
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

35. CTP7 FPGA Architecture
Uses a ZYNQ SoC device as the GbE endpoint and utility/support device, and a V7690T as the main processing device
Uses AXI as the interconnect interface (not IPbus)
AXI with an embedded processor (ARM or Microblaze) is a mainstream design flow in the Xilinx tool set
ARM processor (in ZYNQ) at head of hierarchy, AXI bridge extends connection into V7
Embedded Linux runs on the ARM
Not intended to replace the control PCs
Opens up new possibilities for operation and maintenance
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

36. Integrated Eye Scan (ZYNQ-based Application Example)
Familiar Eye Scan diagrams available in IBERT
Scanning engine ported to CTP7 ZYNQ
Low-latency access to links via AXI interface
Parallel Eye Scans in ~same time as a single scan
Runs in parallel in CTP7 FW during data-taking
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

37. CTP7 Software Releases Test Release Commissioning release
Features: Linux, data access
Peek/Poke Registers Dump/Fill Memories
Status: Done – used in May integration tests
Commissioning release
Features: Client / Server TCP/IP software for RCT test & readout
AXI memory device driver, ctp7Server, ctp7Client
Status: Done – Used in July integration tests
Initial Operations and Low Luminosity Releases
Features : Full support of Stage-1 use with Trigger Supervisor, Stage-2 testing
Status: Command Line Interface version Done; Awaiting TS integration; Routine use for both Stage-1 & 2
Mid/High Luminosity Releases
Releases for operations in 2016 and beyond
Status: In development
Core: memsvc, rpcsvc done
Application: CLI in develop.
TS version to start in May
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

38. Stage-2/Layer-1 DQM Tower level plots already available in DQM
Currently using ECAL and HCAL TPGs read out from TCC/HTR
Will retrofit the same code to use CTP7 DAQ when ready
Occupancy, for hot-tower identification + possible inefficiencies
Will add new bit-errors monitoring (TPG vs CTP7 compare)
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

39. Stage-2 Commissioning System being built up at P5 Current status
- Layer 1: 22/36 CTP7s
- Layer 2: 11* MP7s
- Demux: 1 MP7
- μGT: 1 MP7
Layer 1
Three crates with 10 CTP7s
12 more CTP7s at CERN
25 Mar. 2015, Wesley Smith
PMG: Trigger Update

40. Trigger Upgrade Summary
Muon Port Cards commissioned, operating
Endcap Muon track-Finder System in production
Endcap Muon track-Finder pre-production prototypes operating at CERN
Endcap Muon Trigger preliminary firmware and software operating at CERN
Stage-1 Calorimeter Trigger operating at CERN
Stage-1 Calorimeter Trigger firmware and software operating at CERN.
Stage-2 Calorimeter Trigger production complete
Stage-2 Calorimeter Trigger final installation and commissioning underway
Stage-2 Calorimeter Trigger operational firmware and preliminary software operating at CERN
25 Mar. 2015, Wesley Smith
PMG: Trigger Update