1. LHC CMS Detector Upgrade Project Level-1 Trigger Upgrade Status Wesley Smith, U. Wisconsin US CMS Project Management Group December 18, 2013 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 1

2. LHC CMS Detector Upgrade Project  L5 - MPCM Software and Firmware Start401.04.03.02.012013-11- 01  Initial firmware version is ready, software for test stand is being built.  L5 - MPCM Production Start401.04.03.02.022013-11-01  Started, 2 pre-production mezzanines are at Rice and being tested. All the parts except optical transmitters are at Rice. Should be ready to start assembly in a few weeks.  L5 -Start MPC-EMUTF Optical Fibers401.04.03.032013-11-01  Short optical pigtails for MPC boards are at Rice. Out of 36 trunk cables for the EMUTF 27 have been delivered to CERN and tested there. The remaining longest 100+ m cables are expected from Draca in January. Installation at p.5 is planned for spring.  L5 -Start Muon Sorter401.04.03.062013-11-01  Preliminary work on hardware design started.  L5 -Start Trigger Production401.04.03.062013-11-01  Started, covered above  L5-Start MSM Production401.04.03.06.022013-11-01  Not started yet 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 2 L5 Active Mu. Trig. Milestones - Rice

3. LHC CMS Detector Upgrade Project 3 Muon Port Card Status (1) ■ Received 90 optical pigtails from CERN in November ■ Fabricated 81 new front panels ■ All parts for 80 mezzanine boards have been ordered and most of them are already at Rice. Optical transmitters from Avago should arrive in the second half of December. ■ 85 optical transmitter boards have been fabricated and assembled in October 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status

4. LHC CMS Detector Upgrade Project  Implemented few minor layout changes in the mezzanine design.  Two pre-production boards have been assembled by Pactron in the middle of November and are being tested at Rice. 4 Muon Port Card Status (2) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status

5. LHC CMS Detector Upgrade Project  Complete testing of the two pre-production mezzanines by the end of December  Assemble 80 production boards in the 1st half of January  Test 80 new mezzanines at Rice in January-March  Set up a uTCA test stand at Rice in January-March  Installation of 36 optical trunk cables at p.5: spring 2014  Installation of the upgraded Muon Port Cards at p.5: late spring- early summer 2014 5 Muon Port Card Mezzanine Plans 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status

6. LHC CMS Detector Upgrade Project  L5-Start EMUTF Test, Install, and Commission 401.04.03.04.03 2013-11- 04  A control PC of the type recommended by CERN was ordered for tests in Florida  L5 -Start EMUTF Infrastructure401.04.03.05 2013-11-01  A control PC of the type recommended by CERN was ordered for tests in Florida  First Optical Patch Panel Prototype delivered (next slide)  L5 -Start EMUTF Infrastructure Production 401.04.03.05.012013-11- 01  Started, covered above  L5-Start EMUTF Software and Firmware401.04.03.04.012013-11- 04  A first version of the software emulator (for the initial endcap TF algorithm) has started. It is nearly ready to be used in CMSSW, but likely will occur in early 2014.  Initial firmware for all MTF cards (FPGA control board, optics board, and PT LUT mezzanine board) has been written and already used for testing of the prototypes.  L5-Start EMUTF Production401.04.03.04.022013-11-04  The preproduction of the MTF7 board with Virtex-7 has started with the submission for PCB manufacture this December, and components procured. 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 6 L5 Active Mu. Trig. Milest. - Florida

7. LHC CMS Detector Upgrade Project Endcap Muon TF block diagram 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 7 Optical plant (fanouts and splitters) From MPCs 60 12-core fibers, 8 cores used in each. 90 trig. primitives per 60° sector 3.2 Gbps From RPC Up to 216 fibers at 1.6 Gbps (may be concentrated to higher bandwidth and fewer fibers) Sector Processors 12 units 60° sector each To Overlap Track Finder Best 3 Muons in each sector (36 total) To GMT

8. LHC CMS Detector Upgrade Project EMUTF Optical Patch Panel 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 8 Prototype patch panel to split, fan- out and bundle fibers to appropriate processor inputs. Units are only 1 U wide.

9. LHC CMS Detector Upgrade Project  Sector Processor (SP) o Each occupies 2 uTCA slots o 12 units in system  All chassis use AMC13 (designed by Boston University) o Clocking, TTC, and DAQ o 3 units  Plan to control boards via PCI express for development (faster)  But will be compatible with IPbus as well for operations per CMS standard o IP-based protocol for controlling hardware devices adopted by CMS. 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 9 uTCA chassis SP Chassis #1 Chassis #2 Chassis #3

10. LHC CMS Detector Upgrade Project Modular Muon Track Finder (MTF) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 10  Modularity makes future partial upgrades easier (e.g. FPGA)  Card layout is simpler  Power density is less Optical module Custom backplane connector Core logic module Optical module Core logic module Pt LUT module Custom backplane

11. LHC CMS Detector Upgrade Project Optical module 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 11 uTCA connector Custom backplane connector Backplane redrivers Optical receivers (2 out of 7 installed) MMC Optical transmitters (2 out of 3 installed) Tested, Done! Works with either V6 or V7 MTF versions

12. LHC CMS Detector Upgrade Project MTF6 Core logic module (Virtex6) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 12 Custom backplane connector Core logic FPGA Control FPGA uTCA connector Control FPGA JTAG PT LUT module connector FMM connector SD card connector MMC USB console MMC JTAG MMC CPU Estimated power consumption: ~50 W (assuming FPGAs nearly full) PT LUT mezzanine not included 1Gb FLASH Main FPGA firmware storage MMC = Module Management Controller Tested

13. LHC CMS Detector Upgrade Project MTF7 design (Virtex-7 version) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 13  Core logic module  FPGAs:  Core logic: XC7VX690T-FFG1927 or XC7VX550T-FFG1927  Control: XC7K70-FB676  Inputs:  80 Virtex-7 GTH links (10 Gbps) directly to Core FPGA o Minimal latency o All available receivers are designated for trigger data –Maximum flexibility  4 additional Kintex GTX links (10 Gbps) to Control FPGA o Delivered to Core FPGA via parallel channel o Longer latency  Outputs:  28 Virtex-7 GTH links (10 Gbps)  Control:  PCI express Gen 2, 2 lanes  IPbus

14. LHC CMS Detector Upgrade Project MTF7 Base board 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 14 Main FPGA Control FPGA DC-DC converters (switchers) Low-Dropout Linear regulators MMC PT LUT connector Custom backplane connector uTCA connector Layout finished RFQ for board production submitted Assembled boards should arrive by end of December

15. LHC CMS Detector Upgrade Project Glue logic FPGA (Spartan-6) Base board connector RLDRAM3 memory 16 chips, 8 on each side (clamshell topology) Total size: 512M x 18 bits ≈ 1GB Upgrade possible to 2 GB with bigger RLDRAM3 chips (no board redesign) DC-DC converters Clock synthesis and distribution PT LUT module 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 15 Tested! Works with either V6 or V7 MTF core logic modules Investigating further latency reduction

16. LHC CMS Detector Upgrade Project Optical communication test 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 16  @3.2 Gbps  47 input channels  Transmission from: o Loopback o Muon Port Card o Earlier VME prototype (2010)  For MPC and VME prototype clock was synchronized with VME crate o Twisted pair LVDS connection to uTCA backplane  @10 Gbps  6 input channels  Asynchronous clock  Transmission from: o Loopback o Earlier 10Gbps prototype (2006)  Results  Zero errors for hours Eye pattern @10 Gbps. GTH receiver input.

17. LHC CMS Detector Upgrade Project Integration tests 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 17  Mimicking real system setup:  Sending data from 2 MPCs to MTF6  Each MPC clocked by separate CCB  MTF6 receives clock from AMC13  CCBs and AMC13 receive clock from common source (TTCvi)  Types of tests: o PRBS o Random data via test FIFOs o Latest MPC data format used  Result: o No errors.

18. LHC CMS Detector Upgrade Project Optical components tests 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 18  Trunk cable identical to what will be used for Endcap TF  Total length (126 m) exceeds actual max fiber length in Endcap TF system (113 m)  Results: o @3.2 Gbps: no errors with 2-way and 4-way splitting [CSC signals] o @1.6 Gbps: 2-way splitting OK, errors with 4-way [RPC signals] o Newer optical receivers lose efficiency at low bitrates MPC or MTF6 MPC or MTF6 63-meter trunk cable (2 lengths) Couplers Fanout Splitters Fanout Endcap TF needs: 2-way splitters: 348 4-way splitters: 60 Rate: 3.2 Gbps

19. LHC CMS Detector Upgrade Project PT LUT tests 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 19  Parameters:  RLDRAM clock : 200 MHz  Address & control: 200 Mbps each bit  Data: 400 Mbps each bit  RLDRAM can tolerate up to ~1GHz clock. However: o Hard to implement in FPGA o Needed for burst-oriented applications mostly o Does not change latency for random address access o Lower clk F  lower power consumption  Tests performed (random data, full 1GB space):  Writing into consecutive addresses  Reading from consecutive addresses  Reading from random addresses  No errors detected  Except soldering defects in one RLDRAM chip

20. LHC CMS Detector Upgrade Project  L5-Start Calorimeter Trigger Processor 401.04.04.022013-11-01  Two Prototype Boards delivered this week  L5-Start CTP7 Software and Firmware401.04.04.02.012013-11-01  Initial SW and FW for Prototype Operation is being prepared  L5 -Start CTP7 Production401.04.04.02.022013-11-01  First two prototype PCB delivered, parts received and assembly underway (see next slides)  Ordered and received 8 Virtex-7 FPGAs.  L5 -Start CIOx Card401.04.04.032013-11-01  Prototype CIOx card made and is adequate for use.  L5 -Start CIOx Software401.04.04.03.012013-11-01  Included in CTP6 software package  L5 -Start CIOx Production401.04.04.03.022013-11-01  Production needed in 2016. Will reschedule.  L5 -Start CTP Infrastructure401.04.04.042013-11-01  Have 3 Vadatech Crates and optical fibers for interconnects in hand.  L5 -Start oRM401.04.04.052013-11-01  oRM Production underway (not US cost – this item is personnel for installation)  L5 -Start oSLB-oRM Optical Fibers401.04.04.062013-11-01  Fibers have been purchased (not US cost – this item is personnel for installation) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 20 L5 Active Cal. Trig. Milest. – Wisc.

21. LHC CMS Detector Upgrade Project Current & Stage 1 Cal. Trigger  Stage 1 Calorimeter Trigger Upgrade  Improved processing of current RCT with new oRSC  Converts current system to optical  New HF (fwd calo) data is available to upgrade path  Prepare for the Stage 2 Upgrade  Layer 1 cards will also read out RCT in Stage 1 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 21

22. LHC CMS Detector Upgrade Project  Two oRSCs are at 904 since mid-October  Integration Row & Lab Test Crate  Continuing to validate FPGA builds at UW  Finishing up VME access now  New oRSC code to test VME  VME-Spartan BE-Kintex FE working  Integrating oRSC into Trigger Supervisor  Using new oRSC code oRSC 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 22

23. LHC CMS Detector Upgrade Project Stage 2 Calo Trigger – Layer 1  Layer 1 cards will receive the calorimeter trigger primitives via optical fiber from  ECAL TCCs with oSLB 4.8 Gbps optical to oRM on legacy RCT o To be installed this winter o Send duplicate to layer-1  HF and HCAL  HTRs o Passive optical splitting before HTRs for deployment of  HTR in parallel 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 23

24. LHC CMS Detector Upgrade Project CTP7 Concept Main Layer-1 processor card Virtex-7 690T for processing ZYNQ for TCP/IP + Linux 60 10Gbps optical Input links 36 10Gbps optical Output links Function: Find E T clusters & transmit to Layer-2 Card Count = 46 36 total + 8 spares + 2 test setups Virtex-7 VX690T FPGA ZYNQ XC7Z03 0 EPP 1.5V Supply 2.5V Supply 3.3V Supply 1V 30A Supply CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx 12X Rx 12X Rx (CTP-6 BG View) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 24

25. LHC CMS Detector Upgrade Project Calorimeter Trigger Processor Prototype for Layer-1  4 Rev. A and 8 Rev. B CTP6s have been built  48 input links in 4 Avago AFBR-820B Rx Modules  12 output links in 1 Avago AFBR-810B Tx Modules  Back-end & Front-end FPGA XC6VXH 250T/380T  Dual SDRAM for DAQ and TCPIP buffering  Custom power modules working at 1.0V, 1.5V, & 2.5V  Embedded Microblaze processor with Linux on back-end FPGA  Custom MMC for control o Phased power-supply enables o Add’l analog voltage, power good and FPGA Load Done sensors  Parallel FPGA Flash - ~6 seconds to load both FPGAs  SI5338 Prog. Clock Synthesizer, multiple options for Reference clocks o 4.8 Gbps: 120, 240, 300 MHz options o 6.4 Gbps: 160, 320, 400 MHz options  Nelco 4000-13 SI laminate o Lower losses & better ε r for 50Ω trace geometry 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 25

26. LHC CMS Detector Upgrade Project CTP6 Layer-1 Prototype MMC Power Supplies BE FPGA FE FPGA Avago Rx (4) Avago Tx Dual SDRAM For DAQ & TCP/IP JTAG USB Interface 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 26

27. LHC CMS Detector Upgrade Project CTP6 Back Side Inter-FPGA Links Link Clock Fanout & Distribution 12X Frontpanel Tx Link Source- Select Muxes (Front or Back- End FPGA) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 27

28. LHC CMS Detector Upgrade Project UW Test Crate BU AMC13 Vadatech MCH UW CTP-6 TTC Downlink UW Aux (Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6) U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 28

29. LHC CMS Detector Upgrade Project CTP6 Status  CTP6 inter-board link tests show wide margins  HF from  HTR to CTP6 integration successful  12 links tested at 6.4 Gbps – no errors  RAM based data between  One CTP6 at 904 to continue SW work, HW Integration 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 29

30. LHC CMS Detector Upgrade Project oRSC to CTP6 Integration Test  Data patterns successfully transferred from the oRSC, captured on the CTP6 and readout via embedded Linux Fiber capture RAM readout SW controlled capture Capture RAM clears on each capture All six links capture expected pattern from oRSC Automatically checked by host 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 30

31. LHC CMS Detector Upgrade Project CTP6 and 7 Status  CTP6 at 904 at CERN since Mid-october  Checked out  Necessary tools work  Most effort with oRSC at the moment  CTP7 prototype board in UW lab, post-production final assembly  Installation of CXP modules – new innovation – see next slides 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 31

32. LHC CMS Detector Upgrade Project  Conforms to 100GbE and 12x10Gbps Infiniband standards  12 optical lanes in each direction  MPO-24 optical connector  Multi-rate capable from 1 Gbps to 10.5 Gbps  850 nm wavelength  Single 3.3V supply  Uses same Avago MicroPOD engine that is used in the Avago MiniPODs  Optically compatible with Wisconsin oRSC Card and Imperial College MP7 Card  Pluggable frontpanel interface means module can be replaced without a card/crate PCB extraction cycle 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status Avago AFBR-83 CXP Module 32

33. LHC CMS Detector Upgrade Project CXP Module, cont’d PMG: Trigger Upgrade Status Optical Tx Board Optical Rx Board MicroPod Optical Rx Interface 12-fiber Ribbon to Optical Tx Engine AC coupling caps (external caps not req’d) MicroPod Tx Optical Engine (bottom side) 18 Dec. 2013, Wesley Smith33

34. LHC CMS Detector Upgrade Project CXP Cage Press-Fit Insertion Operation  Post-reflow operation  84 electrical pins, 4 screws and 2 alignment pins per housing  Electrical pins are press-fit  Requires a tool for properly force application, and a fixture for bottom side board support  Designed and made a fixture for the CTP7 PMG: Trigger Upgrade Status Molex I-Pass Plus EF Connector P/N 170465-0001 Tool centers insertion force over electrical pins CTP7 PCB Molex Insertion Tool Fixture 18 Dec. 2013, Wesley Smith34

35. LHC CMS Detector Upgrade Project CTP7 CXP Board Hole Pattern  84 Electrical holes are spec’d for 14.6 mils, ±2 mils  Finished hole diameter spec should be given to the PCB vendor, letting them work backwards from there  PCB Decal has ringed border to chassis ground PMG: Trigger Upgrade Status Alignment & Screw Retaining Holes Electrical Connection Press-Fit Holes CTP7 PCB Ground Vias (Smaller Holes) 18 Dec. 2013, Wesley Smith35

36. LHC CMS Detector Upgrade Project CTP7 CXP Press-In Fixture  Milled from ¾” aluminum stock  For use after components are soldered on to the board  Clearances for bottom side components  Provides critical board support immediately under the 84 electrical pins  Custom-designed to match CTP7 board design PMG: Trigger Upgrade Status Relieved for Bottom Side Component Clearance Direct board support immediately under the CXP Connector electrical Holes (3 places) Kapton tape for anti- gouge protection Alignment Pin 18 Dec. 2013, Wesley Smith36

37. LHC CMS Detector Upgrade Project CXP Modules on CTP7  Each CXP module provides 12x lanes of optical Tx and Rx, at speeds between 1Gbps and 10.5Gbps  CTP7 uses 3 CXP modules for front side optical connection  Front panel mounting simplifies PCB design  Avago CXP Modules use same optical engine as Avago MiniPODs for complete compatibility with other CMS boards  Press-fit receptacles required design of special CTP7 holding fixture during the press-in process  Front-panel pluggable design means board extraction not required to replace module PMG: Trigger Upgrade Status 18 Dec. 2013, Wesley Smith37

38. LHC CMS Detector Upgrade Project Embedded processing  Xilinx provides two embedded solutions  Soft – Microblaze, on CTP6  Hard – ZYNQ, on CTP7  PetaLinux is an embedded Linux product from Xilinx  Runs on Microblaze and ZYNQ platforms  Standard, debugged drivers for SPI, I 2 C, Ethernet, USART, Flash, etc.  Communications and control functions are most easily handled by processors and software  Advantages  General communications & control functions originally requiring HDL design now can be done by SW engineers writing Linux Apps  More SW engineers than FW engineers  Deployed on CTP6 & oRSC using Microblaze  On the CTP7 using ZYNC processor 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 38

39. LHC CMS Detector Upgrade Project CTP7 Zynq SoC Linux Control Chip ZYNQ has hard ARM core embedded Linux on transforms housekeeping tasks into Linux application Xilinx AXI chip-to-chip exposes Target FPGA into ZYNQ memory space Can store FPGA bitfiles on μSD FAT filesystem for easy upgrades via SCP/pop in a new μSD SDRAM Low Speed Peripherals (EEPROM, Clock Synths, Optical Modules, Crosspoint Switches, Console, etc.) TargetFPGA AXI Chip-to-Chip Interface SPI Config Interface GbE Connection ZYNQSoC(Linux) microSDFlashMemory Target AMC Board 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 39

40. LHC CMS Detector Upgrade Project CTP Software  CTP6 has two FPGAs o Front-end connected to optical links o Back-end connect to Ethernet o Inter FPGA Communication via serial bus: UART  CTP7 follows the same model, replace BE by ZYNQ chip  Development going on now should be portable 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 40

41. LHC CMS Detector Upgrade Project CTP6 Software  All of the FPGAs have a Microblaze core  On the CTP6 BE, PetaLinux is installed  So we use IPBus protocol, in our “softipbus” implementation available in the CACTUS SVN repository  softipbus includes: o A TCP server that runs on Linux and interprets the byte stream as IPBus o A TCP server that runs on Linux and forwards the byte stream along a file descriptor o Functions to de-serialize a byte stream into IPBus transactions o Functions to handle IPBus transactions (READ/WRITE/RWM) o Functions to serialize the responses into a bytes stream  Code is written so it can run on Linux as a server or be used as library on the standalone devices o All the standalone programs are in github –Problem Tracking and Wiki too 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 41

42. LHC CMS Detector Upgrade Project MMC Project: IPMI System Manager μTCA features Gigabit Ethernet to address individual slots and IPMI to manage the power and sensors of individual cards Commercial IPMI management tools available, but Vendor specific Not extensible (GUIs) 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 42 IPMI: Intelligent Platform Management Interface MMC: Memory Management Controller

43. LHC CMS Detector Upgrade Project IPMI System Manger 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 43 API wrapping a socket interface Act as a gateway for other applications into the IPMI system Provide a robust IPMI-over-LAN connection to crates Provide a service for automatic low- level initialization of AMC cards at startup Crate MCH

44. LHC CMS Detector Upgrade Project  Complete oRSC testing and manufacture cards  Test prototype CTP7 in Madison Lab and then bring to CERN for integration tests with MP7 card  Work on implementation and optimzation of Stage-1 algorithms  Design and Performance documented in Level-1 TDR  Collaboration with FNAL and UIC  Continue Firmware development for oRSC and CTP7  Integrate Software for oRSC and CTP7 with online Trigger Supervisor system 18 Dec. 2013, Wesley Smith PMG: Trigger Upgrade Status 44 Calorimeter Trigger Plans