1. B0X –Y Global Correlator 402.06.05 R. Cavanaugh, L3 Manager, Global Correlator 402.06.05 Director’s Review 2-3 February 2016 1 Director's Review – L1 Trigger Overview R. Cavanaugh, 2015 January 15

2.  WBS definition  Basis of Estimate  Schedule  Cost and Labor Profiles  Risk and Contingency  R&D status and plans  ES&H and QA  Summary 2 Outline 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

3. 3 402.06 Organization Chart to L3 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger 402.06 Trigger Jeff Berryhill (FNAL) 402.06.03 Calorimeter Trigger Wesley Smith (UW) 402.06.04 Muon Trigger Darin Acosta (UF) 402.06.05 Global Correlator Rick Cavanaugh (UIC/FNAL)

4.  R.C. (U. Illinois Chicago and Fermilab)  CMS Particle Flow & Tau ID Convener 2007-2008  Fermilab LHC Physics Center Coordinator 2010-2013  Worked on algorithm firmware and testing of CMS Phase-1 Stage-1 Calorimeter Trigger  Wesley Smith (U. Wisconsin) – US CMS HL-LHC L3 Calorimeter Trigger Project Manager  CMS Trigger Project Manager 1994-2007,  Trigger Coordinator 2007 – 2012  Trigger Performance and Strategy Working Group 2012 - 2015  US CMS L2 Trigger Project Manager (construction and operations) 1998 – present  US CMS Phase 1 Upgrade L2 Trigger Project Manager 2013 – present  Sridhara Dasu (U. Wisconsin)  US CMS L3 Manager Calorimeter Trigger (construction & operations) 1998 – present  US CMS L3 Manager Phase 1 Calorimeter Trigger Upgrade 2013 – present  Author of original and upgrade cal. trig. Algorithms 1994 – present  Jeff Berryhill (Fermilab)  CMS & US CMS Phase-1 Stage-1 Calorimeter Trigger Project Manager  Darin Acosta (U. Florida)  CMS Trigger (co)Project Manager, 2012-16  EMU Track-Finder, 1998- present  Alexi Safonov (TAMU)  CSC Trigger Motherboards 4 CMS Correlator Management Experience 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Work in progress:

5.  Tom Gorski (U. Wisconsin) – Cal. Trig. Electrical Engineer – Lead Engineer  Over a decade of engineering on the CMS Calorimeter Trigger  Delivered final phase of original Regional CMS Calorimeter Trigger  Delivered Phase 1 Layer-1 Calorimeter Trigger Upgrade Electronics  Ales Svetek (U. Wisconsin) – Cal. Trig. Firmware Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Firmware  (4 years ATLAS Beam Conditions Monitor Firmware, DAQ, Commissioning, Detector Operations)  Marcelo Vicente (U. Wisconsin) – Cal. Trig. Firmware Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Firmware + HCAL Firmware  2 Years on ECAL Phase 1 Upgrade Trigger Primitive Generation Electronics (oSLB, oRM)  Jes Tikalski (U. Wisconsin) – Cal. Trig. Software Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Software and embedded systems  Alex Madorsky (U. Florida) – Muon Track-Finder  Over a decade of engineering on CMS Trigger, EMU, Track-Finder (since 1999) 5 CMS Correlator Engineering Experience 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Work in progress:

6. 6 Context of the Correlator within L1 Trigger 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Sorting/Merging Layer Muon Track-Finder MPC CSC DT LB RPC Global Correlations (Matching, PT, Isolation, vertexing, etc.) Global Correlations (Matching, PT, Isolation, vertexing, etc.) Splitters fan-out fan-out fan-out ECAL EB HCAL HB HCAL HB HCAL HF HCAL HF single xtal Regional Calo Trigger Layer Global Calo Trigger Layer Calorimeter TriggerMuon Trigger Tracker Track-Finding Track Trigger GEM + iRPC GEM + iRPC Global Trigger Tracker Stubs HGCAL on-det HGCAL on-det HGCAL off-det HGCAL off-det This talk!

7.  402.06.05.01 Correlator Trigger Management  402.06.05.01.01 Correlator Trigger Milestones, Interfaces  402.06.05.01.02 Correlator Trigger Travel  402.06.05.02 Correlator Trigger  402.06.05.02.01 Correlator Trigger M&S (Detail Next Slide)  402.06.05.02.02 Correlator Trigger Engineering  402.06.05.02.03 Correlator Trigger Technical Work  402.06.05.02.04 Correlator Trigger FW  402.06.05.02.05 Correlator Trigger SW  402.06.05.03 Correlator Trigger Infrastructure  402.06.05.03.01 Crates and Power Supplies M&S  402.06.05.03.02 Cables, Fibers and Patch Panel M&S  402.06.05.03.03 Test Facilities M&S  402.06.05.03.04 Infrastructure Engineering  402.06.05.03.05 Infrastructure Technical Work 7 402.06.05 WBS: Correlator Trigger 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

8.  402.06.05.02.01 Correlator Trigger M&S  402.06.05.02.01.01 Corr. Trig. Preproduction Optics  402.06.05.02.01.02 Corr. Trig. Preproduction FPGAs  402.06.05.02.01.03 Corr. Trig. Preproduction Misc. Comp.  402.06.05.02.01.04 Corr. Trig. Preproduction PCB Fabrication  402.06.05.02.01.05 Corr. Trig. Preproduction Assembly  402.06.05.02.01.06 Corr. Trig. Optics  402.06.05.02.01.07 Corr. Trig. FPGAs  402.06.05.02.01.08 Corr. Trig. Misc. Comp.  402.06.05.02.01.09 Corr. Trig. PCB Fabrication  402.06.05.02.01.10 Corr. Trig. Assembly 8 Correlator Trigger M&S Detail 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

9.  EWK scales, the bread and butter physics of HL-LHC  full potential of HL-LHC ultimately determined by datasets it collects  Datasets ultimately determined by ability to efficiently trigger!  CMS Experience from Run 1 & 2  Offline: o significant improvement using particle flow (PF) event reconstruction  HLT: o PF (carefully) pushed into HLT, again with significant improvement o similar Offline vs HLT objects: sharpened turn-on curves, better rates  L1: o final limitation with no tracking information available o mismatched HLT vs L1 objects: degraded turn-on curves, higher rates  Extrapolate to HL-LHC:  CMS investing in tracking for L1 o Expect similar HLT vs L1 objects, better turn-on curves, better rates  Correlator designed to maximize return on that investment! 9 Big Picture 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

10.  Calorimeter Trigger  Barrel: o Full crystal granularity readout (better isolation) o Increased bandwidth (better energy resolution)  Endcap: o High granularity: 3D clusters (or Rec Hits), timing information, …  Muon Trigger o More spatial points (better momentum resolution)  Track Trigger o Good efficiency over full η range o Low pT threshold (2-3 GeV) o Good pT resolution o mm z-position resolution o Track quality information 10 Better/more inputs = better L1 decisions 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

11.  Based on detector, trigger algos, and L1 Menu inspired from Phase-1 upgrades to CMS  Scenario 140 pile-up events per beam crossing:  No tracking at L1: accept rate ≈ 1.5 MHz  Include tracking at L1: accept rate ≈ 260 kHz  Scenario 200 pile-up events per beam crossing  No tracking at L1: accept rate ≈ 4 MHz  Include tracking at L1: accept rate ≈ 500 kHz 11 Better/more inputs = better L1 decisions 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

12.  Without L1 Tracks  mis-assignment of high p T to low p T muons  rate flattens above O(30) GeV  Match L1 Tracks & Muons  better resolution o sharper turn-on  large rate reduction o factor O(10) at 20 GeV 12 Example: Correlating L1 Tracks & Muons 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

13.  Electrons  Match L1 tracks to EM-clusters o reduces rate by factor O(8-10) at 20 GeV  Challenge: tracker material o Retain high efficency for finding L1 track  Possible solution: o Separate algos for low vs high p T electrons  Efficiency ~95% in barrel 13 Example: Correlating L1 Tracks & EM-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

14.  Photons  Isolate EM-clusters from L1 tracks o reduces diphoton rate by factor O(5) for 20 GeV leading photon  Challenge: tracker material o Photon conversions  Possible solution: o Apply annulus track isolation cone  Example:  track iso of EM-cluster above 20 GeV  H to γγ signal eff: ~90%; Bkg eff: ~30% 14 Example: Correlating L1 Tracks & EM-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

15.  Taus  Try two (early) approaches o start w/ calo cluster (TkCaloTaus) –match to tracks –apply track-based isolation o start w/ tracks (TkEmTaus) –match to EM-cluster  Either algorithm able to  maintain ~50 kHz rate with ~50% eff. for H to ττ signal  Rate reduced by factor O(5-6) 15 Example: Correlating L1 Tracks & Calo-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

16.  Find Primary Vertex  Fast: histogram z position of track, weighted by track p T o Millimeter-level precision  Match tracks to PV  Match calo-only jets to tracks  Require jets from common vertex  Efficiency nearly 95% for jets with p T above 50 GeV 16 Example: Event Vertexing 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

17.  Determine MHT from  calo-jets matched to primary vertex  tracks-only matched to primary vertex  Example: Signal ≈ 200 GeV:  track-only MET o Rate reduced by nearly O(100) o Efficiency 80%-85% with few 10s kHz rate  calo-only MET or MHT o Completely out of reach 17 Example: Event Vertexing 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

18.  Main Conclusions:  Thresholds comparable to Run-1, Run-2, Phase-1  Maintains good efficiency for EWK-scale physics 18 Backup: Early example of simplified L1 Menu from CMS Technical Proposal Director's Review -- [MY L2 AREA] OverviewA. Grace, 2015 September 17 L1 tracks Correlated with object

19.  Main Conclusions:  Thresholds comparable to Run-1, Run-2, Phase-1  Maintains good efficiency for EWK-scale physics 19 Backup: Early example of simplified L1 Menu from CMS Technical Proposal 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger L1 tracks Correlated with object

20.  Preprocess + distribute L1 calo, muon, trigger primitives: ~0.5 μs  Time for correlator algorithms: 2.0 μs  Early working point  Depends on global system  Bottom line:  Expect to achieve primary correlator tasks within allowed 2.5 μs latency  All looks fine! 20 Correlator Workflow 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Ask Peter about PV and where determined? TT or L1? Work in progress: will update with more artistic cartoon

21. 21 Architecture for Correlator Trigger 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Inspired by particle flow (PF already used at HLT in Run-I) Global Regional Base processors on existing CMS Virtex7 trigger processor cards  Based on successful Phase 1 architecture for CMS Calorimeter Trigger  Distribute L1 calo, L1 muon, L1 track trigger objects to correlator  Use tracks to find primary vertex  Match tracks with primary vertex  Match tracks with calorimeter objects  Use tracks to calculate isolation of particle objects Work in progress: will update with more artistic cartoon

22.  M&S costs are based on escalated prices of similar components used for the Phase 1 upgrade of the L1 trigger. Details on next slide  Labor costs are estimated from engineers currenlty on staff, or on standard rates as needed. Effort calculated as per the Phase 1 Trigger Upgrade Project.  International travel is estimated at $3K per trip, and domestic travel is estimated at $1K per trip. 22 Basis of Estimate 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

23.  Assume ~350 tracks sent from track trigger at 100 bits per track  35 kb per BX  Assume similar amount of information from calorimeter and muon triggers  35 kb + 35 kb = 70kb per BX  Total input to Correlator is 105 kb per BX  Total Bandwidth at 40 MHz: 4.2 Tbs  Assume present day boards with 80x10 Gbps links running 192 bits at 40 MHz with 80% packing efficiency  CTP7 ($15k) can handle 492 Gbs  Number of boards per dedicated task (physics object reconstruction/ID)  Regional: total number of boards per task: 4.2 Tbs / 0.492 Tbs ≈ 9  Global: total number of boards per task: 3 (one third of regional-layer) 23 Cost Model 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

24.  Assume 7 dedicated tasks: reco 6 physics objects {e,γ,μ,τ,jets,sums} + 1 dev (test new algos in situ)  Regional: total number of boards = 7x9 = 63  Global: total number of boards = 7x3 = 21  Number (15%) of spare boards = 10+3 = 13  Test stand: 2+1=3  Total number of fibres ($10) and patch panels  15 120 = (84 boards / 80% packing eff) x [ 72 fibres x (1 input + 1 output) ]  Test stand: 540  Assume 1 crate ($10k) holds 10 boards  Total number of crates = 84/10 ≈ 9  Test stand: 1  Total estimated unescalated M&S cost: $1.8M 24 Cost Model 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

25. 25 Construction Schedule 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Pre- produ ction Prototyping and demonstrator Production FY25 FY24 FY23FY22FY21FY20 FY19FY18 FY17 CD4 CD1 CD2 CD3 CD0 Specifications and Technology R&D TDR Installation L1 Trigger LS 2 LS 3 Physics LHC Schedule CDR PDR CD3A FDR ESR Test Installation and Commissioning

26. 26 Cost: through FY 23 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Cost = AY $M (No Contingency) L3 AreaM&S*LaborTotalR&D Global Correlator1819k1958k3746k1435k *Includes travel Warning: I am not confident of these numbers – I am still attempting to accurately map numbers from budget spread sheet to these categories

27. 27 Cost Profile 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Note: I am fairly confident of these numbers and think these costs are accurate

28. 28 Labor FTE Profile 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Note: I am fairly confident that these labor profiles are correct

29.  Goal:  Allow development of correlation trigger electronics – specify: o Planned Algorithms o Necessary trigger primitives o Link counts and formats  Plan (with CMS HL-LHC Technical Proposal Milestones):  Initial definition of trigger algorithms, primitive objects and inter-layer objects (TP.L1.1) – 2QCY16  Baseline definition of trigger algorithms, primitive objects and interchange requirements with subdetectors. (TP.L1.3) – 2QCY17  Detailed software emulator demonstrates implementation of core HL- LHC trigger menu with baseline objects (TP.L1.4) – 4QCY17 o Used to inform the final implementation of the trigger hardware. 29 Correlator Trigger Algorithm R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

30.  Strongly related to hardware R&D conducted for Calorimeter Trigger (WBS 402.06.03)  Similar HW R&D and Milestones  Hardware R&D Milestones - I  Initial demonstration of key implementation technologies (TP.L1.2) – 4QCY16 o e.g. > 25 Gb data links, general applicability across HL-LHC o Start Construction of initial prototype circuits for demonstration of feasibility of trigger design, leads to:  Definition of hardware technology implementation baseline (TP.L1.5) – 1QCY18 o Testing and revisions of prototypes. o Used with algorithm and emulation baseline to define what is needed for → 30 HL-LHC Correlator HW R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

31.  Hardware R&D Milestones – II  Full-function correlator prototypes produced which allow local comparison with emulator (TP.L1.6) – 4QCY18 o First boards which have sufficient channels, processing capability and bandwidth optical links to meet the requirements of the final boards o These boards will cover only a portion of the correlator processing logic, however, and only local comparisons will be possible between hardware behavior and the emulator.  Demonstrator trigger system shows correlator integration and scaling, global/full-chain comparison with emulator (TP.L1.7) – 4QCY19 o End-to-end comparisons over a slice of the detector which include multiple full-capability prototype boards and the prototype full-capability infrastructure o Goal of demonstrating a prototype trigger system with its infrastructure and testing environment capable of being connected to its front end detector for test-beam validation to follow.  Final Milestone:  HL-LHC Trigger TDR (TP.L1.8) – 1Q2020 o Based on results from Trigger Demonstrators. 31 HL-LHC Correlator HW R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

32.  Phase 1 upgrade: two generations (V5, V6) before production boards—similar path reasonable for HL-LHC  Today: CTP7 (V7) a very capable “Gen 0” demonstrator for HL-LHC  Supporting HL-LHC Tracking Trigger and Calorimeter Trigger R&D  Comparatively “young” platform (< 2 years old) w/ new technology  Develop Correlator Trigger “Gen 0” test stand  Two CTP7 boards for regional correlation layer  One CTP7 board for global correlation layer  One crate plus ~500 fibers and patch panels  Evolve test stand with next generation prototype boards  Locate test stand at UW (or FNAL) – Make available as official facility for US-CMS  Establish baseline trigger algorithms between hardware and emulator  Explore algorithms beyond baseline (inspired by particle flow) 32 HL-LHC Correlator Demonstrator 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

33.  C&S understood since based on Phase 1 Trigger Upgrade Systems experience  Boards are extrapolations of existing Phase 1 Trigger Upgrade Cards  C&S based on experience of the same team that built and wrote software and firmware for Phase 1 Trigger Upgrade 33 Cost and Schedule Risk 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

34.  Senior Engineer becomes unavailable (Low Risk)  Hire new engineer, subcontract to consulting firm, use FNAL engineer  Software or Firmware does not meet requirements (Low Risk)  Hire extra expert effort to recover schedule and help personnel  Boards are delayed (design, manufacture or testing) (Low Risk)  Hire extra effort to speed up testing schedule  Vendor non-performance (Low Risk)  Acquire spending authority to use alternative vendors (while original funds are being unencumbered).  Input or output electronics (non-trigger) delayed (Low Risk)  Built in capabilities of trigger electronics provide signals for their own inputs & outputs 34 Managed Trigger Risks & Mitigation 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

35.  Safety: follows procedures in CMS-doc-11587, FESHM  L3 Manager (R.C.) responsible for applying ISM to trigger upgrade. o Under direction of US CMS Project Management.  Modules similar to others built before, of small size and no high voltage  Quality Assurance: follows procedures in CMS-doc-11584  Regularly evaluate achievement relative to performance requirements and appropriately validate or update performance requirements and expectations to ensure quality.  QA: Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments.  All modules have hardware identifiers which are tracked in a database logging QA data through all phases of construction, installation, operation and repair.  Graded Approach:  Apply appropriate level of analysis, controls, and documentation commensurate with the potential to have an environmental, safety, health, radiological, or quality impact.  Four ESH&Q Risk levels are defined and documented in CMS-doc-11584. 35 Trigger ESH&Q 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger

36. Summary 36 Director's Review -- [MY L2 AREA] OverviewA. Grace, 2015 September 17

37. 37 Summary Director's Review -- [MY L2 AREA] OverviewA. Grace, 2015 September 17