B05 – 4 Global Correlator R. Cavanaugh U. Illinois Chicago and Fermilab, L3 Manager, HL-LHC Global Correlator Director’s Review of US-CMS HL-LHC Upgrades 2-3 February Director's Review – L1 Trigger Overview R. Cavanaugh, 2015 February 2-3
WBS definition Basis of Estimate Schedule Cost and Labor Profiles Risk and Contingency R&D status and plans ES&H and QA Summary 2 Outline 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger
Organization Chart to L3 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Trigger Jeff Berryhill (FNAL) Calorimeter Trigger Wesley Smith (UW) Muon Trigger Darin Acosta (UF) Global Correlator Rick Cavanaugh (UIC/FNAL)
R.C. (U. Illinois Chicago and Fermilab) CMS Particle Flow & Tau ID Convener Fermilab LHC Physics Center Coordinator Phase-1 Stage-1 Global Calorimeter Trigger Wesley Smith (U. Wisconsin) – US CMS HL-LHC L3 Calorimeter Trigger Project Manager CMS Trigger Project Manager , Trigger Coordinator 2007 – 2012 Trigger Performance and Strategy Working Group US CMS L2 Trigger Project Manager (construction and operations) 1998 – present US CMS Phase 1 Upgrade L2 Trigger Project Manager 2013 – present Sridhara Dasu (U. Wisconsin) US CMS L3 Manager Calorimeter Trigger (construction & operations) 1998 – present US CMS L3 Manager Phase 1 Calorimeter Trigger Upgrade 2013 – present Author of original and upgrade cal. trig. Algorithms 1994 – present Darin Acosta (U. Florida) CMS Trigger (co)Project Manager, EMU Track-Finder, present Ivan Furic (U.Florida) CSC and EMU Track-Finder projects, 2008-present Alexi Safonov (TAMU) CSC Trigger Motherboards Jeff Berryhill (Fermilab) CMS & US CMS Phase-1 Stage-1 Calorimeter Trigger Project Manager 4 CMS Correlator Management Experience 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 3
Tom Gorski (U. Wisconsin) – Electrical Engineer – Lead Engineer Over a decade of engineering on the CMS Calorimeter Trigger Delivered final phase of original Regional CMS Calorimeter Trigger Delivered Phase 1 Layer-1 Calorimeter Trigger Upgrade Electronics Ales Svetek (U. Wisconsin) – Firmware Engineer 3 years on Phase 1 Calorimeter Trigger Upgrade Firmware (4 years ATLAS Beam Conditions Monitor Firmware, DAQ, Commissioning, Detector Operations) Marcelo Vicente (U. Wisconsin) – Firmware Engineer 3 years on Phase 1 Calorimeter Trigger Upgrade Firmware + HCAL Firmware 2 Years on ECAL Phase 1 Upgrade Trigger Primitive Generation Electronics (oSLB, oRM) Jes Tikalski (U. Wisconsin) – Software Engineer 3 years on Phase 1 Calorimeter Trigger Upgrade Software and embedded systems Alex Madorsky (U. Florida) – Electrical Engineer Over a decade of engineering on CMS Trigger, EMU, Track-Finder (since 1999) Eric Hazen (Boston U.) – Electrical Engineer Long history of engineering on CMS hadron calorimeter electronics and trigger Shouxiang Wu (Boston U.) – Electrical Engineer Long history of engineering on CMS hadron calorimeter electronics and trigger 5 CMS Correlator Engineering Experience 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 3
WBS Definition 6 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger
7 Level-1 Trigger Architecture 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Sorting/Merging Layer Muon Track-Finder MPC CSC DT LB RPC Global Correlations (Matching, PT, Isolation, vertexing, etc.) Global Correlations (Matching, PT, Isolation, vertexing, etc.) Splitters fan-out fan-out fan-out ECAL EB HCAL HB HCAL HB HCAL HF HCAL HF single xtal Regional Calo Trigger Layer Global Calo Trigger Layer Calorimeter TriggerMuon Trigger Tracker Track-Finding Track Trigger GEM + iRPC GEM + iRPC Global Trigger Tracker Stubs HGCAL on-det HGCAL on-det HGCAL off-det HGCAL off-det U.S. Covers fraction DR Question 1
Without L1 Tracks mis-assignment of high p T to low p T muons rate flattens above O(30) GeV Match L1 Tracks & Muons better resolution o sharper turn-on large rate reduction o factor O(5-10) at 20 GeV 8 Example: Correlating L1 Tracks & Muons 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
Electrons Match L1 tracks to EM-clusters o reduces rate by factor O(8-10) at 20 GeV Challenge: tracker material o Retain high efficiency for finding L1 track Possible solution: o Different selections high vs low p T electrons Efficiency ~95% in barrel 9 Example: Correlating L1 Tracks & EM-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
Photons Isolate EM-clusters from L1 tracks o reduces diphoton rate by factor O(5) for 20 GeV leading photon Challenge: tracker material o Photon conversions Possible solution: o Apply annulus track isolation cone Example: track iso of EM-cluster above 20 GeV H to γγ signal eff: ~90%; Bkg eff: ~30% 10 Example: Correlating L1 Tracks & EM-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
Taus Try two (early) approaches o start w/ calo cluster (TkCaloTaus) –match to tracks –apply track-based isolation o start w/ tracks (TkEmTaus) –match to EM-cluster Either algorithm able to maintain ~50 kHz rate with ~50% eff. for H to ττ signal Rate reduced by factor O(5-6) 11 Example: Correlating L1 Tracks & Calo-clusters 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
Find Primary Vertex Fast: histogram z position of track, weighted by track p T o Millimeter-level precision Match tracks to PV Match tracks to calo-only jets Calculate vertex of each jet Require jets have similar vertex (e.g. within 1 cm) Efficiency nearly 95% for jets with p T above 50 GeV 12 Example: Event Vertexing 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
Determine MHT from calo-jets matched to common vertex tracks-only matched to primary vertex Example: Signal ≈ 200 GeV: track-only MET o Rate comes in well below 750kHz menu limit o Efficiency 80%-85% with few 10s kHz rate calo-only MET or MHT o Completely out of reach 13 Example: Event Vertexing 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger From CMS Technical Proposal: CERN-LHCC DR Question 1
14 Model for L1 Correlator Trigger Hardware 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Global Processing Regional Processing Crate A (“barrel-”) Crate B (“barrel+”) … …… … Crate D (“endcap+”) L1 Calo L1 Muons L1 Tracks L1 Calo L1 Muons L1 Tracks L1 Calo L1 Muons L1 Tracks … Based on successful Phase 1 Architecture for CMS Calorimeter Trigger … Global Trigger Global Trigger Base processors on existing CMS Virtex7 trigger processor boards Calculate global sums (MET, SET, etc), sort objects, remove duplicates/ghosts, etc Inspired by particle flow (PF already used at HLT in Run-I) L1 Objects DR Question 1
15 Correlator Trigger Workflow 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger Bottom line: Expect to achieve all primary correlator tasks within allowed 2.5 μs latency Meets the requirements! Bottom line: Expect to achieve all primary correlator tasks within allowed 2.5 μs latency Meets the requirements! DR Question 1
WBS includes all Engineering and Technical activities as well as M&S to produce the Correlator L1 Trigger electronics. The system takes as its input data on optical fibers from the CMS Track, Calorimeter, and Muon Triggers and provides processed trigger data for the CMS Global Trigger. WBS includes managing production of the boards, engineering in support of the production of the boards, procurement of the optical components, FPGAs and all other components on the Correlator L1 Trigger electronics. WBS also includes fabrication of the PCBs and assembly of the finished Correlator L1 Trigger electronics. Costs associated with production of boards are assumed to be consistent with the costs experienced with the Phase 1 Trigger Upgrade with appropriate economies of scale applied when quotes justify them. Less expensive optical parts and FPGAs lack bandwidth to process trigger data in planned number of cards and crates, resulting in a multiplication of the system in size by factors of 2-4, which costs significantly more in both M&S and labor. Technology extrapolations may potentially reduce costs but not included. 16 WBS Overview 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
Correlator Trigger Management Correlator Trigger Milestones, Interfaces Correlator Trigger Travel Regional Correlator Trigger Regional Correlator Trigger M&S (Detail Next Slide) Regional Correlator Trigger Engineering Regional Correlator Trigger Technical Work Regional Correlator Trigger FW Regional Correlator Trigger SW Global Correlator Trigger Global Correlator Trigger M&S (Detail Next Slide) Global Correlator Trigger Engineering Global Correlator Trigger Technical Work Global Correlator Trigger FW Global Correlator Trigger SW Correlator Trigger Infrastructure Crates and Power Supplies M&S Cables, Fibers and Patch Panel M&S Test Facilities M&S Infrastructure Engineering Infrastructure Technical Work WBS: Correlator Trigger 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
X (X=2,3) (Regional, Global) Correlator Trigger M&S X.01 Corr. Trig. Preproduction Optics X.02 Corr. Trig. Preproduction FPGAs X.03 Corr. Trig. Preproduction Misc. Comp. X.04 Corr. Trig. Preproduction PCB Fabrication X.05 Corr. Trig. Preproduction Assembly X.06 Corr. Trig. Optics X.07 Corr. Trig. FPGAs X.08 Corr. Trig. Misc. Comp. X.09 Corr. Trig. PCB Fabrication X.10 Corr. Trig. Assembly 18 Correlator Trigger M&S Detail 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
M&S costs are based on escalated prices of similar components used for the Phase 1 upgrade of the L1 trigger. Labor costs are estimated from engineers currently on staff, or on standard rates as needed. Effort calculated as per the Phase 1 Trigger Upgrade Project. International travel is estimated at $3K per trip, and domestic travel is estimated at $1K per trip. 19 Basis of Estimate Overview 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
Assume ~350 tracks sent from track trigger at 100 bits per track 35 kb per BX Assume similar amount of information from calorimeter and muon triggers 35 kb + 35 kb = 70kb per BX Total input to Correlator is 105 kb per BX Total Bandwidth at 40 MHz: 4.2 Tbs Assume present day boards with 80x10 Gbps links running 192 bits at 40 MHz with 80% packing efficiency CTP7 ($15.4k) can handle 492 Gbs Number of boards per task (physics object reco/ID) Regional: total number of boards per task: 4.2 Tbs / Tbs ≈ 9 Global: total number of boards per task: 3 (one third of regional-layer) Number of boards: 7 tasks = 6 physics objects {e, γ, μ, τ, jets, sums} + 1 dev (test new algos in situ) (7x9=63 system, 2 test stand, 10 spare) 75 regional boards at $15.4k each, (7x3=21 system, 1 test stand, 3 spare) 25 global boards at $15.4k each. Infrastructure Power Supplies, Fibers, Patch Panels summed to $17k/crate of 12 cards Total Cost: $1.8M (US pays for $1M) 20 Cost Estimate (from CMS HL-LHC TP) 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
R&D M&S 2 prototype cards at $20k apiece in FY17,18,19 each $13K ($15K) in FY17 (18,19) for crates and other testing infrastructure. Production M&S costs (63 system, 2 test stand, 10 spare) 75 regional boards at $15.4k each, (21 system, 1 test stand, 3 spare) 25 global boards at $15.4k each. Design one board for both, different FW Production Board M&S breakdown Based on costs of the Phase 1 cards FPGA costs (Virtex plus ZYNQ) of $6.6k. Optical Components $3.6K, Miscellaneous Components $1.4k, Printed Circuit Board $2k Assembly $1.8k. Production Infrastructure M&S Breakdown Based on costs of the Phase 1 Infrastructure Power Supplies, Fibers, Patch Panels summed to $17k/crate of 12 cards 21 M&S for Production and R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger NB: Assumed Phase 1 μTCA Form Factor smaller than other FF’s considered (e.g. ATCA) means costs are higher since overall card infrastructure is amortized over fewer channels ⇒ most conservative cost model DR Question 2
Electronics Engineering and Technical work to design, produce, and test the Correlator Trigger electronics Software Engineering to produce the software to program, test, operate, diagnose, configure, validate and read out the Correlator L1 Trigger upgrade electronics. This includes software to interface with the Trigger Online System Firmware Engineering to implement the full functionality of the Correlator L1 Trigger upgrade electronics Including implementing the trigger algorithms, diagnostics, data acquisition and readout. SW and FW produced in releases for testing boards, commissioning boards and operating them under initial, low, medium and high luminosity HL-LHC conditions. All labor costs based on actual costs for corresponding tasks for the Phase 1 Trigger Upgrade 22 Correlator Trigger Labor 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
The Correlator Trigger labor for the R&D period FY17-19 includes: 3514 FTE hours/year divided equally between University-based Electronic, Firmware and Software Engineers 502 FTE hours/year of Fermilab-based Electronic Engineering for FY18,19. The Correlator Trigger labor for the production phase FY20-23 includes: 3012 FTE hours/year divided equally between University-based Electronic, Firmware and Software Engineers 251 FTE hours/year of Fermilab-based Electronic Engineering. 23 Correlator Trigger Labor FTE 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
24 Construction Schedule 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2 FY25 FY24 FY23FY22FY21FY20 FY19FY18 FY17 CD4 CD1 CD2 CD3 CD0 Specification and Technology R&D Trigger TDR Pre- production Installation LS 2 LS 3 Physics LHC Schedule CDR PDR CD3A FDR Prototyping and Demonstrators Production Readiness Review Production and Test Test & Commission Reality Check: Ph. 1 Cal. Trig. Board: CTP7 1 st Proto: Q4/CY13 6 Pre. Prod. Proto: Q2-3/CY14 1 st Production Board: Q1/CY15 50 Board Production Complete Q2CY15 Installation Complete: Q2/CY15 Commissioned, Operations start: Q3/CY15 ⇒ 21 Months: 1st prototype to pp operations
25 Cost: through FY Feb-2016 R. CavanaughHL-LHC Correlator Trigger Cost = AY $k (No Contingency) L3 AreaM&S*LaborTotalR&D Global Correlator1221k1044k2265k1417k *Includes travel NB: escalation at 3%/year starting in FY17 DR Question 2
26 Cost Profile 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
27 Labor FTE Profile 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
M&S: 50% Based on conceptual design documented in CMS HL-LHC Technical Proposal Items based on existing Phase 1 Trigger upgrade with documented costs Travel: 20% Based on LOE required as determined from Phase 1 Trigger Upgrade Labor: 50% Based on expert judgment using documented experience of similar work required for the Phase 1 Trigger Upgrade Development of activities defined at a conceptual level informed by the experience of the Phase 1 Trigger Upgrade Technical requirements are moderately challenging, but straightforward extrapolation from the Phase 1 Trigger Upgrade Note: Phase 1 Upgrade production/installation finished in Commissioning still underway. Using recent data for labor, M&S in the cost, contingency, risk estimates 28 Contingency Calculation 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
C&S understood since based on Phase 1 Trigger Upgrade Systems experience Boards are extrapolations of existing Phase 1 Trigger Upgrade Boards Using most conservative costing model (construct with Phase 1 hardware) R&D and technology advances offer opportunities to reduce cost. C&S based on experience of the same team that built and wrote software and firmware for Phase 1 Trigger Upgrade Exploiting new commercial tools offer opportunities to reduce amount of custom written FW and SW Opportunities already realized in Phase 1 Cal. Trig. Upgrade 29 Cost and Schedule Risks 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
Senior Engineer becomes unavailable (Low Risk) Hire new engineer, subcontract to consulting firm, use FNAL engineer Software or Firmware does not meet requirements (Low Risk) Hire extra expert effort to recover schedule and help personnel Boards are delayed (design, manufacture or testing) (Low Risk) Hire extra effort to speed up testing schedule Vendor non-performance (Low Risk) Acquire spending authority to use alternative vendors (while original funds are being unencumbered). Input or output electronics (non-trigger) delayed (Low Risk) Built in capabilities of trigger electronics provide signals for their own inputs & outputs 30 Managed Trigger Risks & Mitigation 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 2
None Since global correlator trigger has full self-test capabilities, satisfying performance requirements validated w/o requiring connected inputs or outputs. Trigger electronics can store up sequences of test patterns and inject them into the front end of the trigger electronics at speed These patterns are based on actual LHC pp data taken before LS3 Trigger electronics can receive output, process and record this at speed for subsequent readout by DAQ. 31 External Dependencies 02-Feb-2016 R. CavanaughB01-4: HL-LHC Global Correlator
Safety: follows procedures in CMS-doc-11587, FESHM L3 Manager (R.C.) responsible for applying ISM to trigger upgrade. o Under direction of US CMS Project Management. Modules similar to others built before, of small size and no high voltage Quality Assurance: follows procedures in CMS-doc Regularly evaluate achievement relative to performance requirements and appropriately validate or update performance requirements and expectations to ensure quality. QA: Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments. All modules have hardware identifiers which are tracked in a database logging QA data through all phases of construction, installation, operation and repair. Graded Approach: Apply appropriate level of analysis, controls, and documentation commensurate with the potential to have an environmental, safety, health, radiological, or quality impact. Four ESH&Q Risk levels are defined and documented in CMS-doc Trigger ESH&Q 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 4
After full testing at institute, shipped to CERN All tests recorded (of all types) for individual boards in database Tests use and validate software and firmware test release Acceptance Testing in Electronics Integration Center (EIC) at CERN Individual labs for Muon, Calorimeter, (and now) Correlator Triggers Boards retested to validate institute test results Tests use software and firmware test release Integration Testing in EIC Row of racks with DAQ, Trigger, Central Clock, Crates of other subsystem electronics Operation of a vertical slice with electronics to be tested installed. Tests use and validate software and firmware commissioning release Integration Testing at P5: Global Runs Test with all CMS with cosmics when beam not running/with beam when running Electronics installed in final locations with final cables Full-scale tests with full CMS DAQ/Trigger/Clocking Tests use software and firmware commissioning release Handover to Operations at P5: Global Runs/Parallel Operation After testing completes, continue with Global Runs/Parallel Operation Validate software and firmware initial operational release 33 QA/QC: Testing and Validation 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 3
Goal: Allow development of correlation trigger electronics – specify: o Planned Algorithms o Necessary trigger primitives o Link counts and formats Plan (with CMS HL-LHC Technical Proposal Milestones): Initial definition of trigger algorithms, primitive objects and inter-layer objects (TP.L1.1) – 2QCY16 Baseline definition of trigger algorithms, primitive objects and interchange requirements with subdetectors. (TP.L1.3) – 2QCY17 Detailed software emulator demonstrates implementation of core HL- LHC trigger menu with baseline objects (TP.L1.4) – 4QCY17 o Used to inform the final implementation of the trigger hardware. 34 Correlator Trigger Algorithm R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 1
Closely related to hardware R&D conducted for Calorimeter Trigger (WBS ) Similar HW R&D and Milestones Hardware R&D Milestones - I Initial demonstration of key implementation technologies (TP.L1.2) – 4QCY16 o e.g. > 25 Gb data links, general applicability across HL-LHC o Start construction of initial prototype circuits for demonstration of feasibility of trigger design, leads to: Definition of hardware technology implementation baseline (TP.L1.5) – 1QCY18 o Testing and revisions of prototypes. o Used with algorithm and emulation baseline to define what is needed for → 35 HL-LHC Correlator HW R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 1
Hardware R&D Milestones – II Full-function correlator prototypes produced which allow local comparison with emulator (TP.L1.6) – 4QCY18 o First boards which have sufficient channels, processing capability and bandwidth optical links to meet the requirements of the final boards o These boards will cover only a portion of the correlator processing logic, however, and only local comparisons will be possible between hardware behavior and the emulator. Demonstrator trigger system shows correlator integration and scaling, global/full-chain comparison with emulator (TP.L1.7) – 4QCY19 o End-to-end comparisons over a slice of the detector which include multiple full-capability prototype boards and the prototype full-capability infrastructure o Goal of demonstrating a prototype trigger system with its infrastructure and testing environment capable of being connected to its front end detector for test-beam validation to follow. Final Milestone: HL-LHC Trigger TDR (TP.L1.8) – 1Q2020 o Based on results from Trigger Demonstrators. 36 HL-LHC Correlator HW R&D 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 1
Phase 1 upgrade: two generations (V5, V6) before production boards—similar path reasonable for HL-LHC Today: CTP7 (V7) a very capable “Gen 0” demonstrator for HL-LHC Supporting HL-LHC Tracking Trigger and Calorimeter Trigger R&D Comparatively “young” platform (< 2 years old) w/ new technology Develop Correlator Trigger “Gen 0” test stand Two CTP7 boards for regional correlation layer One CTP7 board for global correlation layer One crate plus fibers and patch panels Evolve test stand with next generation prototype boards Locate test stand at UW or FNAL – Make available as common facility for US-CMS Establish baseline trigger algorithms between hardware and emulator Explore algorithms beyond baseline (inspired by particle flow) 37 HL-LHC Correlator Demonstrator 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger DR Question 1
Summary Feb-2016 R. CavanaughHL-LHC Correlator Trigger
R&D Program will result in designs for the HL-LHC Correlator Trigger that will meet technical performance requirements Scope and Specifications of this Trigger are sufficiently well-defined to support the C&S estimates Based upon common hardware platforms and components ES&H, QA plans, C&S based on experience with original trigger construction and Phase-1 upgrade Management and Engineering teams are experienced with sufficient design skills, having designed and built original CMS trigger and Phase-1 Upgrade 39 Conclusions: HL-LHC 02-Feb-2016 R. CavanaughHL-LHC Correlator Trigger