Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 1 Task T - CMS at LHC Personnel: Professors: D. Carlsmith, S. Dasu, M. Herndon, D. Reeder, W. Smith Senior Scientist: R. Loveless (  New) (  Retiring) Senior Electronics Engineer: T. Gorski Associate Scientist: P. Chumney, Assistant Scientists: M. Grothe, A. Lanaro Postdoc: Y. Baek Engineers: D. Bradley, A. Golyash, M. Jaworski, A. Mohapatra, S. Rader Graduate Students: M. Anderson, K. Grogg, J. Klukas, K. Lazaridis, J. Leonard, M. Weinberg Technician: R. Fobes PSL Engineers: F. Feyzi, J. Hoffman, P. Robl, D. Wahl, D. Wenman PSL Draft/Tech: B. Dana, G. Gregerson, D. Grim, J. Johnson, D. Northacker, J. Pippin, R. Smith Outline of Talks & Activities: Overview & Trigger Activity -- this talk Computing & Software & Physics Activity -- Sridhara Dasu Common & Muon & Tracking Activity -- Dick Loveless Endcap Muon Alignment Activity -- Duncan Carlsmith

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 2 D. Reeder CMS Muon Institutional Board Chair Founding US CMS Collaboration Board Chair W. Smith CMS Trigger Project Manager CMS Steering Committee & Management Board CMS Electronics Board, Steering Committee & Review Team US CMS Trigger Level 2 Manager US CMS Steering Committee, Project Management Group S. Dasu CMS PRS (Physics Reconstruction & Software) Online Selection Group Co-Convenor US CMS Calorimeter Trigger Level 3 Manager US CMS Advisory Software & Computing Board Chair R. Loveless US CMS Muon Level 2 Construction Project Manager US CMS Common Projects Level 2 Manager CMS Endcap Muon System Tech. Coord., CMS Technical Board US CMS Steering Committee, Project Management Group CMS Tracker Integration Team A. Lanaro US CMS Muon Installation Level 3 Manager Wisconsin Senior Personnel Official Responsibilities

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 3 Calorimeter Trigger Personnel experienced team Physicists (DOE university program support): Faculty: Wesley Smith (CMS Trigger Project Manager) Sridhara Dasu (US CMS Cal. Trigger L3 Manager) Associate Scientist: Pamela Chumney 6 yrs on CMS Cal Trig, former HERMES Trig. Coor. Assistant Scientist Monika Grothe 3 yrs on CMS Cal Trig, 2 years ATLAS higher level trig., 3.5 yrs Babar, SVT Commissioner/Run-Coordinator Engineers (US CMS project support): Tom Gorski -- Lead Engineer & Designer 3.5 yrs on CMS Cal Trig, worked at PSL on SDC & Zeus Cal. Triggers Replaced J. Lackey, who retired but is available for consulting Matt Jaworski -- Design support, Testing Worked on Zeus Calorimeter Trigger Phil Robl -- Copper Link Test/Development Lead PSL Engineer, Worked on Zeus Trigger, SDC, CMS EMU Technician (project support): Robert Fobes -- Repair, assembly, parts, orders, inventory Worked on Zeus Trigger Students (work as TAs for now): Mike Anderson, Kira Grogg, Kristos Lazaridis, Jessica Leonard, Marc Weinberg Trigger emulator & simulations, Trigger Rate Calculations

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 4 Endcap Muon Personnel Physicists (DOE univ. program support): Dick Loveless (EMU level 2 manager) Armando Lanaro (L3 manager for installation) Yongwook Baek (cabling, low voltage, installation) A. Lanaro & Y. Baek based at CERN; responsible for day-to-day work on integration and installation Duncan Carlsmith (Alignment) - CDF Task Jim Bellinger, CDF Scientist Jeff Klukas, Grad Student Engineers(US CMS project support): Farshid Feyzi, Dan Wenman (UW Physical Sciences Lab) Techs(project support -- paid as needed): Bart Dana, Mauro Giardoni, Jay Johnson, Dave Northacker, Ron Smith (UW -PSL) Tracker & Common Projects (Endc. Disks): Dick Loveless PSL Personnel paid by US CMS Project L to R: F. Feyzi, D. Wenman, A. Lanaro J. Johnson, M. Giardoni, “Regis”, Y. Baek

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 5 Computing & Software Personnel Physicists (DOE university program): Sridhara Dasu Wisconsin Tier-2 Manager - DISUN Technical Coordinator, Grid Laboratory of Wisconsin (GLOW) Chair, USCMS Software Advisory Computing Board System Manager (DOE university program): Steve Rader Software Engineers (NSF-ITR & US CMS Computing Project): Dan Bradley Ajit Mohapatra & Will Maier Collaboration with U. Wisconsin Computer Sciences Dept. & Condor Group Prof. Miron Livny Director of Condor Project Leadership role in Grid Projects: PPDG, GriPhyN, iVDGL PI, GLOW

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 6 Detector Frontend Computing Services Readout Systems Filter Systems Event Manager Builder Networks Level-1 Trigger Run Control CMS Trigger & DAQ Systems Level-1 Trigger Requirements: Input: 10 9 events/sec at 40 MHz at full L = Output: 100 kHz (50 kHz for initial running) Latency: 3  sec for collection, decision, propagation Output direct to computer filter farm (no physical level-2)

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 7 CMS Level-1 Trigger W. Smith, Project Manager Wisconsin fully responsible US CMS Trigger US CMS fully responsible US CMS partially responsible UNIQUE: All system present trigger objects w/location, E T, quality -- topological triggers possible

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 8 Wisconsin CMS Task Support (over past decade) UW Support of Personnel J. Lackey, S. Dasu, PSL Engineering UW Equipment Support Teleconference Equipment Video Room Remodeling CAD Printer UW CAD support Purchase of Visula PCB Suite, incl. high speed router New version of software used for Zeus trigger UW Computing Support > $ 700K 5 New Servers, 2 TB Disk, 1 GB/s network, Grad. Student RA 70 CPU system for simulation with Condor + network upgrade Two new Computer Rooms with power & AC Matching for “GLOW” shared computing facility

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC MHz point to point backplane 18 Clock&Control,126 Electron ID & Receiver Cards -- all validated 18 Jet/Summary Cards -- all cards 160 MHz Use 5 Custom Gate-Array 160 MHz GaAs Vitesse Digital ASICs Phase, Adder, Boundary Scan, Electron Isolation, Sort Calorimeter Trigger Crate Spares not included* Data from calorimeter FE on Cu 1.2 Gbaud Into 126* rear Receiver CardsInto 126* rear Receiver Cards

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 10 SORT ASICs (w/heat sinks) EISO Bar Code Input DC-DC Converters Clock delay adjust Clock Input Oscillator DC-DC Adder mezz link cards BSCAN ASICs PHASE ASICs MLUs Back Bar Code Front Regional Cal. Trigger Complete - U. Wisconsin Receiver Card: Electron Isolation & Clock: Jet/Summary: Receiver Mezz. Card BSCAN ASICs Sort ASICs BSCAN ASICs Phase ASIC 158/164 tested (126) 154/154 Tested (126) 23/25 tested (18) 23/25 tested (18) (27/28 Custom Backpl Tested) (18) fraction tested (needed) EISO Clock Front Back Five full crates at CERN Rest waiting here

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC RCT Full Crates Operating at CERN - U. Wisconsin Rear of Full RCT crate fully cabled to HCAL trigger primitive logic Front of Full RCT crate with Jet Capture Card that continuously samples output, checks on the fly for errors and provides readout of 256 crossings Used for integration tests & board checkout after shipment

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 12 New: Jet Capture Card -- Tom Gorski Receives 14 Reg. Sums, 8 HF E T ’s, 8 e/ , and  and  bits on 6 SCSI type cables Captures output from Jet Summary Card going to Global Calorimeter Trigger Being used in final validation of RCT crates, all cards will go through an extended full crate test with the JCC Software developed by Pam Chumney & Tom Gorski Provides calorimeter trigger for CMS Magnet Test & Cosmic Challenge Provides temporary trigger to Global Trigger while Global Calorimeter Trigger is being prepared FPGAs Translators Delays CCC Signals In Signals Out JSC Inputs

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 13 Operate fully functional trigger electronics at CERN Employed in myriad tests & preparation activities Tests in Electronics Integration Center Labs & row of racks for all electronics subsystems Test interfaces & integration as much as possible before move to USC55 MTCC in SX5 Test multiple trigger components with multiple detector components during the magnet test. Verify trigger functions & interfaces w/detectors on surface Installation in Underground Counting Room (USC55) Started Mid-July ‘06 -- “ready for crates” Racks & Infrastructure installed, cooling operational RCT crates, power, infrastructure fully installed (next slide) Passed CERN Safety Review Card installation next Underground Counting Room Trigger Commissioning USC55

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 14 Regional Calorimeter Trigger Crates, Fans, Power Supplies, Monitoring, Controls and other infrastructure installed, validated and passed safety check in July Ready for card installation

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 15 Tech’n. R. Fobes (l.) & Eng’r. T. Gorski (r.): Electronics work in Finish & Test Master Clock Crate Develop detailed timing test system Use 4-crate test-bed in Madison to produce exact procedures Validate Clock Fanout Cards Multiple crate tests Check for data synchronization Repair & Analysis of Board Failures Spare Parts Inventories Install, Commission & Operate Final System in USC55

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 16 Assoc. Scientist P. Klabbers (l.) & Asst. Scientist M. Grothe (r.): (both resident at CERN) Cal Trigger Tasks at CERN - P. Klabbers, M. Grothe & students Retest RCT Crates arriving at CERN Comprehensive long term tests with Jet Capture Card performed in RCT lab in electronics integration center Install RCT Crates in Underground Counting Room Connect & test E,HCAL cables Connect & test to Global Cal. Trigger Install & Test Master Clock system Synchronize all within 4 ns to central timing system Diagnostic & Online Software Downloading detailed test patterns Trigger Emulator Bit-level match to actual electronics Integration with Trigger Supervisor Online Monitoring & Control Configuration of Memory Lookups Reprogramming Function Validation

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 17 Trigger Lab Setup at CERN Staging area - Prevessin 904 Receive boards, crates, cables from Wisconsin Inspect and log in using barcodes and database Assemble crates for Integration/USC55 Power up and run system tests Operate in water-cooled rack for extended tests Integration tests Racks with cooling on a raised floor nearby Storage and Repair Spare Crates and cards also available for use in testing and replacement of suspect cards Soldering station, scope, and tool storage Spare component storage Responsible: P. Chumney & M. Grothe When collisions start need added personnel so group members can actively pursue physics analysis (task B redirection).

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 18 HCAL ECAL RCT GCT GT/ GMT TTC DTTF CSCTF Trigger Integration Progress in Electronics Integration Center (Pr. 904) Large scale successful integration tests in central racks using common TTC infrastructure involving trigger primitives, regional & global processing: Calorimeter Trigger Muon Trigger Global Trigger

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 19 ECAL/HCAL/RCT Integration Tests Successful! HCAL HTR to RCT Used JCC to verify data 6 cables used - all in sync Latency measurements w/PMT as source of HCAL TPG ECAL TCC to RCT Valid link established Channel number sent as data 2 channels on 20 m cables sent in sync to RCT HCAL HTR & ECAL TCC to RCT 28 cables to RCT from 9 TCC & 18 HCAL HTR outputs Common clocking: LTC, TTCci 28 channels in sync - links stable for >10 hours error free running “Hydra” link cable checked as an option for two SLB’s to one RCT RMC ECAL - HCAL Pattern tests RCT can “lookup” incoming data LUTs can be programmed to duplicate incoming data Have ORCA emulation to help predict outputs HCAL HTR ECAL TCC RCT

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 20 RCT Integration Tests Successful! RCT to Local Timing Controller (for MTCC) Tests Receive 56 HCAL links (448 towers) in Green Barracks Send trigger out with RCT Jet Capture Card OR of towers in each half using Quality Bit Validated connection with LTC/GT GT ANDs top & bottom - reduces rate RCT Jet Capture Card to GT Tests Backup Global Calorimeter Trigger JCC in spare VME slot of RCT Crate Currently sends up to 4 trigger bits Threshold  E T, e/  candidate, MIP Tests using RCT pattern successful! ECAL TCC patterns yield JCC output Synchronized to Bunch crossing 0 RCT- GCT Source Card First test with new GCT Completed successfully in July

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 21 Calorimeter Trigger in Cosmic Challenge RCT: Crate fully operational in Green Barracks at Point 5 Cable connections made to HCAL Full modules above and below connected Allows for top-bottom coincidence HCAL crate to RCT RCT input from HCAL RCT output trigger

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 22 Physicist Trigger M&O Tasks (all from base program) Change trigger Configuration Respond to changing beam/detector conditions & physics priorities Study new trigger configurations Test runs, Monte Carlo studies, data studies Trigger Physics Analysis Understand detailed impact of trigger on physics Preparation for luminosity increases Monte Carlo studies of new conditions, validate with present data Respond to changing apparatus Changes in material, configuration, etc.  changes in simulation Operations - 24x7 support during running Write, test & maintain electronics test programs Maintain & update bad channel list & run daily checking programs Run Control maintenance Trigger data validation and calibration Online & Offline analysis of rates & efficiencies Monte Carlo & data trigger simulation maintenance Continuous validation of trigger using simulation & readout data Will need to augment UW personnel to perform tasks & do physics Propose to redirect personnel from Task B (ZEUS)

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 23 The Future - SLHC Luminosity upgrade x10 – SLHC : L = cm -2 s -1 Extends LHC mass reach by ~ 20-30% Modest changes to machine Detector upgrades needed -- especially the trigger & tracker Time scale ~ 2015 Trigger: higher luminosity & tracking data Double operational frequency from 40 MHz to 80 MHz Design for much higher rejection power for pileup to retain output rate of 100 kHz -- more sophistication Correlation of muon & cal triggers with new tracking trigger Status of planning starting in FY04 Workshops: US CMS in Madison Feb ‘04, CMS at CERN Feb ‘04 & Imperial College Jul ‘04, CERN, Jul ‘05 CMS SLHC Level-1 Trigger and Tracker project started. W. Smith & G. Hall (Imperial) chosen to lead effort. Workshop at Perugia April ‘06 proposes plan CMS management accepts plan and charges W. Smith, G. Hall, J. Nash with writing SLHC EOI for submission to CMS management in Sept. and to LHCC by end of the year. As project ramps up, needs additional personnel Use redirection of Task B effort

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 24 UW CMS Trigger Concerns Additional scientific staff to permit physics analysis Present staff will be saturated w/trigger maintenance & operations Will need additional staff to permit physics analysis, use of Wisconsin computing resources. Propose to redirect Task B (ZEUS) staff to meet shortfall Project support of Lead Designer/ Expert Engineer: T. Gorski Vital for commissioning and operations US CMS project support for M&O supervised by base program Estimated technical support project needs for M&O: Based on Zeus Cal. Trigger M&O Technician Operates, repairs, maintains test facility Repairs boards & infrastructure under physicist guidance Total required = 0.5 FTE Expert Engineer ~ 5 trips/year for 2-3 weeks to make difficult repairs Complicated/Subtle problems, Modifications to trigger electronics Designer - available for consultation ~ 2 trips/year for 2-3 weeks for review & design issues

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 25 Trigger Summary Calorimeter Trigger Production & Testing finished Setup of full testing facility/lab at CERN EIC done Successful Integration tests w/ECAL,HCAL, Global Cal Trigger Final Installation started Very good progress, but… Major work from now on: Production Shipping, Retesting, Installation of > 1500 Boards Software Development: online DQM, controls, offline DQM, integration with Trigger Supervisor, Emulator & Simulation More Integration Tests Installation and Commissioning Operations! Physics! SLHC R&D

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 26 Task T Endcap Muon Activities (Talk by R. Loveless) Project Management: Dick Loveless Construction, test, integrate, install & commission 468 CSCs, electronics & infrastructure (& lower all this in ‘06/’07!) Chamber Install., Cabling & Test UW responsibility -- led by UW scientist A. Lanaro, assisted by UW postdoc Y. Baek Low Voltage System Wisconsin responsibility for all chambers & crates -- R. Loveless & UW Eng. A. Golyash Tested and ordered Wiener Maraton system Endcap Alignment Wisconsin responsibility -- Design & procurement complete System installation -- A. Lanaro & Y, Baek Alignment Reconstruction & Corrections -- D. Carlsmith, assisted by J. Bellinger leverage CDF expertise

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 27 Task T Tracking Activities (Talk by R. Loveless) Started July 2005 with responsibility for design of infrastructure for new Tracking Integration Facility (integrate tracker & operate 25% at any time) August 2006: TIF Infrastructure is delivered & installation almost complete  Additional Engineering: Mechanics and installation of Tracker Interface Box (PP1) -- connection point for all pipes, cables, optical fibers inside vacuum cryostat Personnel: Loveless, PSL engineers Dan Wenman & Farshid Feyzi Tracking Reconstruction & Physics Analysis: Implement silicon-only tracking algorithms from CDF into CMS. Use calorimeter & muon systems to identify tracks Personnel: Prof. Herndon, postdoc, 1-2 students

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 28 Task T Physics Activities (Talk by S. Dasu) Preparation for CMS Physics TDR (2006) & early data analysis New energy regime  rapid discoveries Must begin operations (late 2007) with well understood trigger, reconstruction and analysis software Wide-ranging physics interest and expertise in the group Analysis efforts coming to forefront as detector work finishes We studied simulated Higgs, SUSY, SM EW and QCD datasets Prof. Dasu & Dr. Klabbers verified that trigger systems capture all the important physics efficiently while satisfying DAQ bandwidth requirements Ensure trigger systems operate at full performance at turn-on and beyond Prof. Dasu led PTDR.v2 online selection work (L1T and HLT) UW grad student Leonard worked on L1 rates and efficiencies Dr. Grothe wrote the diffractive physics sections Full simulation, trigger design and analysis in new CMS SW Substantial summer student work supervised by Dasu, Herndon & Smith: Anderson (H  ), Grogg (H  W(  e )W(  jj)) Lazaridis (H  Z(  ee)Z(  )), Weinberg (pp  SUSY  Like sign leptons) Herndon (Z  ) for tracking reconstruction studies Creation of physics samples is computationally intensive  Tier-2 effort

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 29 UW Computing Activities (Talk by S. Dasu) Rader [DOE Univ.] & Bradley, Maeir, and Mohapatra [NSF] Support A CMS Tier-2 computer center from FY2005 Data-Intensive Science University Network (DISUN) institute Leadership in Grid Community OSG - Open Science Grid (develop tools and provide cycles) GLOW - Grid Laboratory of Wisconsin (Lead role in development and implementation with CS Condor group to benefit entire UW campus: IceCube, CS, Genomics, Chem. Eng., Med. Phys., etc.) Benefits both CMS and ATLAS + Computing support for all HEP tasks Simulation production using local UW resources The single largest institute producing CMS events Over 60M events produced, simulated, reconstructed & analyzed with ORCA in Over 17 M events produced with new CMSSW since commissioning 2 weeks ago Rader: UW Tier-2 Systems Lead 547 kSI2000 CPU 428 batch slots 90 TB raw storage 10 Gbps bandwidth 24h/7d operation ~200 user accouts Mohapatra : Production czar Bradley: Software expert

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 30 UW Software Activities (Talk by S. Dasu) Trigger Controls Software Prof. Dasu: Core VME software and framework Dr. Klabbers: Testing and MTCC operations software Dr. Grothe: Integration with CMS Trigger Supervisor Trigger Emulator Software UW Student Leonard: RCT emulator in CMSSW Trigger Monitoring Software New effort needed now to develop under supervision of Klabbers Trigger Physics Studies New effort needed now to ensure good trigger for first physics Computing Resources Management Software SW Engineer Bradley: New scheduler, Computing-On-Demand Contributions to Condor, OSG and ROOT projects New Effort: Postdoc redirection from Task B and student support (Leonard full year, others from summer 07)

Wesley Smith, U. Wisconsin, August 22, 2006 DOE Review: Task T: CMS at LHC - 31 Wisconsin Concerns DOE Core Program Manpower at Wisconsin Major effort on trigger software required Tasks: board testing, monitoring/controls, diagnostics, configuration downloads & documentation, modeling, physics simulation, etc. In US CMS Plan as core program manpower Major effort on testing, installation, integration tests & commissioning Planned as activity of base program manpower For Trigger, Computing,, Muon Projects (and now Tracker) Project-supported personnel need core program supervision Wisconsin responsibilities are growing with time! Long term trigger M&O + Physics Analysis + SLHC Increase in personnel needed Particularly so physicists can participate in the physics harvest Needs continued support & eventual growth Provide growth from Task B redirection