CMS Centres for Control, Monitoring, Offline Operations and Analysis Lucas Taylor, Northeastern University, Boston Patricia McBride, Kaori Maeshima, Erik Gottschalk Fermilab CHEP’07, Victoria 2-7 September 2007 Abstract The CMS experiment is about to embark on its first physics run at the LHC. To maximize the effectiveness of physicists and technical experts at CERN and worldwide and to facilitate their communications, CMS has established several dedicated and inter-connected operations and monitoring centers. These include: a traditional Control Room at the CMS site in France; a CMS Centre for up to 50 people on the CERN main site in Switzerland, and remote operations centers, such as the LHC@FNAL center at Fermilab. We describe how this system of centers coherently supports the following activities: CMS data quality monitoring, prompt sub-detector calibrations, and time-critical data analysis of express-line and calibration streams; and operation of the CMS computing systems for processing, storage and distribution of real CMS data and simulated data, both at CERN and at offsite centers. We describe the physical infrastructure that has been established, the computing and software systems, the operations model, and the communications systems that are necessary to make such a distributed system coherent and effective. Lucas Taylor

LHC & CMS startup LHC CMS First collisions at 14 TeV Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 LHC & CMS startup Completion of magnet cryostating and tests Descent of last magnet Sector 7-8 cooled to 1.9 K along 3 km All technical systems commissioned to 7 TeV operation & machine closed LHC Beam commissioning starts First collisions at 14 TeV 2010 2008 2006 2007 2009 2005 (0.1 – 1) fb-1 (1 – 10s) fb-1 (10s - 100) fb-1 CMS commissioning, data quality monitoring, calibration, alignment, Offline computing operations … CMS Complete CMS closed and ready for extended physics run CSA’07 – final Computing Challenge Cosmic challenge on surface Lucas Taylor

LHC & CMS startup - human implications Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 LHC & CMS startup - human implications Jim Virdee described the challenges of the new LHC machine, new CMS detector and new physics … Fabiola Gianotti (Interlaken) identified some of the corresponding human difficulties we will face “O(10**3) physicists in panic-mode using and modifying the software and accessing the database, GRID …” “…at the beginning they will be confronted with most atypical (and stressful) situations, for which a lot of flexibility will be needed” I’ll describe how CMS Centres address this By co-locating sub-detector offline experts (CERN, FNAL…) By hosting CMS Computing operations teams By giving all 3000 collaborators live access to monitoring information By ensuring communications systems are effective Lucas Taylor

CMS Centers CMS Control Room Control Room Operates CMS Slow control Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centers CMS Control Room Traditional CMS Control of CMS Detector Operations Control Room Operates CMS Slow control Data acquisition Data transfer to Tier-0 Sub detectors: Operate detector Calibrations Data-quality monitoring Constants to HLT Online monitoring CERN Tier-0 IN2P3 FNAL CNAF PIC ASGC GridKA RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 Lucas Taylor

CMS Centers CMS Control Room CMS Centre and LHC@FNAL CMS Centre Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centers CMS Control Room CMS Centre and LHC@FNAL Link to Control room Mirror displays Communications Computing Operations Tier-0 production Data storage / transfer Sub detectors Data quality monitoring (also post Tier-0) Calibration Good/bad runs Software fixes Express analysis Novel CMS Control of CMS Detector Operations Online monitoring CMS Centre (at CERN) CERN Tier-0 Control of CMS Computing Operations IN2P3 Offline monitoring FNAL CNAF PIC ASGC GridKA T2 LHC@FNAL RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 Other centres (?) Lucas Taylor

Completed: Feb. 8, 2007 Open House: Feb. 12, 2007 LHC@FNAL Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 LHC@FNAL Started operations this year Joint centre for CMS & LHC Builds on FNAL Remote Operations Centre used in CMS Cosmic challenge LHC@FNAL features 4 CERN-style consoles (8 workstations) Videoconferencing for 2 consoles Webcams for remote viewing of room Secure access, secure network High Definition (HD) Systems Videoconferencing, Webcams… Role Based Access for LHC controls Screen Snapshot Service (SSS) Completed: Feb. 8, 2007 Open House: Feb. 12, 2007 Lucas Taylor

LHC@FNAL Tier-1 operations team Detector groups, e.g. Si tracker Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 LHC@FNAL Tier-1 operations team Responsible for FNAL Tier-1 Weekday, business hour shifts Central support for seven Tier-2 centers (Universities) Detector groups, e.g. Si tracker Remote shifts, Feb–Jun 2007 Worked with CERN tracker on Data Quality Monitoring Data transfer/bookkeeping Event display, DCS,.... Outreach Visits and 12 minute outreach video and photos from CERN Lucas Taylor

Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN Mid-way through construction; when complete will include New office space for ~250 people Main Monitoring & Operations Room (300 sq. m) 22 consoles with ~5 screens each CMS Computing Operations Rooms Already in use by ~15 operators Meeting Rooms Auditorium plus ~ 6 smaller rooms All with Polycom conf. phone, some with Tandberg VC equipment Miscellaneous rooms: outreach, WiFi visitors room / training centre, rest area with kitchen, locker room… Lucas Taylor

CMS Centre at CERN Main Room Outreach Room Meeting Rooms Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN Main Room Outreach Room Meeting Rooms Lucas Taylor

Lower row of screens for working Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN Lower row of screens for working Top row of screens for monitoring (e.g. displays from Control Room) Lucas Taylor

CMS Centre at CERN Will be in the former PS Main Control Room (below) Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN Will be in the former PS Main Control Room (below) Machine controls now in the Prevessin “CERN Control Centre” (CCC) Lucas Taylor

CMS Centre at CERN … today Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN … today Detailed requirements study Technical studies completed Ergonomics, acoustics, fire, radiation, lighting, etc. Green light for construction Lucas Taylor

CMS Centre at CERN … today Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN … today Technical installations underway 22 desks, ~110 screens Network (2 * 10Gbps uplinks) Lots of wireless 40 kW electrical power Air-conditioned under-floor iced water to 20 peripheral fan coil units Acoustic insulation Carpet, fibre-glass in ceiling and consoles Lighting Natural, strips, spots Dimmable and zoned Outreach displays Lucas Taylor

Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Centre at CERN When completed (March 2008) the CMS Centre will look very similar to the CERN-LHC Control Centre shown below Lucas Taylor

CMS Computing Operations Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 CMS Computing Operations Related CHEP Talks No. 81 Computing Operations at CMS Facilities No. 290 CMS Tier0 - design, implementation and first experiences No. 369 CMS Experiences with Computing Software and Analysis Challenges No. 370 Development of the Tier-1 Facility at Fermilab No. 266 CMS Offline Web Tools CMS “Data Operations” teams Main team at CMS Centre, CERN (15 people) Second main team at LHC@FNAL Ensures CMS workflows function ~24 / 7 Control and monitoring systems Increasing use of “WebTools” Initiate transfers, data placement and removal, production systems… Communications in-person and by telephone and IM with shared desktops Web, some VNC, maybe Webex/EVO (?) Screen Snapshot Service (SSS) for exporting arbitrary displays to the Web Used by CDF, CMS, maybe LHC Web Browser(s) snapshots requests Snapshot Service snapshots Monitored Application(s) http://cmstacwww.cern.ch:8082/snapshot/ShowimageList.jsp http://www-cdfonline.fnal.gov/java/snapshot/ShowimageList.jsp Lucas Taylor

Data Quality Monitoring Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 Data Quality Monitoring Web-based DQM GUI gives access to information in Control Room, offline centres, institutes, Web café, iPhone… Live shifter views - canned with reference histograms Expert exploration of repository of 1000s of detailed histograms Both real-time and offline data DQM systems being refined with practical experience from monthly commissioning runs Related CHEP Talks No. 221 CMS Online Web Based Monitoring No. 279 The Run Control and Monitoring System of the CMS Experiment No. 253 CMS Event Display and Data Quality Monitoring for LHC Startup No. 266 CMS Offline Web Tools No. 434 Data Quality Monitoring and Visualization for the CMS Silicon Strip Tracker No. 268 Data Quality Monitoring for the CMS Electromagnetic Calorimeter Lucas Taylor

Communications WWW Displays Phone Videoconferencing WebCams Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 WWW WebTools, DQM, SSS, … Communications Displays Local applications, DQM web clients, SSS mirrors Phone Headset per console Conference phones Videoconferencing Adjacent meeting room(s) Possibly in main room WebCams Maintain sense of proximity Use with phone for point to point “video conferencing” iMac - under consideration Computer, screen, camera, microphone and iChat for IM and few-person phone-/video-conferencing CMS Control Room mirror mirror ? CMS Centre at CERN LHC@FNAL mirrors Related CHEP talks No. 35 Collaborative Tools and the LHC: An Update No 408 Shaping Collaboration 2006: Action Items for the LHC Lucas Taylor

Summary CMS Centres are currently being established Lucas Taylor CHEP'07, Victoria 2-7 Sept 2007 Summary CMS Centres are currently being established “CMS Centre” at CERN, LHC@FNAL, possibly others … These centres will enhance communication and access to information, thereby helping all 3000 CMS collaborators to play their part in commissioning Offline data quality monitoring, calibrations, express analysis CMS Computing Operations …which will, in turn, increase CMS competitiveness when we start taking LHC data in July 2008 at a new energy frontier Lucas Taylor