CMS Centres for Control, Monitoring, Offline Operations and Analysis Lucas Taylor, Northeastern University, Boston 27 November 2007 Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Online (data quality) monitoring CMS Centres CMS Control Room Slow control, safety Operates detectors Calibrations Data-quality monitoring Data acquisition Data transfer to Tier0 CMS Centre and LHC@FNAL Link to Control room Mirrors displays Communications Computing Operations Tier-0 production Data storage / transfer Sub detectors Data quality monitoring (also post Tier-0) Calibration Good/bad runs Software fixes Express analysis CMS Control Room (Online) at Cessy Control of CMS Detector Operations CMS Online (data quality) monitoring Offline / Computing (data quality) monitoring Other Centres (?) LHC@FNAL CMS Centre (offline) at CERN Control of CMS Offline / Computing Operations CERN Tier-0 IN2P3 FNAL CNAF PIC ASGC GridKA T2 RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN, LHC@FNAL, … Large rooms to co-locate offline experts Physicists working on DQM, calibration, rapid data analysis Expert computing and offline operations teams IT facilities include Monitoring displays, IT systems Web-based applications Give all CMS collaborators live access to latest information Communications systems connecting CMS Control Room CMS Centre, LHC@FNAL, others such centres (DESY?) CMS institutes Tier-1 and Tier-2 centres Grid operations centres (?) CERN Control Centre (CCC) for the LHC CMS Centre will be similar Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Located in the former PS Main Control Room Main Room: 300 sq. m To be refurbished Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Main room: 25-50 people, 22 consoles, 110 screens Computing Operations Room ~ 15 people (done) New offices: ~250 people Video-conferencing (auditorium and ~ 6 meeting rooms) Outreach room, visitors room, lockers, rest area, kitchen… Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Will be in the former PS Main Control Room (below) Machine controls now in the Prevessin “CERN Control Centre” (CCC) Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Main Room Outreach Room Meeting Rooms Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Lower row of screens for working CMS Centre at CERN Lower row of screens for working Top row of screens for monitoring (e.g. displays from Control Room) Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Relocation of existing personnel / removal of everything in room CMS Centre at CERN June 2007 Relocation of existing personnel / removal of everything in room Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN July 2007 Major upgrade of electrical power 40 kW iced-water cooling system with 22 fan-coil heat exchangers Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Opened window from conference room CMS Centre at CERN August 2007 Opened window from conference room 100 fixed network connections (Cat-6 cable) on GPN with 2*10 Gbps uplinks 25 analogue phone lines Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Sept-Oct 2007 Ventillation / heating in ceiling Concrete: coring for services and closure of fire-breaks Acoustic insulation in ceiling Repairs / levelling of false floor Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Carpet tiling (permits re-lifting of false floor) CMS Centre at CERN Nov 2007 Painting / repairs throughout Fire safety systems Video-conferencing Carpet tiling (permits re-lifting of false floor) Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Nov 2007 New door (fire exit) Lighting repairs Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Nov 2007 Construction of consoles Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Nov 2007 Construction of consoles Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre at CERN Nov 2007 (today) Construction of consoles Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Centre Technical Installations Past: PS Main Control Room Present: Installation of CMS Centre Future (Mar 2008): CMS Centre Similar to this: LHC Control Room Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

LHC@FNAL Completed: Feb. 8, 2007 Open House: Feb. 12, 2007 Started operations this year Joint centre for CMS & LHC Builds on FNAL Remote Operations Centre used in CMS Cosmic challenge LHC@FNAL features 4 CERN-style consoles (8 workstations) Videoconferencing for 2 consoles Webcams for remote viewing of room Secure access, secure network High Definition (HD) Systems Videoconferencing, Webcams… Role Based Access for LHC controls Screen Snapshot Service (SSS) Completed: Feb. 8, 2007 Open House: Feb. 12, 2007 Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

LHC@FNAL Tier-1 operations team Detector groups, e.g. Si tracker Responsible for FNAL Tier-1 Weekday, business hour shifts Central support for seven Tier-2 centers (Universities) Detector groups, e.g. Si tracker Remote shifts, Feb–Jun 2007 Worked with CERN tracker on Data Quality Monitoring Data transfer/bookkeeping Event display, DCS,.... Outreach Visits and outreach video Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations CMS “Data Operations” teams Main team at CERN (15 people) Second main team at LHC@FNAL Ensures CMS workflows work ~24 / 7 Will also use CMS Centre when ready Computing operations Building 8 1st Floor Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations (Obvious) strategy for monitoring and control: Web-based GUIs Computing centres and CMS collaborators spread globally Location and platform independence CMS “WebTools” project Builds on Yahoo (YUI) toolkit Currently in use for Data production request system Data transfers / management Phedex, DBS, SiteDB … More applications coming (DQM) CMS CSA06: ~1 PB in ~1 month >12 PB in ~6 months Related CHEP Talks No. 81 Computing Operations at CMS Facilities No. 290 CMS Tier0 - design, implementation and first experiences No. 369 CMS Experiences with Computing Software and Analysis Challenges No. 370 Development of the Tier-1 Facility at Fermilab No. 266 CMS Offline Web Tools Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations Screen Snapshot Service (SSS) Runs on arbitrary machine (e.g. shifter in control room or operator console) Takes snapshots of screens and exports them to Web Remote people can watch monitoring screens with a Web browser Used by CDF, CMS, maybe LHC Web Browser(s) requests snapshots Snapshot Service snapshots Monitored Application(s) http://cmstacwww.cern.ch:8082/snapshot/ShowimageList.jsp http://www-cdfonline.fnal.gov/java/snapshot/ShowimageList.jsp Related CHEP Talks No. 81 Computing Operations at CMS Facilities No. 290 CMS Tier0 - design, implementation and first experiences No. 369 CMS Experiences with Computing Software and Analysis Challenges No. 370 Development of the Tier-1 Facility at Fermilab No. 266 CMS Offline Web Tools Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations: Data Quality Monitoring Web-based Graphical User Interface Building on CMS WebTools Top-level (shift) views Reference plots Repository of expert plots Universal uniform access Control Room, offline centres, institutes, Web café … Related CHEP Talks No. 221 CMS Online Web Based Monitoring No. 279 The Run Control and Monitoring System of the CMS Experiment No. 253 CMS Event Display and Data Quality Monitoring for LHC Startup No. 266 CMS Offline Web Tools No. 434 Data Quality Monitoring and Visualization for the CMS Silicon Strip Tracker No. 268 Data Quality Monitoring for the CMS Electromagnetic Calorimeter Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations: Data Quality Monitoring Deployed and refined in monthly commissioning runs Will gradually migrate non-Web applications to Web Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

CMS Operations: Data Quality Monitoring (Iguana) Iguana widely used for Magnet test & cosmic challenge Online commissioning Offline debugging / analysis Nice pictures, press, outreach Web-based Iguana prototype Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Communications Strategic importance for 2500 remote physicists Displays DQM web clients (location independent) Local systems (e.g. trouble-shooting) SSS snapshots (mirror remote screens) Phone 1 headset + speaker-phone per console Conference phones Videoconferencing Main room: open connection to Control Room, CMS Centre, LHC@FNAL … Also adjacent meeting room(s) WebCams (on big screens) Maintain sense of proximity Use with phone for point to point “video conferencing” iMac - under consideration Computer, screen, camera, microphone iChat for IM and few-person video-conferencing Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Excellent CERN / IT support ! Communications Robert Aymar CERN Director General Jim Virdee CMS Spokesperson HD video communications Recent demonstration of effectiveness CERN <-> USA LHC@FNAL inaugurated with live video link to CMS and LHC Crucial for remote physicists, especially as we start to take data and do physics analysis LHC@FNAL Excellent CERN / IT support ! Ray Orbach, US Under Secretary of Science at Fermilab CMS surface hall Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007

Summary CMS Centres are currently being established Functions Main “CMS Centre” at CERN (Meyrin) Remote centres: LHC@FNAL, probably DESY, maybe others (?)… Functions Offline data quality monitoring, calibrations, express analysis CMS Computing / Offline Operations … liaison with Tier0, 1, 2 sites Emphasize communication and access to information Enable all 3000 CMS collaborators to play their part in commissioning … and hence … Increase CMS competitiveness when we start taking LHC data in July 2008 at a new energy frontier Lucas Taylor WLCG Meeting, CERN, 27 Nov 2007