J. Patrick - Major Challenges/MOPA021 – a new Remote Operations Center at Fermilab J. Patrick, et al Fermilab

J. Patrick - Major Challenges/MOPA022 AbstractAbstract Commissioning the LHC accelerator and experiments will be a vital part of the worldwide high-energy physics program beginning in A remote operations center, has been built at Fermilab to make it easier for accelerator scientists and experimentalists working in North America to help commission and participate in operations of the LHC and experiments. Evolution of this center from concept through construction and early use will be presented as will details of its controls system, management, and expected future use.

J. Patrick - Major Challenges/MOPA023 ContentsContents Introduction Concept Design Construction Early Use Details/Special Features Future Plans Summary & Acknowledgements

J. Patrick - Major Challenges/MOPA024 What is A Place That provides access to information in a manner that is similar to what is available in control rooms at CERN Where members of the LHC community can participate remotely in CMS and LHC activities A Communications Conduit Between CERN and members of the LHC community located in North America An Outreach tool Visitors will be able to see current LHC activities Visitors will be able to see how future international projects in particle physics can benefit from active participation in projects at remote locations.

J. Patrick - Major Challenges/MOPA025 What is Allow experts located at Fermilab to participate in CMS and LHC commissioning and operations. Hardware and software necessary to participate effectively in CMS and LHC. Facilitate communication and help members of the LHC community in North America contribute their expertise to CMS and LHC. An extension of the CERN Control Centre (CCC). For example, to assist members of US/LARP in training and data analysis. An extension of the CMS Control Room. For example, to provide a call center for US-CMS collaborators to access information about CMS and the LHC accelerator. A unique opportunity to have detector and accelerator experts in close proximity to each other solving problems together.

J. Patrick - Major Challenges/MOPA026 Remote operations for LHC and LARP CCC LHC remote operations: training prior to stays at CERN remote participation in studies ‘service after the sale’: to support accelerator components built in the U.S. access to monitoring information software development for LHC controls system (LAFS) CCC at CERN LARP: The US LHC Accelerator Research Program (LARP) consists of four US laboratories, BNL, FNAL, LBNL and SLAC, who collaborate with CERN on the LHC. The LARP program enables U.S. accelerator specialists to take an active and important role in the LHC accelerator during its commissioning and operations, and to be a major collaborator in LHC performance upgrades.

J. Patrick - Major Challenges/MOPA027 How did the Concept Evolve? Fermilab has contributed to CMS detector construction, hosts the LHC Physics Center (LPC) for US-CMS, is a Tier-1 grid computing center for CMS, has designed and fabricated LHC machine components, is part of the LHC Accelerator Research Program (LARP), and is involved in software development for the LHC controls system through a collaboration agreement with CERN called Software (LAFS). The LPC had always planned for remote data quality monitoring of CMS during operations. Could we expand this role to include remote shifts? LARP was interested in providing support for US-built components, training people before going to CERN, and remote participation in LHC studies. We saw an opportunity for US accelerator scientists and engineers to work together with detector experts to contribute their combined expertise to LHC & CMS commissioning. The idea of joint remote operations center at FNAL emerged

J. Patrick - Major Challenges/MOPA028 ConceptConcept Some proof of principle work done by LHC/LARP personnel Thanks to AB/OP colleagues at CERN CMS Remote Operations Center Unified Task Force formed at request of FNAL Director First meeting 4 May 2005 Close-out 19 October 2006 Requirements document created and reviewed CMS LHC CMS/LHC combined Constraints 63 total requirements Review 21 July 2005 Proposal to Directorate and constitutients Construction Authorization and Engineering May 2006 Construction initiated September 2006

J. Patrick - Major Challenges/MOPA029 Site Visits Technology Research, Education, and Commercialization Center (TRECC) – West Chicago, Illinois (Aug. 25, 2005) Gemini Project remote control room – Hilo, Hawaii (Sept. 20, 2005) Jefferson Lab control room – Newport News, Virginia (Sept. 27, 2005) Hubble Space Telescope & STScI – Baltimore, Maryland (Oct. 25, 2005) National Ignition Facility – Livermore, California (Oct. 27, 2005) General Atomics – San Diego, California (Oct. 28, 2005) Spallation Neutron Source – Oak Ridge, Tennessee (Nov. 15, 2005) Advanced Photon Source – Argonne, Illinois (Nov. 17, 2005) European Space Operations Centre – Darmstadt, Germany (Dec. 7, 2005)

J. Patrick - Major Challenges/MOPA0210 DesignDesign Design work done in-house Variation of CCC design due to purpose High Visibility location preferred Laboratory Director Adjacent to meeting and office areas Provide Security Maintain Privacy Special Features Storefront/mullion-free glass Projection Wall & Screens Privacy glass between center and adjacent conference room Window treatment - morning glare

J. Patrick - Major Challenges/MOPA0211 LocationLocation Wilson Hall Main Entrance Cafeteria

J. Patrick - Major Challenges/MOPA0212 RenderingsRenderings

J. Patrick - Major Challenges/MOPA0213 ConsolesConsoles Three Bids submitted Consider Cost & Specifications Same Vendor as for CCC selected

J. Patrick - Major Challenges/MOPA0214 Construction Summary Safety No injuries One incident On-time 12-week schedule Under budget

J. Patrick - Major Challenges/MOPA0215 Construction Slide Show

J. Patrick - Major Challenges/MOPA0216 ComputingComputing Separate Computing Systems/Platforms for Consoles Outreach Videoconferencing Projectors Gateway Server Protected access as appropriate

J. Patrick - Major Challenges/MOPA0217 Early Use Current Organization Engineering Working Group Operations Support Team CMS WorkingGroup LHC Working Group Outreach Working Group Primary user to date is CMS Tier-1 Computing support Remote shifts by Tracking group from March - May Test beam activities by HCAL group Currently Global Commissioning runs CMS data operations - ‘CSA07’ computing challenge LARP SPS Beam Study period LHC Hardware Commissioning Software Applications development

J. Patrick - Major Challenges/MOPA0218 Noteworthy Features Features that are currently available: CERN-style consoles with 8 workstations shared by CMS & LHC Videoconferencing installed for 2 consoles, can be expanded to 4 consoles Webcams for remote viewing of Secure keycard access to Secure network for console PCs (dedicated subnet, physical security, dedicated router with Access Control Lists to restrict access, only available in 12-minute video essay displayed on the large “Public Display” used by docents from the Education Department to explain CMS and LHC to tour groups High Definition (HD) videoconferencing system for conference room HD viewing of and HD display capabilities in the centre Secure group login capability for consoles, with persistent console sessions Role Based Access Control (RBAC) for the LHC controls system (LAFS) Screen Snapshot Service (SSS) for CMS and the LHC controls system

J. Patrick - Major Challenges/MOPA0219 Role Based Access Control (RBAC) An approach to restrict system access to authorized users. What is a ROLE? A role is a job function within an organization. Examples: LHC Operator, SPS Operator, RF Expert, PC Expert, Developer, … A role is a set of access permissions for a device class/property group Roles are defined by the security policy A user may assume several roles What is being ACCESSED? Physical devices (power converters, collimators, quadrupoles, etc.) Logical devices (emittance, state variable) What type of ACCESS? Read: the value of a device once Monitor: the device continuously Write/set: the value of a device Status: Deployed at the end of June 2007 This is a FNAL/CERN collaboration (LAFS) working on RBAC for the LHC control system. See Posters TPPA04, TPPA12, WPPB08 Details & Special Features The software infrastructure for RBAC is crucial for remote operations in that it provides a safeguard. Permissions can be setup to allow experts outside the control room to read or monitor a device safely.

J. Patrick - Major Challenges/MOPA0220 Screen Snapshot Service (SSS) An approach to provide a snapshot of a graphical interface to remote users. What is a snapshot? An image copy of a graphical user interface at a particular instance in time. Examples: DAQ system buffer display, operator control program, … A view-only image, so there is no danger of accidental user input. Initially implemented for desktops, but could be targeted to application GUIs. What is the role of the service ? Receives and tracks the snapshots from the monitored applications. Caches the snapshots for short periods of time. Serves the snapshots to requesting applications/users. Prevents access from unauthorized applications/users. Acts as a gateway to private network applications for public network users. How does this work? Applications capture and send snapshots to the service provider in the background. Users would access snapshots using a web browser. Status: SSS is being used by CDF for operations. SSS is being used by CMS Silicon Tracker Group, and is installed at CERN and in the ROC. More information at: Details & Special Features Monitored Application(s) Snapshot Service Web Browser(s) snapshots requests

J. Patrick - Major Challenges/MOPA0221 Future Plans Ramp up CMS shifts LHC Hardware Commissioning Keep LHC Project Associates engaged after return US/LARP deliverable monitoring LAFS Continue Applications development LHC Beam Participation SPS and other Injector Beam Studies LARP Instrumentation LHC Commissioning and Beam Studies (especially for luminosity upgrades)

J. Patrick - Major Challenges/MOPA0222 SummarySummary Remote operations is the next step to enable collaborators to participate in operations from anywhere in the world. The goals are to have: secure access to data, devices, logbooks, monitoring information, etc.; safeguards, so actions do not jeopardize or interfere with operations; collaborative tools for effective remote participation in shift activities; remote shifts to streamline operations. Fermilab has built the Remote Operations Center, which is shared by scientists and engineers working on the LHC and CMS. Collaborative Design Built rapidly For the LHC it provides a means to participate remotely in LHC studies, access to monitoring information, a training facility, and supports the collaborative development of software for the LHC controls system. For CMS it provides a location (in a U.S. time zone) for CMS remote commissioning and operations shifts, and Tier-1 grid monitoring shifts. Already a popular stop for visitors and dignitaries.

J. Patrick - Major Challenges/MOPA0223 AcknowledgementsAcknowledgements Task Force Especially Erik Gottschalk from whom many slides were taken Design, Engineering, and Construction Team from Fermilab/FESS Gary Van Zandbergen Steve Dixon Merle Olson Tom Prosapio CERN AB/OP Especially Djanko Manglunki Staff of Sites Visited Users of Facility