CMS Michele Gulmini, CHEP2003, San Diego USA, March Run Control and Monitor System for the CMS Experiment Michele Gulmini CERN/EP – INFN Legnaro On behalf of the CMS DAQ collaboration CHEP 2003, San Diego USA, March 2003

CMS Michele Gulmini, CHEP2003, San Diego USA, March Outline Run Control and Monitor System : RCMS RCMS Architecture Session Managers Subsystem Controllers Services RCMS Prototypes RCMS for Small DAQ Systems RCMS Demonstrators –Performance and Scalability Tests Plans Summary

CMS Michele Gulmini, CHEP2003, San Diego USA, March Run Control and Monitor System RCMS Internet Intranet Internet Intranet UI The Run Control and Monitor System (RCMS) is the collection of hardware and software components responsible for controlling and monitoring the CMS experiment during the data taking. RCMS enables users to access and control the experiment from any part in the world providing a “virtual counting room”, where physicists and operators can effectively taking shifts from a distance. RCMS views the experiment as a set of partitions, where a partition is a grouping of entities that can be operated independently. Main operations are configuration, monitoring, error handling, logging and synchronization with other subsystems. RCMS Trigger Event Builder Event Filter DCS Computing Services UI RCMS Context

CMS Michele Gulmini, CHEP2003, San Diego USA, March Partitions Example Session Manager-A UI Services Connection Services Services CS Ctrl TRG Ctrl DCS Ctrl EVF Ctrl CS Sub- System Glbl Mu Cal DCS Sub- System EVB Sub-System TRG Sub-System EVF Sub-System Session Manager-B UI EVB Ctrl FED Builder Sub-System RU Builder Sub-System FED-B Ctrl RU-B Ctrl

CMS Michele Gulmini, CHEP2003, San Diego USA, March RCMS Logical Layout The execution of the RCMS is organized on the basis of “Sessions”. A Session is the allocation of the hardware and software of a CMS partition needed to perform data-taking. Multiple Sessions may coexist and operate concurrently Each Session is associated with a Session Manager (SMR), that coordinates all the actions

CMS Michele Gulmini, CHEP2003, San Diego USA, March Sub-System Controller (SSC) A SSC consists of a Function Manager (FM) and a local database (DB) service. There is one FM per partition that receives requests from a Session Manager (SMR) and transforms them into the corresponding requests for actions that are sent to the sub-system. The local DB service can be used as a proxy to the services.

CMS Michele Gulmini, CHEP2003, San Diego USA, March Basic RCMS Services –SECURITY SERVICE login and user account management; –RESOURCE SERVICE (RS) information about DAQ resources and partitions; –INFORMATION AND MONITOR SERVICE (IMS) Collects messages and monitor data; distributes them to the subscribers; –JOB CONTROL Starts, monitors and stops the software elements of RCMS, including the DAQ components –PROBLEM SOLVER Uses information from the RS and IMS to identify mulfunctions and attempts to provide automatic recovery procedures where applicable

CMS Michele Gulmini, CHEP2003, San Diego USA, March Resource Service Block Diagram The Resource Service (RS) handles all the hardware and software components of the DAQ system including its partitions. SS UserDB RS IMS Job Ctrl PS SSC Services Connection Session Manager RCMS UI ConfDB LogDB

CMS Michele Gulmini, CHEP2003, San Diego USA, March Information and Monitor Service Block Diagram The Information and Monitor Service (IMS) collects the information (log, warning, errors, monitoring, etc.) from the sub-systems and provides them to the subscribers. SS UserDB RS IMS Job Ctrl PS SSC Services Connection Session Manager RCMS UI ConfDB LogDB

CMS Michele Gulmini, CHEP2003, San Diego USA, March Time Requirements –Configuration and setup of the system: minutes –Control (state change, execution of commands): seconds –Monitoring: depending on the amount of data required Information and Monitor Service: Tens of subscribers Peak: about 2000 messages (status change, log) Average: Tens to a few hundred messages/s

CMS Michele Gulmini, CHEP2003, San Diego USA, March RCMS Prototypes RCMS for small DAQ Systems –Fully functional RCMS systems targeted to small DAQs (Production systems, Testbeam DAQ systems) –Real-life examples used to check the RCMS functionality. RCMS demonstrators –Partially functional RCMS systems targeted to prove scalability issues. –Test bed systems used to emulate slices or parts of the hierarchical structure of the final DAQ. –Help to confirm the architecture and to evaluate and eventually select the technologies to be used in the final system.

CMS Michele Gulmini, CHEP2003, San Diego USA, March RCMS for small DAQs Current Running Prototype: –Designed to work together with XDAQ CMS online software framework (XDAQ: See Chep2003 J. Gutleber talk - “Using XDAQ in Application Scenarios of the CMS Experiment”) –Available services: Resource Service (RS) Information and Monitor Service (IMS) SubSystem Controllers (Function Managers) Session Managers GUIs Technologies and tools: Java Servlets (Apache Tomcat) Sun “Java Web Services Developer Package” (JWSDP) –JAXP, JAXM, XPath,... SOAP communication protocol Databases –XMLDB interface »eXist native XML database –mySQL

CMS Michele Gulmini, CHEP2003, San Diego USA, March RCMS for Small DAQs – Current Applications CMS Muon Drift Tubes Chamber Production DAQ (Legnaro - Italy) Testbeam (CERN – next May) CMS Tracker “ROD System Tests” (CERN) Testbeam (CERN – next May) CMS TriDAS (CERN) DAQ Column TDR Demonstrator

CMS Michele Gulmini, CHEP2003, San Diego USA, March Session and Function Manager Prototype SS UserDB RS IMS Job Ctrl PS SSC Services Connection Session Manager RCMS UI ConfDB LogDB XML definition JavaImplementation FSM SM/FM servlet Function Managers and Session Manager have a built in Finite State Machine (FSM) to command the controlled components, and to track their state; The FSM is composed of a XML definition and a Java class implementation representing the actions to be performed; The definition and the implementation of the FSMs are managed by the Resource Service; Session Manager and Function Managers are launched when a new “Session” is opened, and can have a hierarchical structure;

CMS Michele Gulmini, CHEP2003, San Diego USA, March RS and IMS Prototype IMS Xpath Filter Engine JAXM XML message Tomcat servlet container NOTIFY PUBLISH Subs Info JDOM FS Java Publisher JAXM Java Subscriber JAXM SUBSCRIBE Tomcat/Jetty Soap DB (eXist, File,mySQL) XDAQ Application XOAP XMLDB Servlet container (TOMCAT) Java Servlet Resource Service XML Java client Java Objs XML Parser (CASTOR) XML Parser C++ client XML Parser XML DB Inter face REL DB XML:DB SOAP

CMS Michele Gulmini, CHEP2003, San Diego USA, March RCMS GUIs Generic GUI: –Insertion and retrieval of resources (PCs, software, partitions, etc.) –Ability to command, set and retrieve parameters from XDAQ applications –Scripting facility –Customisation facilities (plugins) Muon DT TestBeam GUI

CMS Michele Gulmini, CHEP2003, San Diego USA, March Legnaro T2 CMS farm: 136 P GHz processors RCMS Demonstrators

CMS Michele Gulmini, CHEP2003, San Diego USA, March Demonstrator 1 Exploring the ability to command a set of XDAQ executives running “empty” applications The time measured represents the time required to perform a state change of the entire cluster..... XDAQ PC XDAQ FM FM: Function Manager..... XDAQ XDAQ FM..... XDAQ XDAQ FM FM PC SOAP

CMS Michele Gulmini, CHEP2003, San Diego USA, March Demonstrator nodes 100 ms

CMS Michele Gulmini, CHEP2003, San Diego USA, March Demonstrator 2 Simplified version of a log message service based on Web Services technologies (Glue platform) 15 clients and a variable number of Web Services used The performance scales linearly with the number of instances of the service available

CMS Michele Gulmini, CHEP2003, San Diego USA, March IMS Prototype Test (I) PUBLISH IMS IMS IMS Publisher mySQL DB Percistency on eXist XML native DB not plotted – very slow Between 200 and 300 SOAP messages/s handled by the IMS prototype

CMS Michele Gulmini, CHEP2003, San Diego USA, March IMS Prototype Test (2) PUBLISH NOTIFY IMS IMS IMS IMS IMS Publisher IMS IMS Subscriber SUBSCRIBE mySQL DB SOAP Performance improves augmenting the number of service instances Notification mechanism not optimized Test to be completed

CMS Michele Gulmini, CHEP2003, San Diego USA, March IMS hierarchical structure –Performance test done with the present prototypes: –Commanding a cluster of DAQ application fits the requirements –Information and Monitor Service prototype needs further investigation –Notification architecture –Hierarchical structure IMS hierarchical structure:..... IMS IMS proxy XDAQ XDAQ..... XDAQ XDAQ

CMS Michele Gulmini, CHEP2003, San Diego USA, March Future – OGSA??? RCMS architecture is service and web oriented Web services development tools (Apache Axis, Glue) may help to deploy reliable services quickly Open Grid Service Architecture (OGSA) ( is Web Services based An alpha release of the framework is now available First official release foreseen in a few months time OGSA could be adopted for the RCMS services, providing several advantages: RCMS open to the Grid world Well supported and reliable framework Useful built-in services OGSA is under evaluation: The RCMS Resource Service has been successfully ported (Globus 3.0 alpha release) functionality and performance tests in progress

CMS Michele Gulmini, CHEP2003, San Diego USA, March Summary and Plans RCMS architecture defined Prototypes developed aiming: –Control of small DAQs to be used in Testbeam applications: Next May Testbeams (CMS Tracker and Muon DT) will provide important feedbacks on its functionality –Demonstrators aiming the validation of the architecture in terms of performance and scalability Further investigation needed mainly on the IMS Open Grid Software Architecture (OGSA) under evaluation Problem Solver development in progress: –Error detection and recovery Databases studies and evaluation foreseen