Online Monitoring with MonALISA Dan Protopopescu Glasgow, UK Dan Protopopescu Glasgow, UK

MonALISA Is a distributed service able to:  collect any type of information from different systems  analyze this information in real time  take automated decisions and perform actions based on it  optimize work flows in complex environments Read more at

Uses  Monitoring distributed computing, i.e. GRIDs  Optimizing flow in complex system (VRVS, optics cable networks)  ALICE also uses ML for monitoring online reconstruction  Some benchmark figures for the service:  ~ 800k monitored parameters at 50k updates/second  > 10k running (alien) jobs monitored simultaneously  > 100 WAN links We are proposing ML as a high level monitoring and possible control system along with (or on top of) existing slow controls systems as epics, pvss etc.

Advantages  MonALISA is simple to install, configure and use  ApMon APIs are available in C, C++, Java, Python and Perl  ROOT plugin allows macros to send data directly to MonaLISA  Can easily interface with (or sit on top of) any existing or future slow controls subsystem (epics, pvss)  Data is stored in a standard PgSQL (or MySQL) database that can be accessed by other applications, independently of ML  Automatic data summarizing  Several data repositories (and hence DBs) can exist (local and remote)  Easy access via WebService (WS) from service and/or repository  Fully supported by development team; work is being done in this direction

Capabilities Based on monitored information, actions can be taken in:  ML Service  ML Repository Actions can be triggered by:  Values above/below given thresholds  Absence/presence of values  Correlations between several values Possible actions types:  External command  Plain event logging  Annotation of repository charts; RSS feeds   Instant messaging

Components Service Repository LUS/Proxies ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions GUI

Service setup Service Repository LUS ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions ML Service setup: wget tar -zxvf MonaLisa.tar.gz cd MonaLisa/./install.sh cd../MonaLisa/Service/CMD/./MLD start ML Service setup: wget tar -zxvf MonaLisa.tar.gz cd MonaLisa/./install.sh cd../MonaLisa/Service/CMD/./MLD start

Repository setup Service Repository LUS ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions ML Repository setup: wget tar -zxvf MLrepository.tgz [configure it] cd MLrepository./start.sh ML Repository setup: wget tar -zxvf MLrepository.tgz [configure it] cd MLrepository./start.sh

ApMon setup Service Repository LUS/Proxies ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions ApMon setup: wget tar -xzvf ApMon_perl.tar.gz cd ApMon_perl [create your script, say mysend.pl] perl mysend.pl ApMon setup: wget tar -xzvf ApMon_perl.tar.gz cd ApMon_perl [create your script, say mysend.pl] perl mysend.pl

Simple monitoring script Service Repository LUS ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions cat mysend.pl use ApMon; my $apm = new ApMon({"glasgow.jlab.org:8884" => {"sys_monitoring" => 0, "general_info" => 0}}); while (1) {# loop forever # get values from = getmypar(“pspec_logic_ai_0”); $apm->sendParameters(”Detector", sleep (20); } cat mysend.pl use ApMon; my $apm = new ApMon({"glasgow.jlab.org:8884" => {"sys_monitoring" => 0, "general_info" => 0}}); while (1) {# loop forever # get values from = getmypar(“pspec_logic_ai_0”); $apm->sendParameters(”Detector", sleep (20); }

Time history Service Repository LUS ApMon Web Server ApMon Actions based on aggregated information Actions based on aggregated information Actions based on local information Actions based on local information Quick actions Time history example: cat mor.properties page=hist Farms=JlabML Clusters=Detector Nodes=MOR Functions=pspec_logic_ai_0 ylabel=Tagger rate title=MOR annotation.groups=2 Time history example: cat mor.properties page=hist Farms=JlabML Clusters=Detector Nodes=MOR Functions=pspec_logic_ai_0 ylabel=Tagger rate title=MOR annotation.groups=2

Web interface

Java GUI

Application control Key Keystore  ML Clients  TCP based subscribe mechanism serialized, compressed objects with optional encryption  ML Proxies  Application commands are encrypted  ML Services  Standard and/or user’s sensors and/or application modules ML Service ApMon Your Application Your custom Java client GUI client ML Repository Your mon module Your custom view App MonC bash Your application Your app module LUS

Alert-based Actions MySQL daemon is automatically restarted when it runs out of memory Trigger: threshold on VSZ memory usage ALICE Production jobs queue is automatically kept full by the automatic resubmission Trigger: threshold on the number of aliprod waiting jobs Administrators are kept up-to-date on the services’ status Trigger: presence/absence of monitored information via instant messaging, RSS feeds, toolbar alerts etc.

Summary  MonALISA is a very promising tool for online experiment monitoring and interfacing with a variety of slow control subsystems; GlueX are seriously considering ML for this task  Easy to configure, understand and use  Experience from Grid monitoring and more  Support from the developers group for implementation of new modules/features  Online experiment monitoring tests of were recently carried on; demo repository is at

More examples / Extras

Integrated Pie Charts

History Plots, Annotations

AliEn Services Monitoring  AliEn services  Periodically checked  PID check + SOAP call  Simple functional tests  SE space usage  Efficiency

Job Network Traffic Monitoring  Based on the xrootd transfer from every job  Aggregated statistics for  Sites (incoming, outgoing, site to site, internal)  Storage Elements (incoming, outgoing)  Of  Read and written files  Transferred MB/s

Individual Job Tracking  Based on AliEn shell cmds.  top, ps, spy, jobinfo, masterjob  Using the GUI ML Client  Status, resource usage, per job

Head Node Monitoring  Machine parameters, real-time & history, load, memory & swap usage, processes, sockets

MonALISA in AliEn  The MonALISA framework is used as a primary monitoring tool for the ALICE Grid since 2004  Presently the system is used for monitoring of all (identified) services, jobs and network parameters necessary for the Grid operation and debugging  The number of concurrently monitored and stored parameters today is ~ in 75 ML Services  The add-on tools for automatic events notification allow for more efficient reaction to problems  The framework design and flexibility answers all requirements for a monitoring system  The accumulated information allows to construct and implement automated decision making algorithms, thus increasing further the efficiency of the Grid operations