DIAMON

What is DIAMON ? Technology stack Current Situation & Plans

What is DIAMON? A service which allows to check what IS and what WAS going on.

The User Interface Informs on Health of computers / applications / equipment Computer/ application problem logs Host events: installations, restarts, reboots Responsible of computers Allows to Restart software or computer Subscribe to mail/sms notifications Introspect remotely processes JMX/CMX History Created in 2006 based on LASER-1 technology DIAMON V.2 based on C2Mon since 2012 Targeted at Developers + Operators (+ Service Managers)

The User Interface Shows general information on entity: responsible, operational, location Shows metrics and their health: computer, FECs, workstations, PLCs, module driver status, timing board status, process up/down Allows access to do diagnostic Introspect remotely processes JMX/CMX Check FIP Board status Check loaded drivers for modules Check Timing board state Check Host events : installations, restarts, reboots Check Problem logs Check running processes Actions: Restart software or computer Ping computer, SSH to machine Subscribe to mail/sms notifications Targeted at Developers + Operators (+ Service Managers)

What do we monitor ? Applications: metrics via JMX/RDA/CMX + up/down state Computers: FECs, Workstations, servers, VMs CPU/disk/uptime/network/ SNMP devices, e.g. WhiteRabbit Dedicated checks, e.g. JMS service performance Timing Board status Module Driver Status (correct/incorrect) FIP Module status 3400 computers 6880 processes metrics 3400 computers 6880 processes metrics

Entities are organized in tree structure Details Tools to filter Groups / Hosts

Find computer/process/metrics quickly HOSTSProcesesMetrics

Problem Overview for Host

Metric Details for Host

Host Events : restarts, reboots, installations..

Shows logs from Tracing

Custom Extension s Module Driver Status FIP board state

Remote “live” introspection of Java/C/C++ Processes C/C++ Services Java Services

Get Responsible quickly

Enable your notifications

Notification Example Alert: IO Wait on computer too high IO Wait back to OK

Manage your notifications If in WARN/ERROR this sends value change notification Filter items EGroup Edit Mode Notification Level Enable regular problem report

DIAMON GUI as portal to more information JAPC ToolboxCCDB Tracing CMWAdmin CCDB DIAMON Configuration Only for JMX & CLIC …

How is DIAMON used GUI Operators, Equipment specialists, Service Managers User specific configurations = “views” (CCDB) Backend Notify on problems: > 90 Messages/day, 29 users Applications read data from DIAMON via RDA (i.e. MOON) Host alarms are expected by OP FECs host metrics in Lemon Driver Feedback FIP problems

Technology Stack

CERN Control and Monitoring Platform Acquisition and filtering of metrics Evaluation of rules and alarms Provides historical data and replay Extensible and modular The C2MON Project C2MON Server DAQ API Client API myApp myDAQ Acquisition Filtering Business Client Apps 22 myMod

CLICRDA LASERJMX DIAMON (C2Mon) CCDB RDALASER History ~ datapoints ~8271 equipments 24 DAQs for acquisition & filtering PING 1 Server Lemon MOONLemon Acquisition & Filtering > 15 Equipment types > 26 Million updates/day > 26 Million updates/day Business Logic Layer > 150k data points > 300k alarms > 20K commands > 50K business rules > 28 Million updates/day > 70 configurations/day DIAMON GUI TRACING LASER configures C2MON 4 Outputs SNMP

The CLIC Agent C++ Process on every machine Compatible with Windows/Linux down to PPC4 Accessible via JMS Java API or /mcr/bin/dmnsh command Sends metrics every x seconds Extensions: Allows access to Timing board state Allows access to CMX enabled C/C++ processes Allows access to FIP Bus Only client-side acquisition of metrics CLIC communication via STOMP

What is good.. System is used by large user group on daily basis Acquisition and standardizing information from various sources (RDA, JMX, SNMP, HOST metrics…) C2Mon platform is quite reliable & evolving in right direction Online reconfiguration is VERY useful Mail/sms Notification very useful for pre-failure detection What is not so good: CCDB driven (Re)configuration SQL based history limited to 3 months & slow (C2Mon) Limited rule evaluation (C2Mon) Notification messages sometimes not really clear Some information is still missing compared to e.g. lemon Notifications or triggering rules cannot be easily trained

General Experience Sometimes not clear who is responsible for certain problem People have most experience, not the system We can show the responsible for host, but what about a module or timing card problem ? No information on responsible for deployed software process No concept of acknowledging a problem Currently rather host, less “service” oriented Notifications cannot be trained

Plans & Dreams Move configuration part away from CCDB Make information easy accessible: why only TN? Replace SQL history storage Enable sending/receiving data via REST C2MON Replace JCache implementation (Terracotta -> Ignite?) Find better solution for rule engine

Appendix

CERN Control and Monitoring Platform Acquisition and filtering of metrics Evaluation of rules and alarms Provides historical data and replay Extensible and modular The C2MON Project C2MON Server DAQ API Client API myApp myDAQ Acquisition Filtering Business Client Apps 29 myMod

C2MON in Action DAQ C2MON Server Terracotta … C2MON Server DAQ … Access Dashboard Data Analysis Terracotta Standby 30 Removes database dependency Allows scaling horizontally Client JMX SNMP Alarm Web Interface

Technologies used for C2MON C2MON server: No J2EE server and only open source!  Java 7, Spring 4.2  persistence framework: MyBATIS (server), Hibernate (client) Dependency management through Maven Database: Oracle, but no major dependencies. Works also with HSQL, MySQL Middleware: JMS ActiveMQ Message transport format: XML and JSON Remote caching solution for C2MON server cluster: Terracotta/Ehcache, (Apache Ignite foreseen)  horizontally scalable  proven technology  open source  support contract possible