TANGO MONITORING SYSTEM ELI BEAMLINES, Dolní Břežany, Česká republika, 07.06 Michał Gajda National Synchrotron Radiation Center SOLARIS

M/Monit – monitoring tool watching over your hosts Table of Contents M/Monit – monitoring tool watching over your hosts ABRT – monitoring tool reporting crashes of applications Splunk-light – monitoring and collecting tool for device logs

Difference between Monit and M/Monit Easy to install Easy to configure Easy to use Permanent license

What can it monitor? System Services Clustering Services RAID Monit - possibilities What can it monitor? System Services Clustering Services RAID Name Services FTP Services WWW Services Mail Services Files and directories and more…

Monit instance on one of our host Monit – use case Monit instance on one of our host

Monit - configuration Examplary Monit configuration file: Setting polling Logging system Setting port, address of http interface Login and password to access http interface Eventqueue directory Setting M/Monit instance Setting system monitoring Setting network interface to monitor

Extends Monit capabilities Manages Monit instances M/Monit M/Monit: Extends Monit capabilities Manages Monit instances Collects data from Monit instances Storage for Monit instances

Quick status overview: Hosts status Events from 24-hours period M/Monit – dashboard Quick status overview: Hosts status Events from 24-hours period Active errors and warnings

M/Monit - use case

Events history for host/host groups sorted by: Service group M/Monit - use case Events history for host/host groups sorted by: Service group Service type Event type Period of time

Network incoming/outgoing M/Monit - use case Statistics: CPU % usage Memory % usage Disk read [bytes/s] Process threads Process children Network incoming/outgoing

M/Monit - use case

ABRT – automatic bug reporting tool Free Linux tool for crash reporting Easy to install and configure Can catch and send crash reports from various applications and send them via e-mail

What can it report: Java crashes Python crashes C/C++ crashes ABRT – possibilities What can it report: Java crashes Python crashes C/C++ crashes Ruby crashes Kernel crashes Xorg crashes

After ABRT installation: ABRT – configuration After ABRT installation: Install addons(python, java, etc.) from Linux repository Allow core dumps, from cli: ulimit –c unlimited Configure libreport mailx plugin to send an e-mail /etc/libreport/plugin/mailx.conf: Configure mail transfer agent(MTA) to send e-mail via mail server

ABRT – use case Examplary e-mail from ABRT E-mail includes: Abrt version Analyzer Architecture Core backtrace Cmdline Count Duphash Last occurance Reason Environment variables And more…

Splunk-light free – possibilities and limitations Used for: Real time log monitoring Log acquisition Log analyzation Log visualisation Log buckets rotation

Splunk-light free – installation Download and install Splunk-light i.e using .rpm file Start Splunk-light from /opt/splunk/bin:

Splunk-light free - splunkforwarder Download, install and configure splunkforwarder on each host to send data to splunk indexer(database): Install splunkforwarder from .rpm file Configure splunk forwarder, from cli /opt/splunkforwarder/bin:

Splunk-light free – Tango devices Before collecting logs from splunk: Configure logging level for devices Configure logging target for devices Configure rolling file threshold(RFT- 20MB default)

Splunk-light free - possibilities Splunk-light features: Searches Hosts Sources Sourctypes Patterns Reports Errors Dashboards Charts Tables and more…

Splunk-light free – dashboard Possibility to create custom dashboards to visualize data inputs: Charts Tables Gauges Single values Maps

Splunk-light free – use case Custom dashboard to visualize specific data

Splunk-light free – use case Ramping procedure example: Search from last hour Over 120 000 logs in 5 minutes

Splunk-light free – use case Quick log analyzation: Checking log levels

Splunk-light free – use case Results: 224 error logs

Splunk-light free – use case Checking patterns to see errors scale

Thank you for your attention Questions