IT Monitoring Service Status and Progress Alberto AIMAR, IT-CM-MM

Outline Monitoring Data Centres Experiments Dashboards Architecture and Technologies Status and Plans

Monitoring Data Centre Monitoring Experiment Dashboards Monitoring of DC at CERN and Wigner Hardware, operating system, and services Data Centres equipment (PDUs, temperature sensors, etc.) Used by service providers in IT, experiments Experiment Dashboards Sites availability, data transfers, job information, reports Used by WLCG, experiments, sites and users Both hosted by CERN IT, in different teams

Mandate Focus for 2016 Regroup monitoring activities hosted by CERN/IT (Data Centres, Experiment Dashboards, ETF, HammerCloud, etc) Continue existing services Uniform with CERN IT practices Management of services, communication, tools (e.g. GGUS and SNOW tickets) Starting with Merge Data Centres and Experiment Dashboards monitoring technologies Review existing monitoring usage and needs (DC, WLCG, etc) Investigate new technologies Unchanged support while collecting feedback and working

Data Centres Monitoring

Data Centres Monitoring

Experiment Dashboards Operation Teams Sites Analysis + Production Real time and Accounting views Data transfer Data access Site Status Board SAM3 Google Earth Dashboard Sites Data Management Monitoring General Public Users Outreach Job Monitoring Infrastructure Monitoring Operation Teams Operation Teams Experiment Dashboard covers the full-range of experiments’ computing activities Provides information to different categories of users Sites Sites 300-500 users per day

Experiment Dashboards Job monitoring, sites availability, data management and transfers Used by experiments operation teams, sites, users, WLCG

WLCG Transfer Dashboard

ATLAS Distributed Computing

Higgs Seminar

Architecture and Technologies

Unified Monitoring Architecture Data Sources Storage/Search Transport Data Access Data Centres Processing WLCG kafka

Processing & Aggregation Current Monitoring Data Sources z Transport Storage &Search Display Access Data Centres Monitoring Metrics Manager Flume HDFS Kibana Lemon Agent AMQ ElasticSearch Jupyter XSLS Kafka Oracle Zeppelin ATLAS Rucio ElasticSearch Data mgmt and transfers Flume FTS Servers HDFS AMQ DPM Servers Oracle GLED Dashboards (ED) XROOTD Servers ElasticSearch Kibana CRAB2 Oracle Zeppelin Monitoring Job CRAB3 ElasticSearch HTTP Collector WM Agent Processing & Aggregation SQL Collector Farmout Grid Control MonaLISA Collector Spark Real Time (ED) CMS Connect Hadoop Jobs Accounting (ED) PANDA WMS GNI API (ED) ProdSys Oracle PL/SQL Nagios ESPER Infrastructure Monitoring AMQ VOFeed Spark SSB (ED) HTTP GET OIM Oracle PL/SQL SAM3 (ED) HTTP PUT ES Queries GOCDB API (ED) ESPER REBUS

Processing & Aggregation Unified Monitoring Data Sources Transport z Storage &Search Data Access Metrics Manager Lemon Agent XSLS ATLAS Rucio FTS Servers Hadoop HDFS DPM Servers ElasticSearch XROOTD Servers Other Kibana CRAB2 Flume Grafana CRAB3 AMQ Jupyter WM Agent Kafka Processing & Aggregation Zeppelin Farmout Grid Control Other CMS Connect PANDA WMS Spark ProdSys Hadoop Jobs Nagios GNI VOFeed Other OIM GOCDB REBUS

Unified Data Sources AMQ Data Sources Flume AMQ Transport DB FTS Flume DB Flume Kafka sink Rucio XRootD HTTP feed Flume HTTP Jobs … Logs Flume Log GW Lemon syslog Lemon metrics Flume Metric GW app log Decouple producer/consumer, job/stores Data is channeled via Flume, validated and modified if necessary Adding new Data Sources is documented and fairly simple 21/07/2016 ASDF meeting

(e.g. Enrich FTS transfer metrics with WLCG topology from AGIS/Gocdb) Unified Processing Transport Flume Kafka sink Flume sinks Kafka cluster (buffering) * Processing (e.g. Enrich FTS transfer metrics with WLCG topology from AGIS/Gocdb) Decouple producer/consumer, job/stores Data now 100 GB/day, at scale 500 GB/day Current retention period 12 h, at scale 24 h 21/07/2016 ASDF meeting

Data Processing Stream processing Batch processing Data enrichment Join information from several sources (e.g. WLCG topology) Data aggregation Over time (e.g. summary statistics for a time bin) Over other dimensions (e.g. compute a cumulative metric for a set of machines hosting the same service) Data correlation Advanced Alarming: detect anomalies and failures correlating data from multiple sources (e.g. data centre topology-aware alarms) Batch processing Reprocessing, data compression, reports Technologies: Reliable and scalable job execution (Spark), Job orchestration and scheduling (Marathon/Chronos), Lightweight and isolation deployment (Docker) 21/07/2016 ASDF meeting

Unified Access Storage & Search Data Access HDFS Flume sinks ElasticSearch Plots Reports … Scripts Decouple producer/consumer, job/stores Others CLI, API Multiple data access methods (dashboards, notebooks, CLI) Mainstream and evolving technology 21/07/2016 ASDF meeting

Status and Plans

WLCG Monitoring Data Sources and Transport Storage and Search Moving all data via new transport (Flume, AMQ, Kafka) Storage and Search Data in ES and Hadoop Processing Doing aggregation and processing via Spark Display and reports Using only standard features of ES, Kibana, Spark, Hadoop Introduce notebooks (e.g. Zeppelin) and data discovery General Selecting technologies, learning on the job, looking for expertise Evolve interfaces (dashboards for users, shifters, experts, managers)

WLCG Monitoring

Data Centres Monitoring (metrics) Replacement of the Lemon Agents Mainstream technologies (e.g. collectd) Support legacy sensors Starting in 2016Q4 Update meter.cern.ch Kibana and Grafana dashboards Move to new central ES service Move to the Unified Monitoring

Data Centres Monitoring (logs) Currently collecting syslog plus several application logs (e.g. EOS, Squid, HC, etc.) More requests coming for storing logs (e.g. Castor, Tapes, FTS) Update the Logs Service to the Unified Monitoring For archive in HDFS For processing in Spark For visualization in ES and Kibana

Participating to ES Benchmarking ES Service is separate from Monitoring Service Looking to use "standard“ CERN solutions VMs, remote storage, etc Data flow started few days ago, will take some weeks Identical ES settings and version 3 master nodes, m2_medium sized VMs 1 search node, m2_large size 3 data nodes, providing the same amount of data. bench01 data nodes 3 full node m2 VMs (VM takes a full hypervisor) 600GB io2 sized volumes attached as storage backend (Ceph) loop back device in /var as fast storage (SSD) bench02 data nodes: 3 full node VMs running on bcache enabled hypervisors

Conclusions / Services Proposed Monitor, collect, visualize, process, aggregate, alarm Metrics and Logs Infrastructure operations and scale Helping and supporting Interfacing new data sources Developing custom processing, aggregations, alarms Building dashboards and reports

monit.cern.ch

Reference and Contact Dashboard Prototypes monit.cern.ch Feedback/Requests (SNOW) cern.ch/monit-support Early-Stage Documentation cern.ch/monitdocs 21/07/2016 ASDF meeting

Examples

Ildar Nurgaliev – CERN openlab Data Centre Overview Grafana Ildar Nurgaliev – CERN openlab 11/08/2016

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Transfer Sites Dashboard Ildar Nurgaliev – CERN openlab 11/08/2016

Sample Plots - Zeppelin PyPlot for report- quality plots 0-access bins Old and new datasets Most accessed datasets Interactive built-in plots for discovery Ildar Nurgaliev – CERN openlab 11/08/2016

Ildar Nurgaliev – CERN openlab Zeppelin Notebooks Developer’s view User’s view Ildar Nurgaliev – CERN openlab 11/08/2016