Monitoring Evolution and IPv6 Alberto AIMAR, IT-CM-MM

Outline Context Data Centres Monitoring Experiments Dashboards Architecture Plans Status Demo

Monitoring Data Centre Monitoring Experiment Dashboards Monitoring of DC at CERN and Wigner Hardware, operating system, and services Data Centres equipment (PDUs, temperature sensors, etc.) Used by service providers in IT, experiments Experiment Dashboards Sites availability, data transfers, job information, reports Used by WLCG, experiments, sites and users Both hosted by CERN IT, in different teams

Context Focus for 2016 Regroup monitoring activities hosted by CERN/IT (Data Centres, Experiment Dashboards, ETF, HammerCloud, etc) Continue existing services Uniform with CERN IT practices Management of services, communication, tools (e.g. GGUS and SNOW tickets) Starting with Merge Data Centres and Experiment Dashboards monitoring technologies Review existing monitoring usage and needs (IT, WLCG, etc) Investigate new technologies Unchanged support while collecting user feedback

Unified Monitoring Architecture Data Sources Storage/Search Transport Views Data Centers Processing WLCG Data kafka

Experiment Dashboards Operation Teams Sites Analysis + Production Real time and Accounting views Data transfer Data access Site Status Board SAM3 Google Earth Dashboard Sites Data Management Monitoring General Public Users Outreach Job Monitoring Infrastructure Monitoring Operation Teams Operation Teams Experiment Dashboard covers the full-range of experiments’ computing activities Provides information to different categories of users Sites Sites 300-500 users per day

Experiment Dashboards Job monitoring, sites availability, data management and transfers Used by experiments operation teams, sites, users, WLCG

Processing & Aggregation Current Monitoring Data Sources z Transport Storage &Search Display & Reports Data Centres Monitoring Metrics Manager Flume HDFS Kibana Lemon Agent AMQ ElasticSearch Jupyter XSLS Kafka Oracle Zeppelin ATLAS Rucio ElasticSearch Data mgmt and transfers Flume FTS Servers HDFS AMQ DPM Servers Oracle GLED Dashboards (ED) XROOTD Servers ElasticSearch Kibana CRAB2 Oracle Zeppelin Monitoring Job CRAB3 ElasticSearch HTTP Collector WM Agent Processing & Aggregation SQL Collector Farmout Grid Control MonaLISA Collector Spark Real Time (ED) CMS Connect Hadoop Jobs Accounting (ED) PANDA WMS GNI API (ED) ProdSys Oracle PL/SQL Nagios ESPER Infrastructure Monitoring AMQ VOFeed Spark SSB (ED) HTTP GET OIM Oracle PL/SQL SAM3 (ED) HTTP PUT ES Queries GOCDB API (ED) ESPER REBUS

Processing & Aggregation Unified Monitoring Data Sources Transport z Storage &Search Display & Reports Metrics Manager Lemon Agent XSLS ATLAS Rucio FTS Servers Hadoop HDFS DPM Servers ElasticSearch XROOTD Servers Other CRAB2 Flume Kibana CRAB3 AMQ Jupyter WM Agent Kafka Processing & Aggregation Zeppelin Farmout Other Grid Control CMS Connect PANDA WMS Spark ProdSys Hadoop Jobs Nagios GNI VOFeed Other OIM GOCDB REBUS

Status Producers and Transport Storage and Search Processing Moving all data via new transport (Flume, AMQ, Kafka) Storage and Search Data in ES and Hadoop Processing Doing aggregation and processing via Spark Display and reports Experimenting using only the standard features of ES, Kibana, Spark, Hadoop Introduce notebooks and data discovery General Selecting technologies, learning on the job, looking for expertise Evolve interfaces (e.g. dashbords for users, shifters, sites, managers)

IPv6 and Monitoring for WLCG Data Sources We are confident that there are no major issues : No major changes vs the check in 2013 Evolution to the new architecture will take IPv6 into account Using the main stream technologies, very little code of our own Data sources Relies on external systems providing monitoring data Depends in data provided by FTS, Rucio, Panda, CRAB3, Xrootd, etc. MonALISA is external, used by ALICE and other projects (tested by ML devs) Transport Receives data via AMQ/Stomp, Flume, UDP, databases and HTTP sources. It is matter of staying up to date with ipv6-ready versions Metrics Manager Lemon Agent XSLS ATLAS Rucio FTS Servers DPM Servers XROOTD Servers CRAB2 CRAB3 WM Agent Farmout Grid Control CMS Connect PANDA WMS ProdSys Nagios VOFeed OIM GOCDB REBUS

IPv6 and Monitoring for WLCG z Storage &Search Display & Reports Transport Hadoop HDFS ElasticSearch Flume Other Kibana AMQ Jupyter Kafka Processing & Aggregation Zeppelin Spark Other Hadoop Jobs GNI Other Storage Storing mostly the host names only, as strings In a few cases the current Experiment Dashboards may store WN IP and will be fixed in the migration ElasticSearch has an IPv4 data type, but not IPv6 at the moment. Will come. Display and reports Only showing IPv4 and IPv6 hosts, names as strings Web applications can easily be made reachable by IPv6 nodes, actions will be needed (just like any other web server)

Plans Unified architecture and technologies Focus on migrating to common architecture Review the existing architecture, areas and data Update to new technologies in several areas Better perfomance and new versions with new features and major improvements Look into technologies as needed (collectd, Kafka, Grafana, etc.) Benefit from experience and feedback received from Experiments , WLCG and IT groups Move to central services Central service for ES is being created, InfluxDB for time series DBoD Continue to use central Hadoop services Continue with standard operations and upgrades At least for all 2016 Make available the new monitoring platform, in parallel with the existing ones

Conclusions No major changes vs. What reported in 2013 Mainstream technologies benefit from community effort and/or official support for IPv6 readiness. Evolution to the new architecture is used to review the whole monitoring data IPv6 is one of the reviews we will do No specific issues for monitoring detected

Demo Data in ElasticSearch FTS and Xrootd transfers data Examples of Dashboards Data discovery and error investigation Specific views for specific tasks VO overview Site manager

Data Centres Monitoring

Monitoring Technologies Area Services and Components Technologies Functions Data Collectors Metric Manager CERN Metrics registration Lemon Agent CERN+Flume Metrics producers (about 15000) Transport Gateway Flume Transport host metrics XSLS Service metrics Messaging Active MQ Messaging of metrics River Kafka Streaming of metrics Aggregation Foz Spark Processing streamed metrics Archive HDFS Hadoop Long term storage Displays Meter ElasticSearch + Kibana Dashboard for metrics Timber ElasticSearch+ Kibana Dashboard for logs Meter Proxy CLI and HTTP Interface to ES Alerts GNI Alarms handling