Kibana, Grafana and Zeppelin on Monitoring data Internal group presentaion Ildar Nurgaliev OpenLab Summer student

Presentation structure About IT-CM-MM Section and myself Visualisation with Kibana 4 and Grafana Motivation and comparison Reporting and Plotting with Zeppelin Improvement by using Spark 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Start End University name Degree Faculty  2011 2015 KFU - High School ISIT Bachelor Applied Informatics 2014 Innopolis university Artificial Intelligence now Master Data Science Education 2013 2014 Solution Developer at Fujitsu GDC (Global Delivery Center) - enterprise application | desktop application | thin web client (Ext-JS, Java, Spring-framework, Hibernate, Swing) 2016 Researcher - Cognitive Architecture “NEUCOGAR”. - Graph matching and percolation theory for huge graphs. (C++, math algorithms, Random graph models) Experience 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Since 2014 11/08/2016 Ildar Nurgaliev – CERN openlab

IT-CM-MM Section (Monitoring) Evolution of Monitoring tools for the CERN Tier-0 & WLCG Monitoring of DC at CERN and Wigner meter, timber Experiment Dashboards data transfers, job information, WLCG reports 11/08/2016 Ildar Nurgaliev – CERN openlab

Visualisation Comparison Upcoming upgrade of DC Monitoring (meter, timber) from Kibana 3 Data Centre Overview dashboard Kibana4 Grafana Investigation on Grafana Porting Host Metrics Porting FTS dashboard ACL and SSO Comparison of Kibana vs. Grafana (Table) 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Data Centre Overview Kibana 3 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Data Centre Overview Kibana 4 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Data Centre Overview Grafana 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab HOST Metrics Kibana 3 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab HOST Metrics Kibana 3 Mem Util. Very similar plots 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab HOST Metrics Grafana Templated repeat visualisation 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab HOST Metrics Grafana Auto visualisation generate: Mem Util 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Kibana 4 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Select VO = ATLAS;CMS 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Some endpoints selected 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Automated Menu (Grafana feature) 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Transfer Sites Dashboard 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Kibana 4 Ranking Country Dashboard (nominal axis) 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab FTS Monitoring Grafana Ranking Country Dashboard (tables, pie, NO bar charts!) 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab ACL in Grafana For testing purposes 3 organisations (groups) created: MONIT - default for newcomers (Viewer permission) [contains all the dashboards now] ATLAS CMS Tried to see whether users can access, read or write across those organisations Summary: Users could be assigned to different organizations Using this multiple subscription we could attach an LDAP as centralised ACL Grafana has good support for SSO SSO accounts automatically mapped Nested groups are not supported in Grafana 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Comparison Feature   Aspect - Plots look refined without banners or edit button - Expandable visualizations - Not very refined Search and exploring - Lucene query for visualisation - By dashboard developer - basically it has 'data exploration' - show document structure - inbuild search highlights - Save search as object - By dashboard user Reusage objects - No visualisation reuse - Manual repeat search query Visualisation, Dashboards, Search are saved as objects (plugable into many dashboards) General plots All plots are Time-series based - Time series - Nominal axis - Basic Heat Map Export & Share Visualisation -> CSV, PNG, Render image url Visualisation -> CSV, JSON 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Comparison Feature   Role-based access (RBA) - RBA/ACL supported by default as “Organisations” - no built-in RBA/ACL - Commercial plug-in Plotting derived fields Yes Scripted fields Scripted fields, TimeLion purposeful tool Combined plots very flexible visualisation Overlapping plots No Plots from  different sources As many as you wish (ES, Graphite, Influx Db) One ES source only Support for Templates Automatic visualisations from limited set of values for a var, change datasource on the fly 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Scrutiny Report Plots Scrutiny Report: Overview of usage of all WLCG resources (more than 50 pages) Motivation: Automate generation of plots for the ATLAS report with Zeppelin Datasets popularity plots Discovery of unused data Used to optimise improve data management policies (e.g. replicas and lifetime of data) 11/08/2016 Ildar Nurgaliev – CERN openlab

Current Workflow - Input Data Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop Summaries generated on a web server as CSV files CSV files manually downloaded and imported in Excel 11/08/2016 Ildar Nurgaliev – CERN openlab

Current Workflow - Plots Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop Summaries generated on a web server as CSV files CSV files manually downloaded and imported in Excel 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Why Zeppelin Web-based notebook that Enables interactive data analytics with powerful dynamic visualisations Supports several technologies out of the box (Python, SQL, Spark, Hadoop, etc) Provides interactive forms:  (Text Input Forms, Select Forms, Checkbox forms, …) 11/08/2016 Ildar Nurgaliev – CERN openlab

Datasets Popularity Plots Starting from the same CSV files Show the number of times ATLAS data were accessed to find datasets usage Volumes of data (V) vs. number of accesses (X) for last N months Number of times accessed in periods of 3, 6, 9, 12 months and infinity 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Sample Plots - View PyPlot for report- quality plots 0-access bins Old and new datasets Most accessed datasets Interactive built-in plots for discovery 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Zeppelin Notebooks Developer’s view User’s view 11/08/2016 Ildar Nurgaliev – CERN openlab

Discovery of Unused Datasets Starting from the same CSV files Show Top-N unused data by project/datatype for last X months for every month it is clear which datasets/datatypes are unused (e.g. mc16_7TeV) 11/08/2016 Ildar Nurgaliev – CERN openlab

Unused Data by Type and Creation Time 11/08/2016 Ildar Nurgaliev – CERN openlab

Aggregation with Spark - Ongoing Replace PIG scripts with Spark Improve execution speed Use Mesos/Chronos monitoring infrastructure for scheduling jobs 11/08/2016 Ildar Nurgaliev – CERN openlab

Ildar Nurgaliev – CERN openlab Thank you CERN! List of studied technologies Elasticsearch Kibana dashboard Grafana dashboard Hadoop Apache PIG Apache Spark Zeppelin notebook 11/08/2016 Ildar Nurgaliev – CERN openlab