1. Kibana, Grafana and Zeppelin on Monitoring data
Internal group presentaion
Ildar Nurgaliev
OpenLab Summer student

2. Presentation structure
About IT-CM-MM Section and myself
Visualisation with Kibana 4 and Grafana
Motivation and comparison
Reporting and Plotting with Zeppelin
Improvement by using Spark
11/08/2016
Ildar Nurgaliev – CERN openlab

3. Ildar Nurgaliev – CERN openlab
Start
End
University name
Degree
Faculty
 2011
2015
KFU - High School ISIT
Bachelor
Applied Informatics
2014
Innopolis university
Artificial Intelligence
now
Master
Data Science
Education
2013
2014
Solution Developer at Fujitsu GDC (Global Delivery Center)
- enterprise application | desktop application | thin web client
(Ext-JS, Java, Spring-framework, Hibernate, Swing)
2016
Researcher
- Cognitive Architecture “NEUCOGAR”.
- Graph matching and percolation theory for huge graphs. (C++, math algorithms, Random graph models)
Experience
11/08/2016
Ildar Nurgaliev – CERN openlab

4. Ildar Nurgaliev – CERN openlab
Since 2014
11/08/2016
Ildar Nurgaliev – CERN openlab

5. IT-CM-MM Section (Monitoring)
Evolution of Monitoring tools for the CERN Tier-0 & WLCG Monitoring of DC at CERN and Wigner meter, timber Experiment Dashboards data transfers, job information, WLCG reports
11/08/2016
Ildar Nurgaliev – CERN openlab

6. Visualisation Comparison
Upcoming upgrade of DC Monitoring (meter, timber) from Kibana 3
Data Centre Overview dashboard
Kibana4
Grafana
Investigation on Grafana
Porting Host Metrics
Porting FTS dashboard
ACL and SSO
Comparison of Kibana vs. Grafana (Table)
11/08/2016
Ildar Nurgaliev – CERN openlab

7. Ildar Nurgaliev – CERN openlab
Data Centre Overview
Kibana 3
11/08/2016
Ildar Nurgaliev – CERN openlab

8. Ildar Nurgaliev – CERN openlab
Data Centre Overview
Kibana 4
11/08/2016
Ildar Nurgaliev – CERN openlab

9. Ildar Nurgaliev – CERN openlab
Data Centre Overview
Grafana
11/08/2016
Ildar Nurgaliev – CERN openlab

10. Ildar Nurgaliev – CERN openlab
HOST Metrics
Kibana 3
11/08/2016
Ildar Nurgaliev – CERN openlab

11. Ildar Nurgaliev – CERN openlab
HOST Metrics
Kibana 3
Mem Util. Very similar plots
11/08/2016
Ildar Nurgaliev – CERN openlab

12. Ildar Nurgaliev – CERN openlab
HOST Metrics
Grafana
Templated repeat visualisation
11/08/2016
Ildar Nurgaliev – CERN openlab

13. Ildar Nurgaliev – CERN openlab
HOST Metrics
Grafana
Auto visualisation generate: Mem Util
11/08/2016
Ildar Nurgaliev – CERN openlab

14. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Kibana 4
11/08/2016
Ildar Nurgaliev – CERN openlab

15. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Grafana
Select VO = ATLAS;CMS
11/08/2016
Ildar Nurgaliev – CERN openlab

16. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Grafana
Some endpoints selected
11/08/2016
Ildar Nurgaliev – CERN openlab

17. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Grafana
Automated Menu (Grafana feature)
11/08/2016
Ildar Nurgaliev – CERN openlab

18. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Grafana
Transfer Sites Dashboard
11/08/2016
Ildar Nurgaliev – CERN openlab

19. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Kibana 4
Ranking Country Dashboard (nominal axis)
11/08/2016
Ildar Nurgaliev – CERN openlab

20. Ildar Nurgaliev – CERN openlab
FTS Monitoring
Grafana
Ranking Country Dashboard (tables, pie, NO bar charts!)
11/08/2016
Ildar Nurgaliev – CERN openlab

21. Ildar Nurgaliev – CERN openlab
ACL in Grafana
For testing purposes 3 organisations (groups) created:
MONIT - default for newcomers (Viewer permission)
[contains all the dashboards now]
ATLAS
CMS
Tried to see whether users can access, read or write across those organisations
Summary:
Users could be assigned to different organizations
Using this multiple subscription we could attach an LDAP as centralised ACL
Grafana has good support for SSO
SSO accounts automatically mapped
Nested groups are not supported in Grafana
11/08/2016
Ildar Nurgaliev – CERN openlab

22. Ildar Nurgaliev – CERN openlab
Comparison
Feature
Aspect
- Plots look refined without banners or edit button
- Expandable visualizations
- Not very refined
Search and exploring
- Lucene query for visualisation
- By dashboard developer
- basically it has 'data exploration'
- show document structure
- inbuild search highlights
- Save search as object
- By dashboard user
Reusage objects
- No visualisation reuse
- Manual repeat search query
Visualisation, Dashboards, Search are saved as objects (plugable into many dashboards)
General plots
All plots are Time-series based
- Time series
- Nominal axis
- Basic Heat Map
Export & Share
Visualisation -> CSV, PNG,
Render image url
Visualisation -> CSV, JSON
11/08/2016
Ildar Nurgaliev – CERN openlab

23. Ildar Nurgaliev – CERN openlab
Comparison
Feature
Role-based access
(RBA)
- RBA/ACL supported by default as “Organisations”
- no built-in RBA/ACL
- Commercial plug-in
Plotting derived fields
Yes
Scripted fields
Scripted fields,
TimeLion purposeful tool
Combined plots
very flexible visualisation
Overlapping plots No
Plots from  different sources
As many as you wish
(ES, Graphite, Influx Db)
One ES source only
Support for Templates
Automatic visualisations from limited set of values for a var, change datasource on the fly
11/08/2016
Ildar Nurgaliev – CERN openlab

24. Ildar Nurgaliev – CERN openlab
Scrutiny Report Plots
Scrutiny Report: Overview of usage of all WLCG resources (more than 50 pages)
Motivation: Automate generation of plots for the ATLAS report with Zeppelin
Datasets popularity plots
Discovery of unused data
Used to optimise improve data management policies (e.g. replicas and lifetime of data)
11/08/2016
Ildar Nurgaliev – CERN openlab

25. Current Workflow - Input Data
Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop
Summaries generated on a web server as CSV files
CSV files manually downloaded and imported in Excel
11/08/2016
Ildar Nurgaliev – CERN openlab

26. Current Workflow - Plots
Cronjobs run PIG (hadoop) scripts that aggregate ATLAS datasets access events from Hadoop
Summaries generated on a web server as CSV files
CSV files manually downloaded and imported in Excel
11/08/2016
Ildar Nurgaliev – CERN openlab

27. Ildar Nurgaliev – CERN openlab
Why Zeppelin
Web-based notebook that
Enables interactive data analytics with powerful dynamic visualisations
Supports several technologies out of the box (Python, SQL, Spark, Hadoop, etc)
Provides interactive forms:  (Text Input Forms, Select Forms, Checkbox forms, …)
11/08/2016
Ildar Nurgaliev – CERN openlab

28. Datasets Popularity Plots
Starting from the same CSV files
Show the number of times ATLAS data were accessed to find datasets usage
Volumes of data (V) vs. number of accesses (X) for last N months
Number of times accessed in periods of 3, 6, 9, 12 months and infinity
11/08/2016
Ildar Nurgaliev – CERN openlab

29. Ildar Nurgaliev – CERN openlab
Sample Plots - View
PyPlot for report-
quality plots
0-access bins
Old and new datasets
Most accessed datasets
Interactive built-in plots for discovery
11/08/2016
Ildar Nurgaliev – CERN openlab

30. Ildar Nurgaliev – CERN openlab
Zeppelin Notebooks
Developer’s view
User’s view
11/08/2016
Ildar Nurgaliev – CERN openlab

31. Discovery of Unused Datasets
Starting from the same CSV files
Show Top-N unused data
by project/datatype for last X months
for every month it is clear which datasets/datatypes are unused (e.g. mc16_7TeV)
11/08/2016
Ildar Nurgaliev – CERN openlab

32. Unused Data by Type and Creation Time
11/08/2016
Ildar Nurgaliev – CERN openlab

33. Aggregation with Spark - Ongoing
Replace PIG scripts with Spark
Improve execution speed
Use Mesos/Chronos monitoring infrastructure for scheduling jobs
11/08/2016
Ildar Nurgaliev – CERN openlab

34. Ildar Nurgaliev – CERN openlab
Thank you CERN!
List of studied technologies
Elasticsearch
Kibana dashboard
Grafana dashboard
Hadoop
Apache PIG
Apache Spark
Zeppelin notebook
11/08/2016
Ildar Nurgaliev – CERN openlab