Download presentation
Presentation is loading. Please wait.
Published byClaire Hart Modified over 6 years ago
1
Update on CERN IT Unified Monitoring Architecture (UMA)
IT-CM-MM
2
News New dedicated ES cluster for Monitoring
Lemon data getting integrated Working on Grafana frontend too On ATLAS matters: New Data sources (DDM Files Queued) New/Improved Processing Jobs (DDM, JM) New/Improved Dashboards (FTS, DDM, JM) Notebooks ideas (Scrutiny) 06/10/2016 ATLAS ADC meeting
3
FTS Workflow Kibana Dashboard ~ production ready:
FTS transfers logs as data source Streaming Job to enrich data (from VOfeeds) Enriched data stored in ES and HDFS Kibana Dashboard ~ production ready: Same look and feel as old one, when possible New views (e.g. error exploration, FTS log links) Banner now on the old dashboard Demo: New FTS Dashboard 06/10/2016 ATLAS ADC meeting
4
ATLAS DDM Workflow Validation ongoing, all good so far
Rucio Events and Traces as data sources Streaming job to transform, enrich (AGIS) and compute 1 min aggregated statistics Enriched data on HDFS, Stats on ES and HDFS Validation ongoing, all good so far Good progress on job resilience (thx Sergey!) Kibana DDM Dashboard prototype ready: Demo: New DDM Dashboard 06/10/2016 ATLAS ADC meeting
5
ATLAS DDM (II) DDM Queued Files (RQF0615374) Scrutiny Notebooks
New data source integrated JSON Data retrieved ~ 5 minutes from a Web server Demo: Kibana dashboard available Scrutiny Notebooks Summer student worked on porting to Zeppelin some of the scrutiny analysis from HDFS data Automatize plot creation in ~ one click Demo: Monitoring Zeppelin 06/10/2016 ATLAS ADC meeting
6
Job Monitoring Workflow Real-time data: all validated
Panda, Tier0, Prodsys as data sources Several processing jobs to enrich (AGIS) and aggregate into accounting hourly stats Both enriched and stats data on HDFS and ES Real-time data: all validated Accounting data: working on pending/running and error stats, everything else validated Jobs by Event-service already available Kibana Dashboards prototype ready Demo: Real-time, Accounting 06/10/2016 ATLAS ADC meeting
7
Kibana upstream improvements
Bucket aggregations (#4584) e.g. weighted average. Will improve efficiency visualization and throughput computation Missing “others” (#1961) e.g. show top 10 sites + an entry with the contribution of all the others Better matrix/heatmap Clickable cells Visualize value on cells 06/10/2016 ATLAS ADC meeting
8
Summary Moving from early prototypes to production ready dashboards
Consolidating workflow Build your own dashboard! Setting up a “Monitoring for WLCG” group to exchange information and feedback 06/10/2016 ATLAS ADC meeting
9
Reference & Contacts Dashboards: Feedback/Requests: Documentation:
monit.cern.ch Feedback/Requests: cern.ch/monit-support (SNOW) Documentation: cern.ch/monitdocs 06/10/2016 ATLAS ADC meeting
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.