Download presentation
Presentation is loading. Please wait.
Published byRobert Townsend Modified over 9 years ago
1
Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t DBCF GT IT Monitoring WG Update on the tool hunt & MonALISA monitoring 12 December 2011
2
Grid Technology Agenda MonALISA monitoring Tool hunt –Storage/Analysis –Real Time visualization –Data Mining tools Update on the tool hunt - 2
3
Grid Technology MonALISA monitoring (by Pablo) Slides: http://cern.ch/go/zM67http://cern.ch/go/zM67 Rates –4 proxies (brokers) –6K values per second, peak rate: 20K per second UDP, Lost rate < 5*10 -6 Security –Shared password, firewall-like config on services Flexible with adding new sensors Data storage: –RDBMS (MySQL, PostgreSQL, memory) –Hadoop implementation (results not so good) Update on the tool hunt - 3
4
Grid Technology Storage/Analysis RDBMS –Oracle, MySQL NoSQL –See slides from last meeting: http://cern.ch/go/7Tzghttp://cern.ch/go/7Tzg Columnar Databases –Infobright Community Edition –Vertica –Teradata Update on the tool hunt - 4
5
Grid Technology Display tools Development tools: –Python, Django –GoogleChart, GraphTool (Matplotlib) –AJAX/jQuery –Java (BE) Real-time Monitoring –OpenTSDB –Graphite –Feedzai Data Mining –Pentaho –JasperSoft Update on the tool hunt - 5
6
Grid Technology Real-Time Monitoring Update on the tool hunt - 6
7
Grid Technology OpenTSDB http://opentsdb.net/ Distributed, scalable Time Series Database Written on top of HBase Scales to: –Thousands of machines –Many billions of data points –At http://www.stumbleupon.com/: 150M Datapoints/Day (We need more) Slides and talk: –http://www.youtube.com/watch?v=WlsyqhrhRZAhttp://www.youtube.com/watch?v=WlsyqhrhRZA –http://opentsdb.net/misc/opentsdb-strata.pdfhttp://opentsdb.net/misc/opentsdb-strata.pdf Update on the tool hunt - 7
8
Grid Technology Graphite http://graphite.wikidot.com/ Designed to handled numeric time-series data At Orbitz.com: 230M distinct metrics per day –Scales horizontally on both the frontend and the backend Written in Python, Webapp in Django Update on the tool hunt - 8
9
Grid Technology Feedzai Pulse http://www.feedzai.com/ Computing Seminar@CERN (Feb 2011): –http://indico.cern.ch/conferenceDisplay.py?confId=12380 2http://indico.cern.ch/conferenceDisplay.py?confId=12380 2 –Castor Monitoring Demo Millions of events in real-time Real-time data processing and analytics Predictive Analytics Update on the tool hunt - 9
10
Grid Technology Business Intelligence Tools Update on the tool hunt - 10
11
Grid Technology Pentaho http://www.pentaho.com/ Supports Big Data Platforms (Hadoop, NoSQL): –Webcast: http://cern.ch/go/6kq6http://cern.ch/go/6kq6 Open Source Community Edition: –http://community.pentaho.com/http://community.pentaho.com/ Enterprise Edition offers more functionality but is not for free… Update on the tool hunt - 11
12
Grid Technology Jaspersoft http://www.jaspersoft.com/ Supports Big Data Platforms (Hadoop, NoSQL): –http://www.jaspersoft.com/bigdatahttp://www.jaspersoft.com/bigdata Open Source Community Edition: –http://jasperforge.org/http://jasperforge.org/ Enterprise Edition offers more functionality but is not for free… Update on the tool hunt - 12
13
Grid Technology Summary Your input is welcome Next step: –Tool matching based on the use cases https://tomtools.cern.ch/confluence/display/MWG/U se+Cases Update on the tool hunt - 13
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.