1. Monitoring for large infrastructure
Radu-Andrei Busnatu, Razvan Dobre

2. Content
About Monitoring
Graphite
How we deployed Graphite
Performance

3. About Monitoring Monitoring Monitoring – observe how system evolves
Graphite
Opentsdb
InfluxDB
etc
Alerting – notify when system metrics get out of order – in other talk
Nagios
Incinga
Zabyx

4. What is does? Stores numeric time-series data
Graphite
What is does?
Stores numeric time-series data
wisper files better than rrd
Local files on disk
Renders graphs of the data on demand

5. Graphite - Components Carbon Graphite backend daemons Subcomponents
Carbon-relay – metrics forwarder and sharding
Carbon-aggregator – metrics aggregator
Carbon-cache
Metrics collection server
Listens on port 2003
Metric format
a.b.c.d.e.value
How to send a metric
echo "local.random.diceroll 4 `date +%s`" | nc -q0 ${SERVER} ${PORT}

6. Graphite - Components Graphite web:
Django web app for viewing metrics and creating basic dashboards
Support memcache for caching
Requires a DB for authentication and saving dashboards
Whisper
File-based time-series database
Better that RRDs – supports backfill
One file per metric

7. Graphite - Architecture

8. How we deployed Graphite – v0
Carbon relay uses
RELAY_METHOD = consistent-hashing
Pros
it worked for small number of metrics
Cons
Metrics for single node were scattered on all nodes
Hard to clean up
Cpu intensive at relay level
Carbon
Cache
Graphite
Web
Carbon Relay
Carbon Cache
Carbon Relay
Carbon Cache
Carbon Cache

9. How we deployed Graphite – v1
Carbon relay uses
RELAY_METHOD = rules
Pros
it worked for larger number of metrics
Easier to clean up
Impact in case of failure was lower
Data Replication
Cons
Python application don’t do very well with millions of metrics
Carbon
Cache
Graphite
Web
Carbon Relay
Carbon Cache
Carbon Relay
Carbon Cache
Carbon Cache

10. How we deployed Graphite – v2

11. How we deployed Graphite – v2
Replaced carbon-cache with:
go-carbon for data collection
Carbon-server for data querying
Carbon-zipper for carbon-server aggregator
Replaced carbon-relay with carbon-c-relay
Graphite web connects to carbon-zipper
Added grafana for proper dashboarding

12. Performance