Log Management Systems A comparison of message and metric management solutions Log Management Systems Presenter: Mehdi Hamidi ( @eXtrem0us )

Introduction

What is LOG? Combination of Time, Tag and Message Indicates State of Applications (?) Human and Machine Readable Messages (?)

Level of logs in syslog standard From Debug to Panic rsyslog, syslog, syslog-ng /var/log/syslog /var/log/rsyslog

Companies and Businesses Even Personal Use! (Twitter, Sensors,... ) Importance of logs Companies and Businesses Even Personal Use! (Twitter, Sensors,... )

LogAnalyzer: a simple solution :)

LogAnalyzer: a simple solution :)

Importance of Logging Systems WHAT Actually We NEED? Collect Messages Metrics Store Visualize Alert

Importance of Logging Systems Heterogeneous Environment Write our own script for each type of log (?) Not in an enterprise environment with lots of devices and services! Technical Fragility and dependency to Individuals Strong Dependency to knowledge about underlying process

Commercial Solutions Splunk (500M/Day is Free, then: 5,000,000 $) Nagios Everything is restricted to Nagios Concept No separation between metrics and messages No stylish diagrams (in free solution) Problems in cloud infrastructure No realtime monitoring No manipulating messages (1,995 $ for commercial solution) Online Services

Good logging system Specifications Have a common interface Decouple data sources from data outputs Prevent mentioned dependencies No effect of adding new data source/output Reliability Persistent Buffering Extensibility High Availability Load Balancing Robustness

Lots of OpenSource Bricks (OSB!) Logging Systems: Fluentd LogStash GrayLog Logalice Rsyslog Scribe Message Stores: ElasticSeach Hadoop MongoDB File RDBMS Redis ... Visualization (Dashboards): Kibana Grafana Gaylog-WebUI PacketBeat Chronograph ... Alerting: Kapacitor Skyline Oculus Cabot Metric Stores: InfluxDB Prometheus Graphite ...

Log Nature Semistructured or Unstructured Generated Massively More Written and less Read (That's why we use NoSQL)

Popular Stacks (metrics): TICK Stack

Popular Stacks (Messages) : ELK Stack

Fluentd VS LogStash

Overview Fluentd: Written in Cruby Used in Google Cloud Platform and Kubernetes Maintained by Tresure Data Logstash: Written in Jruby Used in ELK Stack Maintained by Elastic Co. Both use their own RubyGems Repo Out of the box nature, less dependencies

Configuration Fluentd: Each Input is tagged Logs are routed by tags Logstash: All inputs are Gathered and Scattered Conditional Outputs, No tags

Configuration

Transport and Buffering Fluentd: built-in LogStash: bundled Redis version 5.3: persistent buffering

Full Buffer or Output Exception occurrence Fluentd: Exception: streaming Block input plugin: batch Drop oldest chunk: monitoring LogStash: Retry Discard Dead Letter Queuing

High Availability and Load balancing

High Availability and Load balancing

High Availability and Load balancing

High Availability and Load balancing

Fluentd: 40 M Logstash: 120 M (in big clusters matter) Memory Fluentd: 40 M Logstash: 120 M (in big clusters matter)

Forwarders Fluentd: Fluentbit (Written in C) Fluentd-Forwarder (Written in Go) (all in one) LogStash: Filebeat Metricbeat Packetbeat Winlogbeat (beat family: separated component for each purpose)

Community and Support Fluentd: Poor Japanese Blogs Google Group Logstash: Rich Documents Blogs IRC Meetups and Certs

Plugins are maintained more by other people. Fluentd Plugins Verified Input/Output 554 44 Filter 90 8 Parser 30 2 Formatter 6 Obsolete Plugins are maintained more by other people.

All Plugins are in a Single GitHub Repo. LogStash Plugins Input 52 Filter 46 Output 55 All Plugins are in a Single GitHub Repo.

Questions?

Thanks You! ahamidi66@gmail.com @eXtrem0us

Resources http://logz.io/blog/fluentd-logstash http://docs.fluentd.org/articles/buffer-plugin-overvie https://prometheus.io/docs/introduction/comparison http://logz.io/blog/elk-stack-5-0 https://www.youtube.com/watch?v=1ye0-sityBw https://www.youtube.com/watch?v=0lAHrspviIs&list=PL62pIycqXx-TPwtk4JDd0wMuFAyP0gU1y https://www.youtube.com/watch?v=mfb0R7azKZc https://www.youtube.com/watch?v=_BAWi9Zhmic