1. Log Management Systems
A comparison of message and metric management solutions
Log Management Systems
Presenter: Mehdi Hamidi )

2. Introduction

4. What is LOG? Combination of Time, Tag and Message
Indicates State of Applications (?)
Human and Machine Readable Messages (?)

7. Level of logs in syslog standard
From Debug to Panic
rsyslog, syslog, syslog-ng
/var/log/syslog /var/log/rsyslog

8. Companies and Businesses Even Personal Use! (Twitter, Sensors,... )
Importance of logs
Companies and Businesses
Even Personal Use! (Twitter, Sensors,... )

9. LogAnalyzer: a simple solution :)

10. LogAnalyzer: a simple solution :)

11. Importance of Logging Systems
WHAT Actually We NEED?
Collect
Messages
Metrics
Store
Visualize
Alert

12. Importance of Logging Systems
Heterogeneous Environment
Write our own script for each type of log (?)
Not in an enterprise environment with lots of devices and services!
Technical Fragility and dependency to Individuals
Strong Dependency to knowledge about underlying process

15. Commercial Solutions Splunk (500M/Day is Free, then: 5,000,000 $)
Nagios Everything is restricted to Nagios Concept No separation between metrics and messages No stylish diagrams (in free solution) Problems in cloud infrastructure No realtime monitoring No manipulating messages (1,995 $ for commercial solution)
Online Services

16. Good logging system Specifications
Have a common interface
Decouple data sources from data outputs
Prevent mentioned dependencies
No effect of adding new data source/output
Reliability
Persistent Buffering
Extensibility
High Availability
Load Balancing
Robustness

17. Lots of OpenSource Bricks (OSB!)
Logging Systems:
Fluentd
LogStash
GrayLog
Logalice
Rsyslog
Scribe
Message Stores:
ElasticSeach
Hadoop
MongoDB
File
RDBMS
Redis
...
Visualization (Dashboards):
Kibana
Grafana
Gaylog-WebUI
PacketBeat
Chronograph
...
Alerting:
Kapacitor
Skyline
Oculus
Cabot
Metric Stores:
InfluxDB
Prometheus
Graphite
...

18. Log Nature Semistructured or Unstructured Generated Massively
More Written and less Read
(That's why we use NoSQL)

19. Popular Stacks (metrics): TICK Stack

20. Popular Stacks (Messages) : ELK Stack

23. Fluentd VS LogStash

24. Overview Fluentd: Written in Cruby
Used in Google Cloud Platform and Kubernetes
Maintained by Tresure Data
Logstash:
Written in Jruby
Used in ELK Stack
Maintained by Elastic Co.
Both use their own RubyGems Repo
Out of the box nature, less dependencies

25. Configuration Fluentd: Each Input is tagged Logs are routed by tags
Logstash:
All inputs are Gathered and Scattered
Conditional Outputs, No tags

26. Configuration

27. Transport and Buffering
Fluentd: built-in
LogStash: bundled Redis version 5.3: persistent buffering

28. Full Buffer or Output Exception occurrence
Fluentd: Exception: streaming
Block input plugin: batch
Drop oldest chunk: monitoring
LogStash:
Retry
Discard
Dead Letter Queuing

29. High Availability and Load balancing

30. High Availability and Load balancing

31. High Availability and Load balancing

32. High Availability and Load balancing

33. Fluentd: 40 M Logstash: 120 M (in big clusters matter)
Memory
Fluentd: 40 M
Logstash: 120 M
(in big clusters matter)

34. Forwarders Fluentd: Fluentbit (Written in C)
Fluentd-Forwarder (Written in Go)
(all in one)
LogStash:
Filebeat
Metricbeat
Packetbeat
Winlogbeat
(beat family: separated component for each purpose)

35. Community and Support Fluentd: Poor Japanese Blogs Google Group
Logstash: Rich Documents
Blogs
IRC
Meetups and Certs

36. Plugins are maintained more by other people.
Fluentd
Plugins
Verified
Input/Output
554 44
Filter 90 8
Parser 30 2
Formatter 6
Obsolete
Plugins are maintained more by other people.

37. All Plugins are in a Single GitHub Repo.
LogStash
Plugins
Input 52
Filter 46
Output 55
All Plugins are in a Single GitHub Repo.

38. Questions?

39. Thanks You! ahamidi66@gmail.com @eXtrem0us

40. Resources