The ELK stack - get to know logs Igor Rudyk DevOps / System Integrator

Agenda Introduction. What is ELK, and why do we need it? The ELK stack Logstash ElasticSearch Kibana Architecture Demo

Can you check the errors from yesterday between 9:09 and 9:27

So what’s a log

Log = timestamp + data

Real Real Lifecycle of a log Transmit Analyze Record Store Delete

Default problem Tools? Multiple log time formats Apr 28 20:21:59 [27/Apr/2015:07:05:28 +0000] 071012 09:27:32 Mon, 27-Apr-15 06:27:02 UTC 2015-04-28 20:07:51 +0000 Starts not with timestamp or without timestamp Error messages with really unhelpful info No rotation No scaling Tools? grep awk / sed / cut less / tail vi / vim regular expression ...

Logging Solutions Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial Logstash Logstash shipper or logstash-forwarder (Lumberjack) RPM installation Logstash shipper or logstash-forwarder (Lumberjack, encrypted transport is the default) Output plugins Central server-master with a hot-standby in case of failure Codecs plugins Grok debugger ElasticSearch, MongoDB, AWS S3 and much more Kibana, graylog2 Riemann NO fluentd Input plugins Install from source or via gem Load-balance between multiple hosts or have a master with a hot-standby in case of failure Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected splunk Splunk Universal Forwarder SSL security Splunk YES Graylog2 ElasticSearch loggly Hosted

What is ELK, and why do we need it? ELK is a stack of programs that help dealing with logs. Includes: Aggregation of logs Search capabilities Aggregation of statistics Visualizations

Logstash Filters Outputs Unstructured Documents

Inputs Logs: Lumberjack - resilient, compressed, secure (logstash-forwarder) Remote syslog Files Devices: Event log, Collectd Netflow, WMI Event Queue: Redis, RabbitMQ Kafka, ZeroMQ Streaming APIs: Twitter Email (IMAP) Amazon S3, ganglia, sqs, varnishlog, etc .... http://logstash.net/docs/1.4.2/ - Full list

Filters Why Do I like Logstash? grep date json grok .... http://logstash.net/docs/1.4.2/ - Full list Why Do I like Logstash? It uses Grok filter for parsing standard and non standard logs: Log Line: 27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0 Pattern: %{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}

Outputs Storage: ElasticSearch MongoDB Event Queue: S3 Redis, RabbitMQ Graphite File ... Notification: Zabbix Nagios Riemann PagerDuty Email Event Queue: Redis, RabbitMQ Kafka, ZeroMQ tcp/udp SaaS: AWS CloudWatch Hipchat Jira .... http://logstash.net/docs/1.4.2/ - Full list

Logstash - Forwarder (Shipper) Configuration file { "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } "/usr/share/tomcat7/logs/*.activities.log" "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } } ]

Logstash-Indexer Configuration file input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } output { elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" }

ElasticSearch Configuration file (yaml-based configuration) cluster.name: [[elasticsearch_cluster_name]] node.name: "[[node_name]]" node.master: false / true node.data: false / true index.number_of_replicas: 1 #Security discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]] action.disable_close_all_indices: true action.disable_delete_all_indices: true action.disable_shutdown: true script.disable_dynamic: true

The ELK stack General Architecture Logstash ElasticSearch Kibana

The ELK stack Our Scaled Architecture Logstash-Indexer ElasticSearch Kibana Logstash-Forwarder Logstash-Indexer ElasticSearch Kibana Logstash-Indexer ElasticSearch Kibana

DEMO