2. Combining Metrics and Logs for Holistic System/Application Analysis
Sharath Kumar M N
Elasticsearch Solutions Architect, Peoplesoft
Oracle
May 10, 2017
Confidential – Oracle Internal/Restricted/Highly Restricted

3. Metrics :- A metric is a quantifiable measure that is used to track and assess the status of a specific process
Confidential – Oracle Internal/Restricted/Highly Restricted

4. Logs:- record of incidents or observations
Confidential – Oracle Internal/Restricted/Highly Restricted

5. Need For Holistic Analysis
Confidential – Oracle Internal/Restricted/Highly Restricted

6. Metrics
[01/Nov/2016:21:01: ] "apache" cpu 30
[01/Nov/2016:21:01: ] " apache" cpu 35
[01/Nov/2016:21:01: ] "apache" cpu 32
Logs
[01/Nov/2016:21:01: ] "GET /cluster HTTP/1.1"
[01/Nov/2016:21:01: ] "GET /cpc/auth.do?loginsetup=true&targetPage=%2Fcpc%2F HTTP/1.1"
[01/Nov/2016:21:01: ] "GET /cpc?loginsetup=true&targetPage=%252Fcpc%252F HTTP/1.1" 302 -
Confidential – Oracle Internal/Restricted/Highly Restricted
Confidential – Oracle Internal/Restricted/Highly Restricted
Confidential – Oracle Internal/Restricted/Highly Restricted 6 6

7. Application Metrics + Logs
Confidential – Oracle Internal/Restricted/Highly Restricted

8. Architecture Collect/Ship Data Datastore, Search, analytics engine
Visualize
Edge Nodes
Queue
Parse/ Enrich Data
Confidential – Oracle Internal/Restricted/Highly Restricted

9. Filebeat Lightweight Shipper for Logs Tail files
Ensure At-Least-Once Delivery
Extensions: Modules
Filebeat comes with internal modules (Apache, Nginx, System, and MySQL) 
Filebeat consists of two main components: prospectors and harvesters.They work together to tail files and send event data to the output that you specify.
A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output.
A prospector is responsible for managing the harvesters and finding all sources to read find all files on the drive that match the defined glob paths and starts a harvester for each file
Filebeat keeps the state of each file and frequently flushes the state to disk in the registry file.
Confidential – Oracle Internal/Restricted/Highly Restricted

10. Config file (.yml)
Confidential – Oracle Internal/Restricted/Highly Restricted

11. Metricbeat Lightweight Shipper for Metrics
Extensions: Modules and Metricsets
Come with many modules (Apache, Nginx, System, Redis, MySQL,PostgreSQL, MongoDB, Kafka ….) 
Confidential – Oracle Internal/Restricted/Highly Restricted

12. Config file (.yml)
Confidential – Oracle Internal/Restricted/Highly Restricted

13. Event Structure
Confidential – Oracle Internal/Restricted/Highly Restricted

14. Logstash
Data collection engine with real- time pipelining capabilities 
Unify data from disparate sources and normalize the data into destinations of choice
Parse and Enrich Logs
Confidential – Oracle Internal/Restricted/Highly Restricted

15. Logstash Pipeline
Confidential – Oracle Internal/Restricted/Highly Restricted

16. Parse Logs
frank [10/Oct/2016:13:55: ] "GET /apache_pb.gif HTTP/1.0" {
"clientip": " ",
"userId": "frank",
"timestamp":"10/Oct/2016:13:55: ",
"verb": "GET",
"request": "/apache_pb.gif",
"httpversion": "HTTP/1.0",
"response": 200
"bytes" :2336 }
filter {
grok{
match=>{ "message"=>"%{IPORHOST:clientip} %{USER:userId} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
Confidential – Oracle Internal/Restricted/Highly Restricted

17. Enrich Logs
frank [10/Oct/2016:13:55: ] "GET /apache_pb.gif HTTP/1.0" { .
"geoip": {
"continent_name": "North America",
"country_iso_code": "US",
"region_name": "California",
"city_name": "Mountain View",
"location": { "lat": , "lon": } }
filter {
geoip {
source => "clientip" }
Confidential – Oracle Internal/Restricted/Highly Restricted

18. Elasticsearch Distributed, full text search and analytics engine
Based on Lucene
Its RESTful
Highly available
Very Fast
Connectors for Big Data – “Elasticsearch-Hadoop”
Confidential – Oracle Internal/Restricted/Highly Restricted

19. Kibana
Analytics and visualization platform designed to work with Elasticsearch
 Browser-based interface for near real time analytics
Option of visualizing data in time series.
Perform advanced data analysis and visualize your data in a variety of charts, tables, and maps
Confidential – Oracle Internal/Restricted/Highly Restricted

20. Lets See it in Action!!
Confidential – Oracle Internal/Restricted/Highly Restricted

21. Demo – Attachment Processing
Confidential – Oracle Internal/Restricted/Highly Restricted

22. Demo – Attachment Processing
Confidential – Oracle Internal/Restricted/Highly Restricted

23. Demo – Attachment Processing
Confidential – Oracle Internal/Restricted/Highly Restricted

24. The Best Part is ….
Confidential – Oracle Internal/Restricted/Highly Restricted

25. Open Source Powerful Simple
Confidential – Oracle Internal/Restricted/Highly Restricted
Confidential – Oracle Internal/Restricted/Highly Restricted 25

26. Benefits Holistic Analysis / Unified Analytics
Better Troubleshooting / RCA
Deeper Insights into System and Applications
“Data” is Power – Intrinsic Values
Confidential – Oracle Internal/Restricted/Highly Restricted

27. Sample Use Cases DevOps , IoT, ITOps , Application Monitoring & Analytics
Confidential – Oracle Internal/Restricted/Highly Restricted

28. Q&A
Confidential – Oracle Internal/Restricted/Highly Restricted