Centralized Logfile Search (a.k.a. Tracing) Vito Baggiolini with Gergo Horanyi, Felix Ehm, Stephen Page

Outline Introduction Architecture and current logfile sources (FECs) Kibana, the search GUI Demo Next steps 16/05/2014Vito Baggiolini2

Introduction Purpose: Facilitate diagnostics of the operational control system – No need to know where exactly logfiles are stored – Make logfile history easy to search and correlate – Search logfiles across all levels (Java, CMW, FESA, Drivers, OS, …) Scope: – For operations, not for development (maybe later) – Mainly to search history, not for live logging (-> use cmw-admin or tail -f) – To search many systems at once, less suitable to for individual systems (better to go the log file directly) Implementation: – Before: $plunk, very intuitive and user friendly, commercial, closed-source, – Now: ElasticSearch + Kibana, free, open-source, less intuitive + user friendly 16/05/2014Vito Baggiolini3

Current Architecture and log sources Syslog OS Syslog FESA2 cmw-log FESA3 cmw-log CMW NFS fgcd-log Sources Java Process Logstash1 Logstash2 Parse to key-value Elastic Search Store and search Kibana User Interface 16/05/2014Vito Baggiolini4

Query Bar Filters Fields Results vs Time Detailed Messages To select Time Window 16/05/2014Vito Baggiolini5 Kibana

16/05/2014Vito Baggiolini6 Query Bar Filters To select Time Window To share searches

16/05/2014Vito Baggiolini7 Fields Results vs Time Detailed Messages Click on field -> top values Filter out repeated

Kibana Demo Select time window, zoom in, zoom out Search – Enter query for keyword – Enter query for a field Click on fields to see top values and filter Filter out repeated messages Modify existing filters Display results as tables Share results with others 16/05/2014Vito Baggiolini8

Demo… 16/05/2014Vito Baggiolini9

How to make the best of Kibana Do a simple Search (e.g. for *) and filter Click on field names (not on boxes) to see top values) – Top values only over last 500! – Focus on or exclude a given event 16/05/2014Vito Baggiolini10

How to make the best of Kibana Do a simple Search (e.g. for system:FESA3) and then use filters Click on field names (not on boxes) to see top values) – Top values only over last 500! – Focus on or exclude a given event Play with Filters – E.g. edit existing filters Share temporary views with others Don’t yet invest too much in Dashboards => fix displays(?) Use Lucene Query Language for advanced queries and filters 16/05/2014Vito Baggiolini11

Next steps Make system operational (now in BETA) – Verify scalability – Add redundancy where feasible – Add sanity checks to verify proper functioning Make sure log sources and are parsed correctly to key/value – Finalize standardization of fields pushed into Elastic Search – Add Java logging input (gradually, servers first) 16/05/2014Vito Baggiolini12