1. Centralized Logfile Search (a.k.a. Tracing) Vito Baggiolini with Gergo Horanyi, Felix Ehm, Stephen Page

2. Outline Introduction Architecture and current logfile sources (FECs) Kibana, the search GUI Demo Next steps 16/05/2014Vito Baggiolini2

3. Introduction Purpose: Facilitate diagnostics of the operational control system – No need to know where exactly logfiles are stored – Make logfile history easy to search and correlate – Search logfiles across all levels (Java, CMW, FESA, Drivers, OS, …) Scope: – For operations, not for development (maybe later) – Mainly to search history, not for live logging (-> use cmw-admin or tail -f) – To search many systems at once, less suitable to for individual systems (better to go the log file directly) Implementation: – Before: $plunk, very intuitive and user friendly, commercial, closed-source, – Now: ElasticSearch + Kibana, free, open-source, less intuitive + user friendly 16/05/2014Vito Baggiolini3

4. Current Architecture and log sources Syslog OS Syslog FESA2 cmw-log FESA3 cmw-log CMW NFS fgcd-log Sources Java Process Logstash1 Logstash2 Parse to key-value Elastic Search Store and search Kibana User Interface 16/05/2014Vito Baggiolini4

5. Query Bar Filters Fields Results vs Time Detailed Messages To select Time Window 16/05/2014Vito Baggiolini5 Kibana

6. 16/05/2014Vito Baggiolini6 Query Bar Filters To select Time Window To share searches

7. 16/05/2014Vito Baggiolini7 Fields Results vs Time Detailed Messages Click on field -> top values Filter out repeated

8. Kibana Demo http://logs.cern.ch Select time window, zoom in, zoom out Search – Enter query for keyword – Enter query for a field Click on fields to see top values and filter Filter out repeated messages Modify existing filters Display results as tables Share results with others 16/05/2014Vito Baggiolini8

9. Demo… 16/05/2014Vito Baggiolini9

10. How to make the best of Kibana Do a simple Search (e.g. for *) and filter Click on field names (not on boxes) to see top values) – Top values only over last 500! – Focus on or exclude a given event 16/05/2014Vito Baggiolini10

11. How to make the best of Kibana Do a simple Search (e.g. for system:FESA3) and then use filters Click on field names (not on boxes) to see top values) – Top values only over last 500! – Focus on or exclude a given event Play with Filters – E.g. edit existing filters Share temporary views with others Don’t yet invest too much in Dashboards => fix displays(?) Use Lucene Query Language for advanced queries and filters 16/05/2014Vito Baggiolini11

12. Next steps Make system operational (now in BETA) – Verify scalability – Add redundancy where feasible – Add sanity checks to verify proper functioning Make sure log sources and are parsed correctly to key/value – Finalize standardization of fields pushed into Elastic Search – https://wikis.cern.ch/pages/viewpage.action?pageId=73041150 https://wikis.cern.ch/pages/viewpage.action?pageId=73041150 Add Java logging input (gradually, servers first) 16/05/2014Vito Baggiolini12