1. Using Logstash for PI
Robert Mckeown
Dec 12, 2014 1

2. ETL context
Use Logstash as ETL (Extract, Transform, Load) tool to transform data to required PI format
Logstash

3. Supporting information
Main logstash web site
The Logstash Book
Logstash Foru lots of good Q&A
Doug McClure's 'Logstash is your friend' doc – log-centric but has a good end-to-end example and advice.
This doc is a quick skim to get you started. To become proficient, refer the sites above!

4. Key logstash functions
Logstash is an event pipeline
Inputs → codecs/filters → outputs
Inputs generate events, codecs and filters modify them, outputs ship them
Types are set and used mainly for filter activiation. They always persist with the event
Tags can be used to specify an order for event processing (apply filter A, then filter D then filter F) as well as event routing to specific filters and outputs
Conditionals give you if, else if etc, as well as comparison tests and boolean logic for sophisticated analysis, processing and routing
Chart from Doug McClure

5. Key logstash functions
Pipeline
PI files
myType
Metric files
scacsv
Original chart from Doug McClure

6. Standard set of plug-ins plus two PI specific ones
scapivot
scacsv
scabmcfile

7. Installation
Download Logstash from
Unpack in a dir of your choice
Add logstash to your $PATH for convenience (if desired )
Install additional standard plug-ins aka 'contribs'
cd /path/to/your/logstash
bin/plugin install contrib
Obtain the SCAPI Plugin package
Currently avail here in the CSI Predict, Search and Event Analytics technial sales forum
Note: Logstash is already installed on the current 'standard' SoftLayer1.3 images

8. Running Logstash
Only additional item beyond standard Logstash invocation is to ensure that you reference the custom SCA plugins on the command line (if you are using them)
e.g. my
Logstash is installed at /home/rmckeown/dev/logstash-1.4.2
Plugins installed in /home/rmckeown/dev/logstashDev
Running Logstash would be
/home/rmckeown/dev/logstash-1.4.2/bin/logstash -f myConf.conf –pluginpath /home/rmckeown/dev/logstashDev/scaLogstash
Use of $PATH can make this a bit shorter

9. Example 1 Skinny-format Multiple 'groups' implied No header
'group' name
Date ok(?)
'metric' name
Metric value
Device number
Skinny-format
Multiple 'groups' implied
No header

10. Example 1 See http://logstash.net/docs/1.4.2/inputs/stdin
Host which processed record
Actual record
Timestamp when message/record was processed

11. Example 1 Outputs data using ruby 'awesome_print' Outputs data as json
Output formatted by jsonlint

12. Example 1 - filter Create desired columns Remove arbitary fields
The CSV filter takes an event field containing CSV data, parses it, and stores it as individual fields (can optionally specify the names). This filter can also parse data with any separator, not just commas.
Create desired columns
Remove arbitary fields
Columns added
Field 'interval' removed
Note: two timestamps

13. Example 1 – first csv Desired data output but No header
Data not separated by group

14. Example 1 – Conditional & CSV
Example of conditional
Not standard PI name
No header

15. Still 'skinny'! - Need to 'pivot'
Example 1 – using scaCSV
Custom operator
Output files
Still 'skinny'! - Need to 'pivot'

16. Example 2 – scapivot cpu net Custom operator
Values (metric identities) become column names
Metric values mapped to correct column
cpu
net

17. Example 2 Meta-data in header Selection of header and data lines
Simple format clean up of individual feels (e.g. 'G', '%', '-')

18. Example 2 – basic classification by type
Conditional with regular expression – match
Any line that starts with '20' – this will be our date
Classify these as DATA_Line and for output later
No tags added
Tags added

19. Example 2 – capture timestamp via Grok
Grok is one of the most important plug-ins for use with PI (see )
Grok : Parse arbitrary text and structure it
are used to convert matched strings e.g.TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
For developing Grok patterns is very useful

20. Example 2 – Grok / grokdebug

21. Example 2 – capture timestamp via Grok

22. Example 2 – cleaning up fields
Convert a string field by applying a regular expression and replacement.
Here we are replacing - or % with “”

23. Example 2 – splitting in to LogStash 'CSV'
Aligns with input file
Reformatting timestamp
Watch this spot !

24. Example 2 – Outputting
Subset of fields
Still need to determine this

25. Example 2 – Determining Server Name (associative behavior)
Events are generally **independent**
'Multiline Events' are an exception
Cannot obviously carry information from is available across events
In our NAB example, the server identity is in separate 'event' in the header.
Processing information 'across' events is more challenging
Think outside the box (or outside single instances of Logstash)
Two-step approach. May be others
Of course, it doesn't have to be logstash either
Logstash
Will use as replacement
ServerName : serverX
serverMap
Logstash main processing
Final output
Original file

26. Example 2 – Determining Server Name

27. Example 2 – Replacing server name (translate)

28. Extending Logstash aka Building custom plug-ins
Plug-ins are written primarily in Ruby
Can call out to Java easily (since Logstash runs on jRuby )
Chapter 8 of The Logstash Book – 'Extending Logstash' has all the details
Also, look at the source code for existing plug-ins for lots of good examples on how to proceed

29. Location of plug-ins
Can also specify a directory outside Logstash installation and work out of that
mkdir -p /etc/logstash/{inputs,filters,outputs}
Specify this path when running logstash e.g.
..../logstash/bin/logstash –pluginpath /etc/

30. Extending Logstash aka Building custom plug-ins – scaJDBC (new plug-in & Java interaction)
Plug in name
New config options
Standard CSV

31. Extending Logstash aka Building custom plug-ins - scaJDBC
Inherit from Base
Plug-in name
Config
Register at runtime

32. Create a brand new event
Almost Java!
How many columns?
Create a brand new event
Assign attribute/values for each data item returned from DB
Finalize and dispatch!