1. Introduction to Elasticsearch with basics of Lucene May 2014 Meetup
Rahul Jain
@rahuldausa @

2. Who am I Software Engineer 7 years of software development experience
Built a platform to search logs in Near real time with volume of 1TB/day#
Worked on a Solr search based SEO/SEM software with 40 billion records/month (Topic of next talk?)
Areas of expertise/interest
High traffic web applications
JAVA/J2EE
Big data, NoSQL
Information-Retrieval, Machine learning #

3. Agenda IR Overview Basic Concepts Lucene Elasticsearch
Logstash & Kibana - Short Introduction
Q&A

4. Information Retrieval (IR)
”Information retrieval is the activity of obtaining information resources (in the form of documents) relevant to an information need from a collection of information resources. Searches can be based on metadata or on full-text (or other content-based) indexing” - Wikipedia

5. Basic Concepts
Term t : a noun or compound word used in a specific context
tf (t in d) : term frequency in a document
measure of how often a term appears in the document
the number of times term t appears in the currently scored document d
idf  (t) : inverse document frequency
measure of whether the term is common or rare across all documents, i.e. how often the term appears across the index
obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.
boost (index) : boost of the field at index-time
boost (query) : boost of the field at query-time

6. Credit: http://http://whatisgraphsearch.com/
Basic Concepts TF - IDF
TF - IDF = Term Frequency X Inverse Document Frequency
Credit:

7. Apache Lucene

8. Apache Lucene Fast, high performance, scalable search/IR library
Open source
Initially developed by Doug Cutting (Also author of Hadoop)
Indexing and Searching
Inverted Index of documents
Provides advanced Search options like synonyms, stopwords, based on similarity, proximity.

9. Lucene Internals - Inverted Index
Credit:

10. Lucene Internals (Contd.)
Defines documents Model
Index contains documents.
Each document consist of fields.
Each Field has attributes.
What is the data type (FieldType)
How to handle the content (Analyzers, Filters)
Is it a stored field (stored="true") or Index field (indexed="true")

11. Indexing Pipeline
Analyzer : create tokens using a Tokenizer and/or applying Filters (Token Filters)
Each field can define an Analyzer at index time/query time or the both at same time.
Credit :

12. Analysis Process - Tokenizer
WhitespaceAnalyzer
Simplest built-in analyzer
The quick brown fox jumps over the lazy dog.
[The] [quick] [brown] [fox] [jumps] [over] [the] [lazy] [dog.]
Tokens

13. Analysis Process - Tokenizer
SimpleAnalyzer
Lowercases, split at non-letter boundaries
The quick brown fox jumps over the lazy dog.
[the] [quick] [brown] [fox] [jumps] [over] [the] [lazy] [dog]
Tokens

14. Elasticsearch

15. Introduction Enterprise Search platform for Apache Lucene Open source
Highly reliable, scalable, fault tolerant
Support distributed Indexing, Replication, and load balanced querying

16. Elasticsearch - Features
Distributed RESTful search server
Document oriented
Domain Driven
Schema less
Restful
Easy to scale horizontally

17. Elasticsearch - Features
Highlighting
Spelling Suggestions
Facets (Group by)
 Query DSL
based on JSON to define queries
Automatic shard replication, routing
Zen discovery
Unicast
Multicast
Master Election
Re-election if Master Node fails

18. APIs HTTP RESTful Api Java Api Clients
perl, python, php, ruby, .net etc
All APIs perform automatic node operation rerouting.

19. How to start It’s this Easy.

20. Operations

21. INDEX CREATION
curl -XPUT " -d‘ { "title": "The Godfather", "director": "Francis Ford Coppola", "year": 1972 }'
Credit:

22. INDEX CREATION RESPONSE
Credit:

23. UPDATE curl -XPUT "http://localhost:9200/movies/movie/1" -d' {
"title": "The Godfather",
"director": "Francis Ford Coppola",
"year": 1972,
"genres": ["Crime", "Drama"] }'
New field
Updated Version
Credit:

24. GET curl -XGET "http://localhost:9200/movies/movie/1" -d''
Credit:

25. DELETE curl -XDELETE "http://localhost:9200/movies/movie/1" -d''
Credit:

26. SEARCH Search across all types in the movies index.
Search across all indexes and all types
Search across all types in the movies index.
Search explicitly for documents of type movie within the movies index.
curl -XPOST " -d' {
"query": {
"query_string": {
"query": "kill" } }'
Credit:

27. SEARCH RESPONSE
Credit:

28. Updating existing Mapping
curl -XPUT " -d' {
"movie": {
"properties": {
"director": {
"type": "multi_field",
"fields": {
"director": {"type": "string"},
"original": {"type" : "string", "index" : "not_analyzed"} } }'
Credit:

29. Cluster Architecture
Source:

30. Index Request
Source:

31. Search Request
Source:

32. Who are using Github Stumbleupon Soundcloud Datadog Stackoverflow
Many more…

33. Logstash

34. Logstash Open Source, Apache licensee Written in JRuby
Part of Elasticsearch family
Current version: 1.4.0
This talk is with 1.3.3

35. Logstash Multiple Input/ Multiple Output Centralize logs Collect Parse
Forward/Store

36. Architecture
Source:

37. Logstash – life of an event
Input  Filters  Output
Filters are processed in order of config file
Outputs are processed in order of config file
Input: Input stream
File input (tail)
Log4j
Redis
Syslog
and many more…

38. Logstash – life of an event
Codecs : decoding log messages
Json
Multiline
Netflow
and many more…
Filters : processing messages
Date – Date format
Grok – Regular expression based extraction
Mutate – Change data type
Output : storing the structured message
Elasticsearch
Mongodb
Nagios

39. Quick Start < 1.3.3 version:
basic-agent.conf :
input {
tcp {
type => "apache"
port => 3333 }
output {
stdout {
debug => true
elasticsearch {
embedded => true
< version:
java -jar logstash flatjar.jar agent -f agent.conf – web
1.4 version:
bin/logstash agent –f agent.conf
bin/logstash –web

40. Kibana

41. Source: http://www. slideshare

42. Source: http://www. slideshare

43. Analytics
Analytics source : Kibana.org based on ElasticSearch and Logstash
Image Source :

44. Join us @ http://www.meetup.com/Hyderabad-Apache-Solr-Lucene-Group/
Thanks! @rahuldausa on twitter and slideshare
Find Interesting ?
Join