Introduction to Lucene Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.

Slides:



Advertisements
Similar presentations
Lucene in action Information Retrieval A.A – P. Ferragina, U. Scaiella – – Dipartimento di Informatica – Università di Pisa –
Advertisements

Assignment 2: Full text search with Lucene Mathias Mosolf, Alexander Frenzel.
Lucene Near Realtime Search Jason Rutherglen & Jake Mannix LinkedIn 6/3/2009 SOLR/Lucene Users Group San Francisco.
Lucene/SOLR 2: Lucene search API
Lucene/Solr Architecture
Lucene Tutorial Based on Lucene in Action Michael McCandless, Erik Hatcher, Otis Gospodnetic.
Introduction to Information Retrieval Introduction to Information Retrieval Lucene Tutorial Chris Manning and Pandu Nayak.
Chapter 5: Introduction to Information Retrieval
© Copyright 2012 STI INNSBRUCK Apache Lucene Ioan Toma based on slides from Aaron Bannert
Advanced Indexing Techniques with Apache Lucene - Payloads Advanced Indexing Techniques with Michael Busch
Advanced Indexing Techniques with
Apache Solr Yonik Seeley 29 June 2006 Dublin, Ireland.
Query Models Use Types What do search engines do.
The Lucene Search Engine Kira Radinsky Modified by Amit Gross to Lucene 4 Based on the material from: Thomas Paul and Steven J. Owens.
Lucene in action Information Retrieval A.A – P. Ferragina, U. Scaiella – – Dipartimento di Informatica – Università di Pisa –
Lucene Part3‏. Lucene High Level Infrastructure When you look at building your search solution, you often find that the process is split into two main.
Information Retrieval in Practice
The Lucene Search Engine Kira Radinsky Based on the material from: Thomas Paul and Steven J. Owens.
Lucene Lab General IR Process Start Indexing (start stepping though all files) Tokenize & stem each file Index 1 st, Index User enters (roughly)
Parametric search and zone weighting Lecture 6. Recap of lecture 4 Query expansion Index construction.
0 How to Customize This Template Choose an appropriate title slide and delete the others Edit presentation title, author names, and affiliation(s) Choose.
Overview of Search Engines
Introduction to Information Retrieval Introduction to Information Retrieval Lucene Tutorial Chris Manning, Pandu Nayak, and Prabhakar Raghavan.
Implementing search with free software An introduction to Solr By Mick England.
Full-Text Search with Lucene Yonik Seeley 02 May 2007 Amsterdam, Netherlands.
Full-Text Search with Lucene Yonik Seeley 02 May 2007 Amsterdam, Netherlands slides:
1 Introduction to Lucene Rong Jin. What is Lucene ?  Lucene is a high performance, scalable Information Retrieval (IR) library Free, open-source project.
Apache Lucene in LexGrid. Lucene Overview High-performance, full-featured text search engine library. Written entirely in Java. An open source project.
Softvérové knižnice a systémy Vyhľadávanie informácií Michal Laclavík.
1 Lucene Jianguo Lu School of Computer Science University of Windsor.
Lucene Open Source Search Engine. Lucene - Overview Complete search engine in a Java library Stand-alone only, no server – But can use SOLR Handles indexing.
Lucene Boot Camp I Grant Ingersoll Lucid Imagination Nov. 3, 2008 New Orleans, LA.
Lucene Performance Grant Ingersoll November 16, 2007 Atlanta, GA.
Vyhľadávanie informácií Softvérové knižnice a systémy Vyhľadávanie informácií Michal Laclavík.
Introduction to Microsoft Access 2003 Mr. A. Craig Dixon CIS 100: Introduction to Computers Spring 2006.
Lucene Part2. Lucene Jarkarta Lucene ( is a high- performance, full-featured, java, open-source, text search engine.
A Survey of Patent Search Engine Software Jennifer Lewis April 24, 2007 CSE 8337.
Lucene Part1 ‏. Lucene Use Case Store data in a 2 dimensional way How do we do this. Spreadsheet Relational Database X/Y.
Indexing UMLS concepts with Apache Lucene Julien Thibault University of Utah Department of Biomedical Informatics.
0 ICGA Presentation Template This is the institute’s official slide design – use it for all your presentations.
Lucene-Demo Brian Nisonger. Intro No details about Implementation/Theory No details about Implementation/Theory See Treehouse Wiki- Lucene for additional.
“ Lucene.Net is a source code, class-per-class, API-per-API and algorithmatic port of the Java Lucene search engine to the C# and.NET ”
Iccha Sethi Serdar Aslan Team 1 Virginia Tech Information Storage and Retrieval CS 5604 Instructor: Dr. Edward Fox 10/11/2010.
Lucene Boot Camp Grant Ingersoll Lucid Imagination Nov. 4, 2008 New Orleans, LA.
Design a full-text search engine for a website based on Lucene
Web Information Retrieval Textbook by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze Notes Revised by X. Meng for SEU May 2014.
Lucene Jianguo Lu.
Apache Solr Dima Ionut Daniel. Contents What is Apache Solr? Architecture Features Core Solr Concepts Configuration Conclusions Bibliography.
Introduction to Information Retrieval Introduction to Information Retrieval ΜΕ003-ΠΛΕ70: Ανάκτηση Πληροφορίας Διδάσκουσα: Ευαγγελία Πιτουρά Εισαγωγή στο.
Lucene : Text Search IG5 – TILE Esther Pacitti. Basic Architecture.
ΠΛΕ70: Ανάκτηση Πληροφορίας
Lucene Tutorial Chris Manning and Pandu Nayak
Why indexing? For efficient searching of a document
Jianguo Lu School of Computer Science University of Windsor
Searching AND INDEXING Big data
Query Models Use Types What do search engines do.
CS276 Lucene Section.
Searching and Indexing
Safe by default, optimized for efficiency
Implementation Issues & IR Systems
Vores tankesæt: 80% teknologi | 20% forretning
Query Models Use Types What do search engines do.
Lucene in action Information Retrieval A.A
Poster Title Heading Heading Heading Heading Heading Heading
2016 REPORT.
Introduction to Elasticsearch with basics of Lucene May 2014 Meetup
Getting Started With Solr
Table of Contents 1) Understanding Lucene 2) Lucene Indexing
2016 REPORT.
Presentation transcript:

Introduction to Lucene Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata

Open source search engines  Academic – Terrier (Java, University of Glasgow) – Indri, Lemur (C++, and Java too, UMass & CMU) – Zettair (University of Melbourne)  Apache project (non-academic) – Lucene – Apache license, legally easier for commercial use  Lucene – Java search engine library, with many features – Ports/integration to other languages available (C/C++, C#, Python, Ruby, … ) – Other projects on top of Lucene: Solr and others – Used by: LinkedIn, Twitter, CiteSeer, … 2

Lucene: overview 3 Lucene document Documents Tokens Index and dictionary Lucene query Text query Search results Results display User: document building Lucene: Analyzing Lucene: Indexing Lucene: Analyzing Lucene: Searching User: reading search results User: query building UI User User = Programmer

Lucene document building  A document is a collection of Fields  Document: CV of a student, several fields – Last_Name: Banerjee – First_Name: Arkadeep – Age: 24 – Gender: M – Institute: ISI Kolkata – Location: Kolkata – Description: Arkadeep is a highly motivated student who simply loves challenge. He is …  A field can be a text, a number, a range 4

Building document 5 import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; … Document doc = new Document(); doc.add(new StringField(“Last_Name”,”Banerjee”, …)); doc.add(new StringField(“First_Name”,”Arkadeep”, …)); doc.add(new IntField(“Age”,24, …); doc.add(new TextField(“Description”,description, …)); import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; … … Document doc = new Document(); doc.add(new StringField(“Last_Name”,”Banerjee”, …)); doc.add(new StringField(“First_Name”,”Arkadeep”, …)); doc.add(new IntField(“Age”,24, …); doc.add(new TextField(“Description”,description, …)); Now, let’s understand the fields

Building document  Field.Store – NO : Don’t store the field value in the index – YES : Store the field value in the index  Field.Index – ANALYZED : Tokenize with an Analyzer – NOT_ANALYZED : Do not tokenize – NO : Do not index this field – Other options 6 new StringField(“Last_Name”,”Banerjee”,Field.Store.Yes)); new StringField(“First_Name”,”Arkadeep”,Field.Store.Yes)); new NumericField(“Age”,24,Field.Store.Yes); new TextField(“Description”,desc, Field.Store.No)); new StringField(“Last_Name”,”Banerjee”,Field.Store.Yes)); new StringField(“First_Name”,”Arkadeep”,Field.Store.Yes)); new NumericField(“Age”,24,Field.Store.Yes); new TextField(“Description”,desc, Field.Store.No));

Indexing  IndexWriter: primary class for indexing – Stores the index in a directory – RAMDirectory also possible (in memory) 7 StandardAnalyzer analyzer = new StandardAnalyzer(); FSDirectory dir = FSDirectory.open(new File(“directory_path”)); IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer); IndexWriter w = new IndexWriter(index, config); w.addDocument(document); w.close(); StandardAnalyzer analyzer = new StandardAnalyzer(); FSDirectory dir = FSDirectory.open(new File(“directory_path”)); IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer); IndexWriter w = new IndexWriter(index, config); w.addDocument(document); w.close();

Lucene document building  A document is a collection of Fields  Document: CV of a student, several fields – Last_Name: Banerjee – First_Name: Arkadeep – Age: 24 – Gender: M – Institute: ISI Kolkata – Location: Kolkata – Description: Arkadeep is a highly motivated student who simply loves challenge. He is …  A field can be a text, a number, a range 8 What would we like to search with?

Lucene Analyzer  Tokenizes the text in the Fields  Common Analyzers – WhitespaceAnalyzer Splits tokens on whitespace – SimpleAnalyzer Splits tokens on non-letters, and then lowercases – StopAnalyzer Same as SimpleAnalyzer, but also removes stop words – StandardAnalyzer Most sophisticated analyzer that knows about certain token types, lowercases, removes stop words,...

Analysis example  “The quick brown fox jumped over the lazy dog”  WhitespaceAnalyzer – [The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog]  SimpleAnalyzer – [the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog]  StopAnalyzer – [quick] [brown] [fox] [jumped] [over] [lazy] [dog]  StandardAnalyzer – [quick] [brown] [fox] [jumped] [over] [lazy] [dog]

Analysis example 2  “XY&Z Corporation –  WhitespaceAnalyzer – [XY&Z] [Corporation] [-]  SimpleAnalyzer – [xy] [z] [corporation] [xyz] [example] [com]  StopAnalyzer – [xy] [z] [corporation] [xyz] [example] [com]  StandardAnalyzer – [xy&z] [corporation]

Searching  IndexReader and IndexSearcher 12 int k = 10; StandardAnalyzer analyzer = new StandardAnalyzer(); IndexReader reader = DirectoryReader.open(FSDirectory.open(new File( ".//index"))); IndexSearcher searcher = new IndexSearcher(reader); Query q = new QueryParser("Description", analyzer).parse(query); TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoreDocs; int k = 10; StandardAnalyzer analyzer = new StandardAnalyzer(); IndexReader reader = DirectoryReader.open(FSDirectory.open(new File( ".//index"))); IndexSearcher searcher = new IndexSearcher(reader); Query q = new QueryParser("Description", analyzer).parse(query); TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoreDocs;

Query and QueryParser QueryParser parser = new QueryParser(Version.LUCENE_40, ”LastName”, new StandardAnalyzer()); Query query = parser.parse(q); QueryParser parser = new QueryParser(Version.LUCENE_40, ”LastName”, new StandardAnalyzer()); Query query = parser.parse(q);  QueryParser – Need to parse the query in the same way the documents were indexed – Tell the query which field should it use (field based search) – Use the same analyzer

TopDocs and ScoreDoc TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoreDocs; TopDocs docs = searcher.search(q, k); ScoreDoc[] hits = docs.scoreDocs;  Search returns TopDocs – Reference to the top ranked documents returned by search  TopDoc has ScoreDoc(s) – Each ScoreDoc is a single document

Getting the results System.out.println("Found " + hits.length + " hits."); for(int i=0;i<hits.length;++i) { int docId = hits[i].doc; Document d = searcher.doc(docId); System.out.println((i + 1) + ". " + d.get("First_Name") + " " + d.get("Last_Name")); } System.out.println("Found " + hits.length + " hits."); for(int i=0;i<hits.length;++i) { int docId = hits[i].doc; Document d = searcher.doc(docId); System.out.println((i + 1) + ". " + d.get("First_Name") + " " + d.get("Last_Name")); }  Get the required fields from the documents

Adding/deleting Document s void addDocument(Document d); void addDocument(Document d, Analyzer a); Important: Need to ensure that Analyzer s used at indexing time are consistent with Analyzer s used at searching time // deletes docs containing term or matching // query. The term version is useful for // deleting one document. void deleteDocuments(Term term); void deleteDocuments(Query query);

Index format  Each Lucene index consists of one or more segments – A segment is a standalone index for a subset of documents – All segments are searched – A segment is created whenever IndexWriter flushes adds/deletes  Periodically, IndexWriter will merge a set of segments into a single segment – Policy specified by a MergePolicy – Segments are grouped into levels – Segments within a group are roughly equal size (in log space) – Once a level has enough segments, they are merged into a segment at the next level up  Explicitly invoke optimize() to merge segments

Searching a changing index Directory dir = FSDirectory.open(...); IndexReader reader = IndexReader.open(dir); IndexSearcher searcher = new IndexSearcher(reader); Above reader does not reflect changes to the index unless you reopen it. Reopen ing is more resource efficient than open ing a new IndexReader. IndexReader newReader = reader.reopen(); If (reader != newReader) { reader.close(); reader = newReader; searcher = new IndexSearcher(reader); }

Near-real-time search IndexWriter writer =...; IndexReader reader = writer.getReader(); IndexSearcher searcher = new IndexSearcher(reader); // Now let us say there’s a change to the index using writer writer.addDocument(newDoc); // reopen() and getReader() force writer to flush IndexReader newReader = reader.reopen(); if (reader != newReader) { reader.close(); reader = newReader; searcher = new IndexSearcher(reader); }

Query Syntax Query expressionDocument matches if… javaContains the term java in the default field java junit java OR junit Contains the term java or junit or both in the default field (the default operator can be changed to AND) +java +junit java AND junit Contains both java and junit in the default field title:antContains the term ant in the title field title:extreme –subject:sportsContains extreme in the title and not sports in subject (agile OR extreme) AND javaBoolean expression matches title:”junit in action”Phrase matches in title title:”junit action”~5Proximity matches (within 5) in title java*Wildcard matches java~Fuzzy matches lastmodified:[1/1/09 TO 12/31/09] Range matches

Programmatically constructed queries  TermQuery, TermRangeQuery  NumericRangeQuery  PrefixQuery  BooleanQuery  PhraseQuery, WildcardQuery

Source and acknowledgements  Slides by Manning and Nayak: lucene.pptx lucene.pptx  The Lucene tutorial website:  Apache lucene: 22