1. Open Source IR Tools and Libraries
CS-463 Information Retrieval Models
Computer Science Department
University of Crete

2. Outline
Google Search API
Lucene
Terrier
Lemur
Dragon
Groogle

3. Google Search API

4. Google Search API: Overview
The API exposes the Google engine to developers.
You can write scripts that access the Google search in real-time.
Google no longer issuing new API keys for the SOAP Search API.
Instead, Google provides an AJAX Search API.
You can put Google Search in your web pages with JavaScript.

5. Google Search API: SOAP
Based on the Web Services Technology SOAP (the XML-based Simple Object Access Protocol).
Developers write software programs that connect remotely to the Google SOAP Search API service.
Developers can issue search requests to Google’s index of billions of web pages and receive results as structured data, access information in the Google cache and check the spelling of words.
Limitations
Default limit of 1,000 queries per day.
Can only query for 10 results a time
Can only access Google Web Search (not Google Images, Google Groups and so on).

6. Google Search API: AJAX
Lets you put Google Search in your web pages with JavaScript.
Does not have a limit on the number of queries per day.
Supports additional features like Video, News, Maps, and Blog search results.

7. Google Search API: AJAX
Web Search
Incorporate results from Web Search, News Search, and Blog Search
Local Search
Provides access to local search results from Google Maps.
Video Search
Incorporate a simple search box
incorporate dynamic, search powered strips of video and book thumbnails.

8. Google Search API: Demo

9. Google’s Solutions URL Queue/List Cached source pages (compressed)
Use many
features,
e.g. font,
layout,…
Inverted index
Hypertext
structure

10. Google Search API: References
Google SOAP Search API
Google AJAX Search API
Google AJAX Search API Developer Guide
Google AJAX Search API Samples

11. Apache Software Foundation
Lucene
Apache Software Foundation

12. Lucene Cross-Platform API Implemented in Java
Ported in C++, C#, Perl, Python
Offers scalable, high-performance indexing
Incremental indexing as fast as bath indexing
Index size roughly 20-30% the size of indexed text
Supports many powerful query types

13. Lucene: Modules Analysis Document Index Query Parser
Tokenization, Stop words, Stemming, etc.
Document
Unique ID for each document
Title of document, date modified, content, etc.
Index
Provides access and maintains indexes.
Query Parser
Search / Search Spans

14. Lucene: Indexing A Document is a collection of Fields
A Field is free text, keywords, dates, etc.
A Field can have several characteristics
indexed, tokenized, stored, term vectors
Apply Analyzer to alter Tokens during indexing
Stemming
Stop-word removal
Phrase identification
Document:
Field 1
Field 2
Field N

15. Lucene: Query Parser Syntax
Terms
Single terms and phrases
Fields
E.g. title:"Do it right" AND right
Wildcard Searches
‘?’ for single character
‘*’ for multiple characters
Proximity Searches
“jakarta apache"~10
Fuzzy Searches
Levenshtein Distance or Edit Distance algorithm
Range Searches
mod_date:[ TO ]
title:{Aida TO Carmen}
Boosting a Term
E.g. jakarta^4 apache
Boolean Operators
Fuzzy Searches
Lucene supports fuzzy searches based on the Levenshtein Distance, or Edit Distance algorithm. To do a fuzzy search use the tilde, "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to "roam" use the fuzzy search:
roam~ This search will find terms like foam and roams.
Starting with Lucene 1.9 an additional (optional) parameter can specify the required similarity. The value is between 0 and 1, with a value closer to 1 only terms with a higher similarity will be matched. For example:
roam~0.8 The default that is used if the parameter is not given is 0.5.
Proximity Searches
Lucene supports finding words are a within a specific distance away. To do a proximity search use the tilde, "~", symbol at the end of a Phrase. For example to search for a "apache" and "jakarta" within 10 words of each other in a document use the search:
"jakarta apache"~10
Boosting a Term
Lucene provides the relevance level of matching documents based on the terms found. To boost a term use the caret, "^", symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be.
Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for
jakarta apache and you want the term "jakarta" to be more relevant boost it using the ^ symbol along with the boost factor next to the term. You would type:
jakarta^4 apache This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example:
"jakarta apache"^4 "Apache Lucene" By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2)

16. Lucene: More Advanced Options
Relevance Feedback
Manual
User selects which documents are relevant/non-relevant
Get the terms from the term vector for each of the documents and construct a new query.
Automatic
Application assumes the top X documents are relevant and the bottom Y are non-relevant and constructs a new query based on the terms in those documents.
Span Queries
Phrase Matching
Relevance feedback is a technique used to augment the user’s query with terms from the best matching documents. Grant shows how to use Lucene’s Term Vectors to do this. Relevance feedback has been on my todo-list for a few weeks now, it’s great to have a good starting point.
Span queries provide information about where a match took place within a document. The presentation explains how to use them for phrase matching (I wonder if that could replace named entity detection completely) and how to further use them for question answering. Looks like you could also use span queries as the basis for an efficient result summarizer.
Spans object provides document and position info about the current match.
Phrase Matching uses position distance instead of edit distance. SpanNearQuery class provides functionality similar to PhraseQuery class.

17. Lucene: Basic Demo
The latest version can be obtained from
To build an index just type
java org.apache.lucene.demo.IndexFiles <dir>
To search from an index type:
java org.apache.lucene.demo.SearchFiles <index>

18. Terrier
University Of Glasgow

19. Terrier: Overview (1/2) Stands for TERabyte RetrIEveR.
Open Source API (Mozilla Public Licence).
Modular platform for the rapid development of large-scale IR applications.
It is written in Java (and Perl)
Highly compressed disk data structures.
Handling large-scale document collections.
Standard evaluation of TREC ad-hoc and known-item search retrieval results.
Based on a new parameter-free probabilistic framework for IR (DFR), allowing adaptable term weighting functionalities.

20. Terrier: Overview (2/2)
Includes state-of-the-art functionalities such as:
hyperlink structure analysis,
combination of evidence approaches,
automatic query expansion/re-formulation techniques,
query performance predictors
compression techniques.
Deploys over 50 term weighting/ matching functions.
Has a robust and effective crawler, called Labrador.
Allows a large-scale experimentation conducted in a robust, transparent, reproducible, modular, platform independent, and without constraints and parameters.

21. Terrier: Indexing Direct Index Document Index Lexicon
Create your own Collection decoder and Document implementation.
Centralized or distributed Setting.
Indexer iterates through the collection and creates the following data structures
Direct Index
Document Index
Lexicon

22. Terrier: Indexing
The inverted index is built from the existing direct index, document index and lexicon
In this way, we build the direct and document indices.
Each document in the collection is tokenized and parsed.
We also build temporary lexicons in order to reduce the required memory during indexing

23. Terrier: Retrieval Parsing Pre-processing Matching Post Processing
Post Filtering
Query Language
term1 term2
term1^2.3
+term1 -term2
"term1 term2"~n
The aim of query expansion is to reduce this query/document mismatch by expanding the query using words or phrases with a similar meaning or some other statistical relation to the set of relevant documents. This procedure may have even greater importance in spoken document retrieval, since the word mismatch problem is heightened by the presence of errors in the automatic transcription of spoken documents.

24. Terrier: Retrieval
Terrier automatically select the optimal document weighting model
If Query Expansion is applied an appropriate term weighting model is selected and the most informative terms from the top ranked documents are added to the query
The aim of query expansion is to reduce this query/document mismatch by expanding the query using words or phrases with a similar meaning or some other statistical relation to the set of relevant documents. This procedure may have even greater importance in spoken document retrieval, since the word mismatch problem is heightened by the presence of errors in the automatic transcription of spoken documents.
Remove stop words and apply stemming to the query.

25. Terrier: Sample Applications
Trec Terrier
An application that allows Terrier to index and retrieve from standard TREC test collections.
Instructions are available at

26. Terrier: Sample Applications
Desktop Search
A Swing (graphical) application that can be used to index files from the local machine, and then perform queries on them.
The scripts for running the desktop search application are:
desktop_search.sh (Linux, Mac OSX)
desktop_search.bat (Windows)

27. Terrier: Sample Applications
Interactive Querying
A console application for performing simple queries on an existing index and seeing which documents are returned.
The scripts for running the console application are:
interactive_terrier.sh (Linux, Mac OS X)
interactive_terrier.bat (Windows)

28. Terrier: Demo

29. University of Massachusetts
Lemur
University of Massachusetts

30. Lemur: Overview Support for XML and structured document retrieval
Interactive interfaces for Windows, Linux, and Web
Cross-Platform, fast and modular code written in C++
Free and open-source software

31. Lemur: API
Provides interfaces to Lemur classes that are grouped at three different levels:
Utility level
Common utilities, such as memory management, document parsing, etc.
Indexer level
Converts a raw text collection to data structures for efficient retrieval.
Retrieval level
Abstract classes for a general retrieval architecture and concrete classes for several specific information retrieval

32. Lemur: Indexing
Multiple indexing methods for small, medium and large-scale (terabyte) collections.
Built-in support for English, Chinese and Arabic text.
Porter and Krovetz word stemming.
Incremental indexing.

33. Lemur: Retrieval
Supports major language modelling approaches such as Indri and KL-divergence, as well as vector space, tf-idf, Ocapi and InQuery
Relevance- and pseudo-relevance feedback
Wildcard term expansion (using Indri)
Supports arbitrary document priors (e.g., Page Rank, URL depth)

34. Sorted Vector of Score Extent Results
Lemur: Query Flow
Query Parser
User Query
Scoring Nodes
runQuery()
Sorted Vector of Score Extent Results

35. The Dragon Toolkit
University Of Drexel

36. Dragon: Overview (1/2) Highly scalable to large data set
Well designed Programming API and XML-based Interface
Various document representations including words, multiword phrases, ontology-based concepts, and concept pairs
Various text retrieval models
Text classification, clustering, summarization and topic modeling

37. Dragon: Overview (2/2)
Provides built-in supports for semantic-based IR and TM (different from Lucene and Lemur ).
Integrates a set of NLP tools, which enable the toolkit to index text collections with various representation schemes including words, phrases, ontology-based concepts and relationships.
It is specially designed for large-scale application.
The toolkit uses sparse matrix to implement text representations and does not have to load all data into memory in the running time.
Can handle hundred thousands of documents with very limited memory.

38. Dragon Toolkit Demo

39. Groogle
University of Crete

40. Groogle Wiki Repository Bugzilla Credits
Repository
Bugzilla
Credits

41. Questions ?