Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar.

Slides:



Advertisements
Similar presentations
Lecture 4: Index Construction
Advertisements

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan.
Information Retrieval in Practice
Index Construction David Kauchak cs160 Fall 2009 adapted from:
Algoritmi per IR Prologo. References Managing gigabytes A. Moffat, T. Bell e I. Witten, Kaufmann Publisher A bunch of scientific papers available.
Hinrich Schütze and Christina Lioma Lecture 4: Index Construction
PrasadL07IndexCompression1 Index Compression Adapted from Lectures by Prabhakar Raghavan (Yahoo, Stanford) and Christopher Manning.
PrasadL06IndexConstruction1 Index Construction Adapted from Lectures by Prabhakar Raghavan (Google) and Christopher Manning (Stanford)
Indexes. Primary Indexes Dense Indexes Pointer to every record of a sequential file, (ordered by search key). Can make sense because records may be much.
FALL 2006CENG 351 Data Management and File Structures1 External Sorting.
Information Retrieval IR 4. Plan This time: Index construction.
IR Paolo Ferragina Dipartimento di Informatica Università di Pisa.
1 INF 2914 Information Retrieval and Web Search Lecture 6: Index Construction These slides are adapted from Stanford’s class CS276 / LING 286 Information.
Index Construction: sorting Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading Chap 4.
CpSc 881: Information Retrieval. 2 Hardware basics Many design decisions in information retrieval are based on hardware constraints. We begin by reviewing.
Hinrich Schütze and Christina Lioma Lecture 4: Index Construction
Lecture 4 Index construction
INF 2914 Information Retrieval and Web Search
Indexing Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata.
CSCI 5417 Information Retrieval Systems Jim Martin Lecture 4 9/1/2011.
Algorithms for Information Retrieval Is algorithmic design a 5-mins thinking task ???
Introduction to Information Retrieval Information Retrieval and Data Mining (AT71.07) Comp. Sc. and Inf. Mgmt. Asian Institute of Technology Instructor:
Index Construction David Kauchak cs458 Fall 2012 adapted from:
Document Indexing: SPIMI
Dictionaries and Tolerant retrieval
Binary Merge-Sort Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Merge-Sort(A,i,j)
Inverted index, Compressing inverted index And Computing score in complete search system Chintan Mistry Mrugank dalal.
An Introduction to IR Lecture 4 Index construction 1.
Index Construction 1 Lecture 5: Index Construction Web Search and Mining.
Index Compression David Kauchak cs458 Fall 2012 adapted from:
Index Construction (Modified from Stanford CS276 Class Lecture 4 Index construction)
1 ITCS 6265 Lecture 4 Index construction. 2 Plan Last lecture: Dictionary data structures Tolerant retrieval Wildcards Spell correction Soundex This lecture:
Information Retrieval Techniques MS(CS) Lecture 6 AIR UNIVERSITY MULTAN CAMPUS Most of the slides adapted from IIR book.
Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan.
CPSC 404, Laks V.S. Lakshmanan1 External Sorting Chapter 13: Ramakrishnan & Gherke and Chapter 2.3: Garcia-Molina et al.
PrasadL06IndexConstruction1 Index Construction Adapted from Lectures by Prabhakar Raghavan (Yahoo and Stanford) and Christopher Manning (Stanford)
Index Construction: sorting Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading Chap 4.
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 4: Index Construction Related to Chapter 4:
Chapter 5 Ranking with Indexes 1. 2 More Indexing Techniques n Indexing techniques:  Inverted files - best choice for most applications  Suffix trees.
Evidence from Content INST 734 Module 2 Doug Oard.
Introduction to Information Retrieval COMP4210: Information Retrieval and Search Engines Lecture 4: Index Construction United International College.
Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Christopher Manning and Prabhakar.
Lecture 4: Index Construction Related to Chapter 4:
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 4: Index Construction Related to Chapter 4:
Web Information Retrieval Textbook by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze Notes Revised by X. Meng for SEU May 2014.
Introduction to Information Retrieval Introduction to Information Retrieval Modified from Stanford CS276 slides Chap. 4: Index Construction.
Introduction to Information Retrieval CSE 538 MRS BOOK – CHAPTER IV Index Construction 1.
Introduction to Information Retrieval Introduction to Information Retrieval Introducing Information Retrieval and Web Search.
CS276 Lecture 4 Index construction. Plan Last lecture: Tolerant retrieval Wildcards Spell correction Soundex This time: Index construction.
Introduction to COMP9319: Web Data Compression and Search Search, index construction and compression Slides modified from Hinrich Schütze and Christina.
CS315 Introduction to Information Retrieval Boolean Search 1.
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 7: Index Construction.
Lecture 3 – MapReduce: Implementation CSE 490h – Introduction to Distributed Computing, Spring 2009 Except as otherwise noted, the content of this presentation.
Information Retrieval and Data Mining (AT71. 07) Comp. Sc. and Inf
Index Construction Some of these slides are based on Stanford IR Course slides at
Prof. Paolo Ferragina, Algoritmi per "Information Retrieval"
Chapter 4 Index construction
7CCSMWAL Algorithmic Issues in the WWW
Modified from Stanford CS276 slides Lecture 4: Index Construction
Lecture 7: Index Construction
CS276: Information Retrieval and Web Search
Index Construction: sorting
Lecture 7: Index Construction
3-2. Index construction Most slides were adapted from Stanford CS 276 course and University of Munich IR course.
Lecture 4: Index Construction
Index construction 4장.
CS276: Information Retrieval and Web Search
INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID
Presentation transcript:

Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture: Index construction

Introduction to Information Retrieval Index construction  How do we construct an index?  What strategies can we use with limited main memory?  What about dynamic text collections? Ch. 4

Introduction to Information Retrieval Recall hardware basics  Disk I/O is block-based: Reading and writing of entire blocks (as opposed to smaller chunks).  Block sizes: 8KB to 256 KB.  Servers used in IR systems now typically have several GB of main memory, sometimes tens of GB.  Available disk space is several (2–3) orders of magnitude larger.  Fault tolerance is very expensive: It’s much cheaper to use many regular machines rather than one fault tolerant machine. Sec. 4.1

Introduction to Information Retrieval

 Documents are parsed to extract words and these are saved with the Document ID. I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Doc 1 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Doc 2 Recall: index construction Sec. 4.2

Introduction to Information Retrieval Key step  After all documents have been parsed, the inverted file is sorted by terms. We focus on this sort step. Sec. 4.2

Introduction to Information Retrieval Scaling index construction  In-memory index construction does not scale.  How can we construct an index for very large collections?  Taking into account...memory, disk, speed, etc. Sec. 4.2

Introduction to Information Retrieval Sort-based index construction  As we build the index, we parse docs one at a time.  While building the index, we cannot easily exploit compression tricks (you can, but much more complex)  The final postings are incomplete until the end.  At 12 bytes per non-positional postings entry (term, doc, freq), demands a lot of space for large collections.  T = 100,000,000 in the case of Reuters  So … we can do this in memory in 2009, but typical collections are much larger. E.g. the New York Times provides an index of >150 years of newswire  We need to store intermediate results on disk. Sec. 4.2

Introduction to Information Retrieval Use the same algorithm for disk?  multi-way merge-sort aka BSBI: Blocked sort-based Indexing Sec. 4.2

Introduction to Information Retrieval Multi-way Mergesort  Take Wikipedia in Italian, and compute word freq:  Few GBs  n  10 9 words  How do you proceed ??  Tokenize into a sequence of strings  Sort the strings  Create tuples

Introduction to Information Retrieval Binary Merge-Sort Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Divide Conquer Combine Merge is linear in the #items to be merged

Introduction to Information Retrieval Few key observations  Items = (short) strings = atomic...   (n log n) memory accesses (I/Os ??)  [5ms] * n log 2 n ≈ 3 years In practice it is a “faster”, why?

Introduction to Information Retrieval Implicit Caching… log 2 N M N/M runs, each sorted in internal memory (no I/Os) 2 passes (one Read/one Write) = 2 * (N/B) I/Os — I/O-cost for binary merge-sort is ≈ 2 (N/B) log 2 (N/M) Log 2 (N/M) 2 passes (R/W)

Introduction to Information Retrieval B A key inefficiency B After few steps, every run is longer than B !!! B We are using only 3 pages But memory contains M/B pages ≈ 2 30 /2 15 = 2 15 B Output Buffer Disk 1, 2, 3 Output Run 4,...

Introduction to Information Retrieval Multi-way Merge-Sort  Sort N items with main-memory M and disk-pages B:  Pass 1: Produce (N/M) sorted runs.  Pass i: merge X = M/B-1 runs  log X N/M passes Main memory buffers of B items Pg for run1 Pg for run X Out Pg Disk Pg for run 2...

Introduction to Information Retrieval Cost of Multi-way Merge-Sort  Number of passes = log X N/M  log M/B (N/M)  Total I/O-cost is  ( (N/B) log M/B N/M ) I/Os Large fan-out (M/B) decreases #passes In practice M/B ≈ 10 5  #passes = 1  few mins Tuning depends on disk features Compression would decrease the cost of a pass!

Introduction to Information Retrieval Keep attention...  If sorting needs to manage arbitrarily long strings Key observations:  Array A is an “array of pointers to objects”  For each object-to-object comparison A[i] vs A[j]:  2 random accesses to 2 memory locations A[i] and A[j]   (n log n) random memory accesses (I/Os ??) Memory containing the strings A Again chaching helps, But it may be less effective than before Indirect sort

Introduction to Information Retrieval SPIMI: Single-pass in-memory indexing  Key idea #1: Generate separate dictionaries for each block.  Key idea #2: Don’t sort. Accumulate postings in postings lists as they occur.  With these two ideas we can generate a complete inverted index for each block.  These separate indexes can then be merged into one big index.  Easy append with 1 file per posting (docID are increasing among blocks), otherwise emulate multi-way merging Sec. 4.3

Introduction to Information Retrieval SPIMI-Invert Merging of blocks  BSBI (now dictionary-driven). Sec. 4.3

Introduction to Information Retrieval SPIMI: Compression  Compression makes SPIMI even more efficient.  Compression of terms  Compression of postings  See next lectures Sec. 4.3

Introduction to Information Retrieval Distributed indexing  For web-scale indexing: must use a distributed computing cluster  Individual machines are fault-prone  Can unpredictably slow down or fail  How do we exploit such a pool of machines? Sec. 4.4

Introduction to Information Retrieval Google data centers  Google data centers mainly contain commodity machines.  Data centers are distributed around the world.  Estimate: Google installs 100,000 servers each quarter.  Based on expenditures of 200–250 million dollars per year  This is 10% of the computing capacity of the world !?! Sec. 4.4

Introduction to Information Retrieval Distributed indexing  Maintain a master machine directing the indexing job – considered “safe”.  Break up indexing into sets of (parallel) tasks.  Master machine assigns tasks to idle machines Sec. 4.4

Introduction to Information Retrieval Parallel tasks  We will use two sets of parallel tasks  Parsers  Inverters  Break the input document collection into splits  Each split is a subset of documents Sec. 4.4 Term-based partition one machine handles a subrange of terms Doc-based partition one machine handles a subrange of documents Term-based partition one machine handles a subrange of terms Doc-based partition one machine handles a subrange of documents

Introduction to Information Retrieval Data flow: doc-based partitioning splits Parser Master Inverter Postings IL 1 assign Sec. 4.4 IL 2 IL k Set 1 Set 2 Set k Query must be also distributed

Introduction to Information Retrieval Data flow: term-based partitioning splits Parser Master a-fg-pq-z a-fg-pq-z a-fg-pq-z Inverter Postings a-f g-p q-z assign Sec. 4.4 Set 1 Set 2 Set k Each query-term goes to one machine

Introduction to Information Retrieval Parsers  Master assigns a split to an idle parser machine  Parser reads a document at a time and emits (term, doc) pairs  Parser writes pairs into j partitions  Each partition is for a range of terms’ first letters (e.g., a-f, g-p, q-z) – here j = 3. Sec. 4.4

Introduction to Information Retrieval Inverters  An inverter collects all (term,doc) pairs (= postings) for one term-partition.  Sorts and writes to postings lists Sec. 4.4

Introduction to Information Retrieval MapReduce  Index construction is an instance of MapReduce  a robust and conceptually simple framework for distributed computing  … without having to write code for the distribution part.  Google indexing system (ca. 2002) consists of a number of phases, each implemented in MapReduce. Sec. 4.4

Introduction to Information Retrieval Data flow: term-based partitioning splits Parser Master a-fg-pq-z a-fg-pq-z a-fg-pq-z Inverter Postings a-f g-p q-z assign Map phase Reduce phase Sec. 4.4

Introduction to Information Retrieval Dynamic indexing  Up to now, we have assumed static collections.  It was rare but now more frequently occurs that:  Documents come in over time and need to be inserted.  Documents are deleted and modified.  And this induces:  Postings updates for terms already in dictionary  New terms added/deleted to/from dictionary Sec. 4.5

Introduction to Information Retrieval Simplest approach  Maintain “big” main index  New docs go into “small” auxiliary index  Search across both, and merge the results  Deletions  Invalidation bit-vector for deleted docs  Filter search results (i.e. docs) by the invalidation bit-vector  Periodically, re-index into one main index Sec. 4.5

Introduction to Information Retrieval Issues with main and auxiliary indexes  Merges may touch a lot of stuff  Poor performance  Merging of the auxiliary index into the main index is efficient if we keep a separate file for each postings list [docIDs are greater].  Merge is the same as a simple append.  But this needs a lot of files – inefficient for O/S.  Assumption: The index is one big file.  In reality: Use a scheme somewhere in between (e.g., split very large postings lists, collect postings lists of length 1 in one file etc.) Sec. 4.5

Introduction to Information Retrieval Logarithmic merge  Maintain a series of indexes, each twice as large as the previous one.  Keep smallest (Z 0 ) in memory  Larger ones (I 0, I 1, …) on disk  If Z 0 gets too big (> n), write to disk as I 0 or merge with I 0 (if I 0 already exists) as Z 1  Either write merge Z 1 to disk as I 1 (if no I 1 ) or merge with I 1 to form Z 2  etc. Sec. 4.5

Introduction to Information Retrieval Logarithmic merge  Auxiliary and main index: index construction time is O(T 2 ) as each posting is touched in each merge.  Logarithmic merge: Each posting is merged O(log T) times, so complexity is O(T log T) So logarithmic merge is much more efficient for index construction, but its query processing requires the merging of O(log T) indexes Sec. 4.5

Introduction to Information Retrieval Dynamic indexing at search engines  Most search engines now support dynamic indexing  News items, blogs, new topical web pages  But (sometimes/typically) they also periodically reconstruct the index from scratch  Query processing is then switched to the new index, and the old index is then deleted Sec. 4.5

Introduction to Information Retrieval Other sorts of indexes  Positional indexes  Same type of sorting problem … just larger  Character n-gram indexes:  For each n-gram, need pointers to all dictionary terms containing it – the “postings”.  Permuterm index:  Consider all cyclic rotations of T$, where T is any term Sec. 4.5