Discussion 5 Sara Javanmardi

Assignment 3 Demo Friday Feb 4th: *8am-10:00 am *11am-1pm *2pm-4pm *In my office

Microsoft Spell Checker Contest Speller Contest Dataset

Assignment 4 Indexing Enron Emails

How to Download the compressed file Unzip it

Part1 General Questions

Part2 Quantifying the Data Listing the Files or Subdirectories in a Directory

Part3 Index the data

Posting List To create the posting lists, you have 4 options 1) [term : docID \t]+ 2) [term : docID:termFrequency \t]+ 3 )[term : docID: position of the term in the documment \t]+ 4 )[term : docID:Frequency, position of the term in the documment \t]+

Example Abandon\tdoc1:4:3,301,400,700\tdoc3:102,105\n Bail\tdoc2:1:21\tdoc3:2:100,1012\n . Sorted based on Doc IDs Alphabetically sorted

Index construction How do we construct an index? Ch. 4 Index construction How do we construct an index? What strategies can we use with limited main memory?

The basic steps to construct your index: 1)    Make a pass through the collection assembling term-docID pairs. 2)    To make index construction more efficient, we present terms as termIDs, where a termID is a unique serial number. We can do it in 2 ways: a.    On the fly while we are processing the collection b.    We can compile vocabulary in the first pass and construct the inverted index in the second pass. 3)    Sort the pairs with the terms 4)    Finally, we organize the docIDs for each term into a postings list.

Sec. 4.2 index construction Documents are parsed to extract words and these are saved with the Document ID. Doc 1 Doc 2 I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious

Sec. 4.2 Key step After all documents have been parsed, the inverted file is sorted by terms. We focus on this sort step. We have 100M items to sort.

Problems? How to update? See InvertedIndex.java