1. Modified by Dongwon Lee from slides by
Ch 1 Boolean Retrieval
Modified by Dongwon Lee from slides by
Christopher Manning and Prabhakar Raghavan

2. In the Last Class Ch 19 Ch 21 Index size estimation
Near-duplicate detection using shingling
Ch 21
PageRank
Hubs and Authorities

3. Next 2 Classes Ch 1 Ch 2 IR Basics
Index: Incidence Matrix, Inverted Index
Boolean vs. ranked retrieval models
Ch 2
Processing terms of documents
Improving inverted index
Supporting keyword vs. phrase queries

4. Information Retrieval
Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).

5. IR vs. databases: Structured vs unstructured data
Structured data tends to refer to information in “tables”
Employee
Manager
Salary
Smith
Jones
50000
Chang
Smith
60000
Ivy
Smith
50000
Typically allows numerical range and exact match
(for text) queries, e.g.,
Salary < AND Manager = Smith.

6. Unstructured data Typically refers to free text Allows
Keyword queries including operators
More sophisticated “concept” queries e.g.,
find all web pages dealing with drug abuse
Classic model for searching text documents

7. Unstructured (text) vs. structured (database) data in 1996

8. Unstructured (text) vs. structured (database) data in 2009

9. Sec. 1.1
Unstructured data in 1680
Which plays of Shakespeare contain the words Brutus AND Caesar but NOT Calpurnia?
One could grep all of Shakespeare’s plays for Brutus and Caesar, then strip out lines containing Calpurnia?
Why is that not the answer?
Slow (for large corpora)
NOT Calpurnia is non-trivial
Other operations (e.g., find the word Romans near countrymen) not feasible
Ranked retrieval (best documents to return)
Grep is line-oriented; IR is document oriented.

10. Term-document incidence
Sec. 1.1
Term-document incidence
1 if play contains word, 0 otherwise
Brutus AND Caesar BUT NOT Calpurnia

11. Incidence vectors So we have a 0/1 vector for each term.
Sec. 1.1
Incidence vectors
So we have a 0/1 vector for each term.
To answer query: take the vectors for Brutus, Caesar and Calpurnia (complemented)  bitwise AND.
AND AND =

12. Answers to query Antony and Cleopatra, Act III, Scene ii
Sec. 1.1
Answers to query
Antony and Cleopatra, Act III, Scene ii
Agrippa [Aside to DOMITIUS ENOBARBUS]: Why, Enobarbus,
When Antony found Julius Caesar dead,
He cried almost to roaring; and he wept
When at Philippi he found Brutus slain.
Hamlet, Act III, Scene ii
Lord Polonius: I did enact Julius Caesar I was killed i' the
Capitol; Brutus killed me.

13. Basic assumptions of Information Retrieval
Sec. 1.1
Basic assumptions of Information Retrieval
Collection: Fixed set of documents
Goal: Retrieve documents with information that is relevant to the user’s information need and helps the user complete a task

14. The classic search model
TASK
Get rid of mice in a politically correct way
Misconception?
Info Need
Info about removing mice
without killing them
Mistranslation?
Verbal form
How do I trap mice alive?
Misformulation?
Query
mouse trap
SEARCH ENGINE
Query Refinement
Results
Corpus

15. Can’t build the incidence matrix
Sec. 1.1
Can’t build the incidence matrix
Consider
1 M documents
500K unique terms in all documents
500K x 1M matrix has half-a-trillion 0’s and 1’s.
But the matrix is extremely sparse
What’s a better representation?
We only record the 1 positions.

16. Sec. 1.2
Inverted index
For each term t, we must store a list of all documents that contain t.
Identify each by a docID, a document serial number
Can we used fixed-size arrays for this?
Brutus 1 2 4 11 31 45
173
174
Caesar 1 2 4 5 6 16 57
132
Calpurnia 2 31 54
101
What happens if the word Caesar is added to document 14?

17. Inverted index We need variable-size postings lists
Sec. 1.2
Inverted index
We need variable-size postings lists
On disk, a continuous run of postings is normal and best
In memory, can use linked lists or variable length arrays
Some tradeoffs in size/ease of insertion
Posting
Brutus 1 2 4 11 31 45
173
174
Dictionary
Caesar 1 2 4 5 6 16 57
132
Linked lists generally preferred to arrays
Dynamic space allocation
Insertion of terms into documents easy
Space overhead of pointers
Calpurnia 2 31 54
101
Postings
Sorted by docID (more later on why).

18. Inverted index construction
Sec. 1.2
Inverted index construction
Documents to
be indexed.
Friends, Romans, countrymen.
Tokenizer
Token stream.
Friends
Romans
Countrymen
More on
these later.
Linguistic modules
Modified tokens.
friend
roman
countryman
Indexer
Inverted index.
friend
roman
countryman 2 4 13 16 1

19. Indexer steps: Token sequence
Sec. 1.2
Indexer steps: Token sequence
Sequence of (Modified token, Document ID) pairs.
Doc 1
Doc 2
I did enact Julius
Caesar I was killed
i' the Capitol;
Brutus killed me.
So let it be with
Caesar. The noble
Brutus hath told you
Caesar was ambitious

20. Indexer steps: Sort Sort by terms Core indexing step And then docID
Sec. 1.2
Indexer steps: Sort
Sort by terms
And then docID
Core indexing step

21. Indexer steps: Dictionary & Postings
Sec. 1.2
Indexer steps: Dictionary & Postings
Multiple term entries in a single document are merged.
Split into Dictionary and Postings
Doc. frequency information is added.

22. Where do we pay in storage?
Sec. 1.2
Where do we pay in storage?
Lists of docIDs
Terms and counts
Pointers

23. Query processing: AND Consider processing the query: Brutus AND Caesar
Sec. 1.3
Query processing: AND
Consider processing the query:
Brutus AND Caesar
Locate Brutus in the Dictionary;
Retrieve its postings.
Locate Caesar in the Dictionary;
“Merge” the two postings: 2 4 8 16 32 64
128
Brutus 1 2 3 5 8 13 21 34
Caesar

24. Sec. 1.3
The merge
Walk through the two postings simultaneously, in time linear in the total number of postings entries 34
128 2 4 8 16 32 64 1 3 5 13 21 2 4 8 16 32 64
128
Brutus
Caesar 2 8 1 2 3 5 8 13 21 34
If the list lengths are x and y, the merge takes O(x+y)
operations.
Crucial: postings sorted by docID.
What is this?

25. Intersecting two postings lists (a “merge” algorithm)
The time complexity
of the “merge” algorithm:
O(x+y) = O(N)

26. Sidetrack: What is Algorithm?
Algorithm: a sequence of finite instructions for doing some task
unambiguous and simple to follow
Eg, Euclid’s algorithm to determine the greatest common divisor (gcd) of two integers greater than 1
Divide the larger number (A) by the smaller number (B) to get the remainder A’
Divide the larger number (B) by the smaller number (A’) to get the remainder B’ …
Divide the larger number (A’) by the smaller number (B’) to get the remainder A’’ …
Repeat until you get 0 or 1: Then, gcd is the last divider

27. Sidetrack: Eg, Optimal Wedding
When a person has a chance to date K persons, the optimal wedding algorithm is:
Date upto (K/3)-th persons
Let the best person among K/3 as B using a criteria C
Start dating again from (K/3 + 1)-th person, p
If p is better than B using C
Stop and Marry p
Otherwise, keep dating till K-th person

28. Sidetrack: How good is an Algorithm?
Measure the rate of growth of an algorithm or problem based upon the size of the input
Compare algorithms to determine which is better for the situation
Input N
Describing the relationship as a function f(N)
f(N): the number of steps required by an algorithm

29. Sidetrack: Why Input Size N?
What are other factors that affect computation time?
Speed of the CPU
Size of memory
Choice of programming languages
Size of the input …
Since an algorithm can be implemented in different machines, by different programming languages, and run on different platforms, we are interested in comparing algorithms using factors that are not affected by implementations

30. Sidetrack: Growth of different f(N)
Work required to finish
log N 1
Size of input N

31. Sidetrack: Informal Definition of Big-O
Q: How long does it take from State College to Washington DC if you drive?
About 3 hours
More than 60 minutes
Less than 10 days
Lower Bound
Upper Bound

32. Sidetrack: Formal Definition of Big-O
For a given function g(n), O(g(n)) is defined to be the set of functions
O(g(n)) = {f(n) : there exist positive constants c
and n0 such that 0  f(n)  cg(n) for all n  n0}
“f(n) = O(g(n))” means that f(n) is no worse than the function g(n) when n is large.
That is, asymptotic upper bound

33. Sidetrack: Visual Illustration of Big-O
cg(n)
Upper Bound
f(n)
f(n) = O(g(n))
Work required to finish
Our Algorithm n0
Size of input

34. Sidetrack: Tight Upper-Bound
We say Big O complexity of
3n2 + 2 = O(n2)  drop constants!
because we can show that there is a n0 and a c such that:
0  3n  cn2 for n  n0
e.g., c = 4 and n0 = 2 yields:
0  3n  4n2 for n  2
You could say
3n2 + 2 = O(n6) or 3n2 + 2 = O(n700)
But this is like answering:
How long does it take to drive to Washington DC?
Less than 11 years vs. Less than 5 hours

35. Sidetrack: Correct Interpretation of O()
Does not mean that it takes only one operation
Does mean that the work doesn’t change as N changes
Is notation for “constant work”
Does not mean that it takes N operations
Does mean that the work changes in a way that is proportional to N
Is a notation for “work grows at a linear rate”
O(1) or “Order One”
O(N) or “Order N”

36. Sidetrack: Ex: Sequential Steps
If steps appear sequentially (one after another), then add their respective O().
loop
. . .
endloop N
O(N + M) M

37. Sidetrack: Ex: Embedded Steps
If steps appear embedded (one inside another), then multiply their respective O().
loop
. . .
endloop M N
O(N*M)

38. Sidetrack: Classes of Big O Functions
constant
double logarithmic
logarithmic
polylogarithmic
linear
quasilinear
quadratic
cubic
polynomial
exponential (or geometric)
combinatorial
double exponential
O(1):
O(log log n):
O(log n):
O((log n)c), c>1:
O(n):
O(n logn) = O(log n!):
O(n2):
O(n3):
O(nc), c>1:
O(cn):
O(n!) = O(nn):
cO(c^n) :

39. Boolean queries: Exact match
Sec. 1.3
Boolean queries: Exact match
The Boolean retrieval model is being able to ask a query that is a Boolean expression:
Boolean Queries are queries using AND, OR and NOT to join query terms
Views each document as a set of words
Is precise: document matches condition or not.
Perhaps the simplest model to build an IR system on
Primary commercial retrieval tool for 3 decades.
Many search systems still use are Boolean:
, library catalog, Mac OS X Spotlight

40. Query optimization What is the best order for query processing?
Sec. 1.3
Query optimization
What is the best order for query processing?
Consider a query that is an AND of n terms.
For each of the n terms, get its postings, then AND them together.
Brutus 2 4 8 16 32 64
128
Caesar 1 2 4 5 8 16 21 32
Calpurnia 13 16
Query: Brutus AND Caesar AND Calpurnia 40

41. Query optimization example
Sec. 1.3
Query optimization example
Process in order of increasing freq:
start with smallest set, then keep cutting further.
This is why we kept
document freq. in dictionary
Brutus 2 4 8 16 32 64
128
Caesar 1 2 4 5 8 16 21 32
Calpurnia 13 16
Execute the query as (Calpurnia AND Brutus) AND Caesar.

42. Query optimization example
Sec. 1.3
Query optimization example
Brutus 2 4 8 16 32 64
128
Caesar 1 2 4 5 8 16 21 32
Calpurnia 13 16
Execution order: (Brutus AND Caesar) AND Calpurnia
Execution order: (Calpurnia AND Brutus) AND Caesar

43. More general optimization
Sec. 1.3
More general optimization
e.g., (madding OR crowd) AND (ignoble OR strife)
Get doc. freq.’s for all terms.
Estimate the size of each OR by the sum of its doc. freq.’s (conservative).
Process in increasing order of OR sizes.

44. More general optimization
If the query is:
friends AND romans AND (NOT countrymen)
how could we use the freq of countrymen?

45. Example: WestLaw http://www.westlaw.com/
Extended Boolean model w. ranking

46. Example: WestLaw http://www.westlaw.com/
Sec. 1.4
Example: WestLaw
Largest commercial (paying subscribers) legal search service (started 1975; ranking added 1992)
Tens of terabytes of data; 700,000 users
Majority of users still use boolean queries
Example query:
What is the statute of limitations in cases involving the federal tort claims act?
LIMIT! /3 STATUTE ACTION /S FEDERAL /2 TORT /3 CLAIM
/3 = within 3 words, /S = in same sentence