1. CMPS 561 Boolean Retrieval
Ryan Benton
August 30, 2010

2. Agenda Indices IR System Models Processing Boolean Query
Algorithms for Intersection

3. Indices

4. Indices
Question:
How do we store documents and terms such that we can retrieve documents
Efficiently
Effectively
With reasonable space requirements?

5. Term-Document Matrix Create table Official Name: Rows: Terms
Columns: Document Ids
Official Name:
Term-Document Incidence Matrix
Also called: Inverted View of Collection
What are terms:
indexed Units (can think as words, but not always).

6. Term-Document Incidence Matrix
Record 1
Record 2
Record 3
Record 4
Term 1 1
Term 2
Term 3
Term 4

7. Term-Document Matrix Why
Rows  Vectors of documents containing term X.
Columns  Vectors of terms contained by document Y.
Technically, take the transpose of Matrix to get the columns vectors.
What are terms:
indexed Units (can think as words, but not always).

8. Document-Term Incidence Matrix
Record 1 1
Record 2
Record 3
Record 4
Note, some people would prefer this view instead. However, the term-Document is useful for other reasons  to be seen.

9. Term-Document Matrix Naïve Way of Building Some Calculations
Create and store the Matrix
Some Calculations
500,000 Terms
1,000,000 Documents
½ trillion entries : 500,000,000,00
All 0’s and 1’s.
Memory Impact
As documents and/or term list grows
Can’t keep in memory

10. Term-Document Matrix Observation: Term-Document Matrix  Sparse
Typically, only a small number of terms in any given document.
If typical document contains 1,000 terms
Matrix, in previous example, has 1 billion 1’s
1,000,000,000
Thus, 99.8% of matrix has 0’s.

11. Inverted Index Also called: Inverted File Dictionary of Terms
Vocabulary
Lexicon
Each term
List of documents in which it appears.
Each document sometimes called a posting.

12. Term-Document Incidence Matrix
Record 1
Record 2
Record 3
Record 4
Term 1 1
Term 2
Term 3
Term 4

13. Inverted Index Term 1 Record 1 Record 3 Term 2 Record 1 Record 24
Term 4
Record 1
Record 2
Record 3
Record 4

14. Inverted Index Note: Storage: Dictionary sorted alphabetically
Each ‘posting list’  sorted by ID
Storage:
Dictionary kept in memory
Postings  Depends on space.
In memory
on disk.

15. Inverted Index, Some Change
Term 1: 2
Record 1
Record 3
Term 2: 2
Record 1
Record 2
Term 3: 3
Record 2
Record 3
Record 4
Term 4: 4
Record 1
Record 2
Record 3
Record 4

16. IR System Models

17. Model S = (D, Q, T, V, F) Retrieval Status Values (RSV)
d Î D
q Î Q
F: D C Q  V
Fq: D  V
Retrieval Status Values (RSV)
T: “index terms”
Note F and Fq are the same in purpose – just different ways of expressing.

18. Model S = (D, Q, T, V, F) £ defined over elements of V is simple order
£¢ defined over elements of D by F is weak order
Breaks element of D into number of subsets
Each subset are simply ordered

19. Subject Catalog Model S = (D, Q, T, V, F) T = set of subject headings
D = 2T
V = { 0, 1 }
Fq(d) where q Î Q, d Î D
1, if q Î d
0, otherwise

20. Coordination Level System
S = (D, Q, T, V, F)
Q = 2T
D = 2T
V = { 0, 1 }
Fq(d) where
1, if q Í d
0, otherwise
F¢q(d) where
1, if |q Ç d| > k

21. Boolean Systems S = (D, Q, T, V, F) D = 2T Q = E V = { 0, 1 }
Fq(d) where
1, if q evaluates to True
With respect to document
0, otherwise

22. What is E? Let t Î T, Then If e Î E, Then If e1, e2 Î E, Then
t Î E
If e Î E, Then
Øe Î E
If e1, e2 Î E, Then
e1 Ú e2 Î E
e1 Ù e2 Î E
Nothing else is in E!

23. Document Representation
Set of Document IDs
D = {da} a=1,2,…,p
Set of all term IDs:
T = {ti} i = 1,2,…,n

24. Document Representation
Relation
D = { < da, ti, mD(da, ti)> }
mD: D x T  {0,1}
mD(da, ti)
1, if da contains ti
0, otherwise
D t = {da Î D | mD(da, t) = 1}
d º D d = {ti Î T | mD(d, ti) = 1}

25. Retrieval Function Retrieval Status Value (RSV) º F
RSVt(da) = mD(da, t)
RSVØe(da) = 1 - RSVe(da)
RSVe1Ùe2(da) = RSVe1(da) Ù RSVe2(da)
RSVe1Úe2(da) = RSVe1(da) Ú RSVe2(da)

26. Processing Boolean Queries

27. Boolean example
q = Ø(d Ú e) Ù (c Ú (a Ù b)) Ù Ø Ú Ú c Ù b d e a

28. Boolean Query Example (Method 1- based on documents )
q = Ø(d Ú e) Ù (c Ú (a Ù b))
Dda = {a,c}
RSVq(da) = 1 Ú Ù d Ø e c b a 1 1 1 1

29. Boolean Query Example (Method 2- based on inverted lists)
D t1Ùt2 = {da Î D | da Î Dt1 Ù da Î Dt2}
D t1Út2 = {da Î D | da Î Dt1 Ú da Î Dt2}
Dt = set of Documents containing term t
T = {a, b, c, d, e}
Da, Db, Dc, Dd, De,

30. Boolean Query Example (Method 2)
Output
Da Ç Db
Input
a Ù b b a Ù

31. Processing Boolean Query (Method 2)
Output Dt
De1
De2
De1Ùe2
De1Úe2
D \ De1
Query t e1 e2
e1Ùe2
e1Úe2
Øe1

32. Boolean Queries (Method 2)
and-queries (tiÙtj)
Construct a merged list M for Dti and Dtj.
Transfer all duplicated records Od on merge list to output
or-queries (tiÚtj)
Transfer all unique records Ou on merge list to output.

33. Boolean Queries (Method 2)
not-queries (tiÙØtj)
Construct a merged list M for Dti and Dtj.
FIRST_List
Remove all items appearing only once from First List
Transfer remaining items to output (i.e. Od ).
Create merge list composed of First_List and list composed of Dti
SECOND_List
Remove items appearing more than once from SECOND_List
Transfer remaining items to output (i.e. Oa).

34. Reminder - Inverted Index
Term 1
Record 1
Record 3
Term 2
Record 1
Record 2
Term 3
Record 2
Record 3
Record 4
Term 4
Record 1
Record 2
Record 3
Record 4

35. Example Query ((t1Út2) ÙØ t3) Let’s do the first part (t1Út2)
Dt1: {R1, R3)
Dt2: {R1, R2)
M(Dt1, Dt2) : {R1, R1, R2, R3}
Ou(t1Út2) : {R1, R2, R3}
M  Merging Operation, O  Output Selection

36. Example Query (cont’d)
Now, let’s handle the second part
((t1Út2) ÙØ t3)
Dt3: {R2, R3, R4)
M(Dt1Út2, Dt3) : {R1, R2, R2, R3, R3, R4}
Od((t1Út2)Ùt3) : {R2 , R3}
M(Dt1Út2, D(t1Út2)Ùt3) : {R1, R2, R2, R3, R3}
Oa((t1Út2)ÙØt3) : {R1}

37. Algorithms for Intersection

38. Algorithms – Basic Intersection (aka Merging)
Intersect(p1, p2)
answer  {}
While (p1 != NIL) and (p2 != NIL) Do
if docID(p1) = docID(p2)
Then ADD(answer, docID(p1))
p1  next(p1)
p2  next(p2)
Else if (docID(p1) < docID(p2))
Then p1 next(p1)
Else p2  next(p2)
Return answer

39. Algorithms – Intersection
Complexity: O(x + y)
For any given two posting lists
List A has size x
List B has size y
Note, this is upper bound.
Formally, Complexity: Q(N)
N can be either
Number of documents in collection
Note, this is a tight bound.

40. Observation In many cases, Boolean queries
Conjunctive in nature
Allows for a possible improvement based on posting size (term frequency)

41. Algorithms – Conjunctive Query Merging
IntersectConjunct(t1, t2, …, tz)
Terms  SortByIncreasingFrequency((t1, t2, …, tz))
Results  postings(first(Terms))
Terms  rest(Terms)
while (Terms != NIL) and (Results != NIL) Do
Results  Intersect(result, postings(first(Terms)))
Return Results

42. Why? By using least frequent term In practice
All results guaranteed to be no larger than least frequent term
In practice
The ‘intermediate’ list always places upper bounds on the size.

43. Variations on Boolean Extended Boolean Fuzzy Has standard operations:
AND, OR and NOT
Plus
Term Proximity
Within X words, sentences, paragraphs
Wildcard Matching
Fuzzy
Allow for range
Function F no longer restricted to {0,1}

44. Thank-you
Questions?

45. References
Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, Introduction to Information Retrieval, Chapter 1, 2008.
Abraham Bookstein and William Cooper, “A General Mathematical Model for Information Retrieval Systems”, The Library Quarterly, Vol 26, no. 2, pp
Vijay V. Raghavan’s Notes/Lecture Material
Material in Slides ued with permission