1. CSE3201/4500 Information Retrieval Systems
Term Weighting
CSE3201/4500
Information Retrieval Systems

2. Weighting Terms
Having decided on a set of terms for indexing, we need to consider whether all terms should be given the same significance. If not, how should we decide on their significance?

3. Weighting Terms - tf
Let tfij be the term frequency for term i on document j. The more a term appears in a document, the more likely it is to be a highly significant index term.

4. Weighting Terms - df & idf
Let dfi be document frequency of the i-th term.
Since the significance increases with a decrease in the document frequency, we have the inverse document frequency, idfi = loge (N/dfi) where N is the number of documents in the database; loge is the natural logarithm (ln in the calculator)

5. Weighting Terms - tf. idf
The above two indicators are very often multiplied together to form the “tf.idf” weight, wij = tfij * idfi
or as is now more popular wij = loge (1 + tfij ) (1 + idfi)

6. Example
Consider 5 document collection: D1= “Dogs eat the same things that cats eat” D2 = “No dog is a mouse” D3 = “Mice eat little things” D4 = “Cats often play with rats and mice” D5 = “Cats often play, but not with other cats”

7. Example - Cont. We might generate the following index sets:
V1 = ( dog, eat, cat ) V2 = ( dog, mouse ) V3 = ( mouse, eat ) V4 = ( cat, play, rat, mouse ) V5 = (cat, play)
System dictionary (cat,dog,eat,mouse,play,rat)

8. Example-Cont dfcat=3 idfcat=ln(5/3)=0.51 dfdog=2 idfdog=ln(5/2)=0.91
dfeat=2
idfeat=ln(5/2)=0.91
dfmouse=3
idfmouse=ln(5/3)=0.51
dfplay=2
idfplay=ln(5/2)=0.91
dfrat=1
idfrat=ln(5/1)=1.61

9. Example-Cont V1(cat, eat,dog) V2(dog,mouse)
wcat= tfcat * idfcat = 1 * 0.51 = 0.51
wdog= tfdog * idfdog = 1 * 0.91 = 0.91
weat= tfeat * idfat = 2 * 0.91 = 1.82
V2(dog,mouse)
wmouse= tfmouse * idfmouse = 1 * 0.51 = 0.51

10. Example-Cont V3(mouse,eat) V4(cat,mouse,play, rat)
wmouse= tfmouse * idfmouse = 1 * 0.51 = 0.51
weat= tfeat * idfat = 1 * 0.91 = 0.91
V4(cat,mouse,play, rat)
wcat= tfcat * idfcat = 1 * 0.51 = 0.51
wplay= tfplay * idfplay = 1 * 0.91 = 0.91
wrat= tfrat * idfrat = 1 * 1.61 = 1.61

11. Example-Cont V5 wcat= tfcat * idfcat = 2 * 0.51 = 1.02
wplay= tfplay * idfplay = 1 * 0.91 = 0.91

12. Example - cont. Dictionary: (cat,dog,eat,mouse,play,rat) Weights:
V1 = [cat(0.51), dog (0.91),eat(1.82), 0, 0,0 ]
V2 = [0,dog(0.91),0,mouse(0.51),0,0] V3 = [0,0,eat(0.91), mouse(0.51),0,0] V4 = [cat(0.51), 0,0,mouse(0.51), play(0.91), rat(1.61)] V5 = [cat(1.02),0,0,0, play (0.91),0]

13. A Larger Example
Doc 1: The problem of how to describe documents for retrieval is called indexing.
Doc 2: It is possible to use a document as its own index.
Doc 3: The problem is that a document will exactly match only one query, namely document itself.
Doc 4: The purpose of indexing then is to provide a description of a document so that it can be retrieved with queries that concern the same subject as the document.
Doc 5: It must be a sufficiently specific description so that the document will not be returned for queries unrelated to the document.

14. A Larger Example
Doc 6: A simple way of indexing a document is to give a single code from a predefined set.
Doc 7: We have the task of describing how we are going to match queries against document.
Doc 8: The vector space model creates a space in which both document and queries are represented by vectors.
Doc 9: A vector is obtained for each document and query from sets of index terms with associated weights.
Doc 10: In order to compare the similarity of these vectors, we may measure the angle between them.

15. A Larger Example
If we index these document using all words not on a stop list, we might obtain D1- problem, describe, documents, retrieval, called, indexing D2 - possible, document, own, index D3 - problem, document (*), exactly, match, one, query, namely D4 - purpose, indexing, provide, description, document(*), retrieved, queries, concern, subject D5 - sufficiently, specific, description, document(*), returned, queries, unrelated

16. A Larger Example
If we index these documents using all words not on a stop list, we might obtain D6- simple, way, indexing, document, give, single, code, predefined, list D7- task, describing, going, match, queries, against, documents D8- vector (*), space(*), model, creates, documents, queries, represented D9- vector, obtained, document, query, sets, index, terms, associated, weights D10- order, compare, similarity, vectors, measure, angle

17. A larger Example
We may now choose to stem the terms, which may leave us : D1- problem, describ, docu, retriev, call, index D2- possibl, docu, own, index D3- problem, docu (*), exact, match, on, quer, name D4- purpos, index, provid, descript, docu (*), retriev, queries, concern, subject D5- suffic, specif, descript, docu (*), return, quer, unrelat

18. A Larger Example
We may now choose to stem the terms, which may leave us: D6-simpl, way, index, docu, giv, singl, cod, predefin, list D7- task, describ, go, match, quer, against, docu D8- vect (*), spac (*), model, creat, docu, quer, represent D9- vect, obtain, docu, quer, set, index, terms, associat, weight D10- order, compar, similarit, vect, measur, angle

19. Document Frequencies

20. A Larger Example
We can now calculate the weights of the terms of one of the documents. For document 8, using the tf . idf formula, we give the terms the following weights: vect (2.41), spac (4.60), model (2.30), creat(2.30), docu (0.22), quer (0.51), represent (2.30)

21. CSE3201/4500 Information Retrieval Systems
Retrieval Model
CSE3201/4500
Information Retrieval Systems

22. Retrieval Process

23. Retrieval Paradigms How do we match? Produce non-ranked output
Boolean retrieval
Produce ranked output
vector space model
probabilistic retrieval

24. Advantages of Ranking
Good control over how many documents are viewed by a user.
Good control over in which order documents are viewed by a user.
The first documents that are viewed may help modify the order in which later documents are viewed.
The main disadvantage is computational cost.

25. Boolean Retrieval
A query is a set of terms combined by the Boolean connectives “and”, “or” and “not”.
e.g... FIND (document OR information) AND retrieval AND (NOT (information AND systems))
Each term is matched against this query and either matches (TRUE) or it doesn’t (FALSE)

26. Systems Provide Most systems provide match information such as
FIND (document or information)
1,000 records found
FIND (document OR information) AND retrieval
40 records found
FIND (document OR information) AND retrieval AND *NOT (information AND systems))
10 records found
SHOW

27. An Example
Consider the following document collection: D1 = “Dogs eat the same things that cats eat” D2 = “no dog is a mouse” D3 = “mice eat little things” D4 = “Cats often play with rats and mice” D5 = “cats often play, but not with other cats”
indexed by: D1 = dog, eat, cat D2 = dog, mouse D3 = mouse, eat D4 = cat, play, rat, mouse D5 =cat, play

28. An Example The Boolean query (cat AND dog) returns D1
(cat OR (dog AND eat)) returns D1, D4, D5

29. Problem with Boolean No ranking No weights on query terms
users must fuss with retrieved set size, structural reformulation
users must scan entire retrieved set
No weights on query terms
users cannot give more importance to some terms --- retrieval:2 AND system:1
users cannot give more importance to some clauses --- retrieval:1 AND (system OR model):2

30. Problem with Boolean No weights on document terms
no use can be made of importance of a term in a document --- if occurs frequently
no use can be made of importance of a term in the collection --- if occurs rarely

31. Any Good News for Boolean?
Yes.
Advantages
conceptually simple
computationally inexpensive
commercially available

32. Introduction to Vectors
A.B = |A||B| cos 
A=(a1, a2, a3,…, an), B=(b1, b2, b3,…, bn)
A.B = (a1b1+ a2b2+ a3b3+ …+ anbn)
Magnitude of a vector |A|=(a1, a2, a3,…, an) is defined as

33. Similarity Measures Inner product Cosine
A.B = (a1b1+ a2b2+ a3b3+ …+ anbn)
Cosine

34. The Vector Space Model
Each document and query is represented by a vector. A vector is obtained for each document and query from sets of index terms with associated weights.
The document and query representatives are considered as vectors in n dimensional space where n is the number of unique terms in the dictionary/document collection.
Measuring vectors similarity:
inner product
value of cosine of the angle between the two vectors.

35. Vector Space
Assume that document’s vector is represented by vector D and the query is represented by vector Q.
The total number of terms in the dictionary is n.
Similarity between D and Q is measured by the angle .

36. Inner Product

37. Cosine The similarity between D and Q can be written as:
Using the weight of the term as the components of D and Q:

38. Simple Example (1) Assume:
there are 2 terms in the dictionary (t1, t2)
Doc-1 contains t1 and t2, with weights 0.5 and 03 respectively.
Doc-2 contains t1 with weight 0.6
Doc-3 contains t2 with weights 0.4.
Query contains t2 with weight 0.5.

39. Simple Example (2) The vectors for the query and documents: Doc# wt1
0.5
0.3 2
0.6 3
0.4
Doc-1= (0.5,0.3)
Doc-2= (0.6,0)
Doc-3= (0,0.4)
Query = ( 0, 0.5)

40. Simple Example - Inner Product
D1=0.5x0+0.3x0.5=0.15
D2=0.6x0+0x0.5=0
D3=0x0+0.4x0.5=0.2
Ranking: D3, D1, D2
Doc#
wt1
wt2 1
0.5
0.3 2
0.6 3
0.4
Query = ( 0, 0.5)

41. Simple Example - Cosine
Similarity measured between Query(Q) and
Doc-1
Doc-2
Doc-3
Ranked output: D3, D1, D2

42. Large Example (1)
Consider the same five document collection D1= “Dogs eat the same things that cats eat” D2 = “No dog is mouse” D3 = “Mice eat little things” D4 = “Cats often play with rats and mice” D5 = “Cats often play, but not with other cats” Indexed by V1 = ( dog, eat, cat ) V2 = ( dog, mouse ) V3 = ( mouse, eat ) V4 = ( cat, play, rat, mouse ) V5 = (cat, play)

43. Large Example (2)
The set of all terms (dictionary) (cat, dog, eat, mouse, play, rat)
Using tf.idf weights, we obtain weights v1 = (cat(0.51), eat(1.82), dog(0.91)) v2 = (dog(0.91), mouse(0.51)) v3 = (mouse(0.51), eat(0.91)) v4 = (cat(0.51), play(0.91), rat(1.61), mouse(0.51)) v5 = (cat (1.02), play (0.91))

44. Large Example (3)
In the vector space model, we obtain vectors (0.51, 0.91, 1.82, 0.00, 0.00, 0.00) (0.00, 0.91, 0.00, 0.51, 0.00, 0.00) (0.00, 0.00, 0.91, 0.51, 0.00, 0.00) (0.51, 0.00, 0.00, 0.51, 0.91, 1.61) (1.02, 0.00, 0.00, 0.00, 0.91, 0.00)
6 dimensional space for 6 terms

45. Inner-Product
Query: “what do cats play with?” forms a query vector as (0.51, 0.00, 0.00, 0.00, 0.91, 0.00)
D1= 0.51x0.51+0x0.91+0x1.82+0x0+0x0.91+0x0=0.2601
D2= 0.00x x0+0x0+0.51x0+0x0.91+0x0=0
D3= 0.00x x0+0.91x0+0.51x0+0x0.91+0x0=0
D4= 0.51x0.51+0x0+0x0+0.51x0+0.91x x0=1.0882
D5= 1.02x0.51+0x0+0x0+0x0+0.91x0.91+0x0=1.3483
Ranking: D5, D4, D1, D2, D3

46. Cosine Similarity
Query: “what do cats play with?” forms a query vector as (0.51, 0.00, 0.00, 0.00, 0.91, 0.00)
using the cosine measure (cm), we obtain the following similarity measures: D1 = 0.512/[( )0.5 x( )0.5] D2 = 0.0 D3 = 0.0 D4 = ( )/[( )0.5x( )0.5] D5 = (0.51* )/[( )0.5x( )0.5]
Thus we obtain he ranking: D5, D4, D1, D2, D3 (or D3, D2)