Project Management: The project is due on Friday inweek13. You have to demo the system to me (Lab will be booked) There will be a test in week 10. Course work contains: (1) assignment 1, test in week 10 and the project. The weight for the project is heavy. The purpose for test: (1) small part of coursework and (2) training for the final examination.

Extended Boolean Model: Disadvantages of “Boolean Model” : No term weight is used Counterexample: query q=Kx AND Ky. Documents containing just one term, e,g, Kx is considered as irrelevant as another document containing none of these terms. The size of the output might be too large or too small

Extended Boolean Model: The Extended Boolean model was introduced in 1983 by Salton, Fox, and Wu[703] The idea is to make use of term weight as vector space model. Strategy: Combine Boolean query with vector space model. Why not just use Vector Space Model? Advantages: It is easy for user to provide query.

Extended Boolean Model: Each document is represented by a vector (similar to vector space model.) Remember the formula. Query is in terms of Boolean formula. How to rank the documents?

Fig. Extended Boolean logic considering the space composed of two terms kx and ky only.

Extended Boolean Model: For query q=Kx or Ky, (0,0) is the point we try to avoid. Thus, we can use to rank the documents The bigger the better.

Extended Boolean Model: For query q=Kx and Ky, (1,1) is the most desirable point. We use to rank the documents. The bigger the better.

Extend the idea to m terms qor=k1 p k2 p … p Km qand=k1 p k2 p … p km

Properties: The p norm as defined above enjoys a couple of interesting properties as follows. First, when p=1 it can be verified that Second, when p= it can be verified that Sim(qor,dj)=max(xi) Sim(qand,dj)=min(xi)

Example: For instance, consider the query q=(k1 k2)  k3. The similarity sim(q,dj) between a document dj and this query is then computed as Any boolean can be expressed as a numeral formula.

Exercise: 1. Give the numeral formula for extended Boolean model of the query q=(k1 or k2 or k3)and (not k4 or k5). (assume that there are 5 terms in total.) 2. Assume that the document is represented by the vector (0.8, 0.1, 0.0, 0.0, 1.0). What is sim(q, d) for extended Boolean model? Also try to do more exercise for other Boolean formulas.

Fussy Set Theory Definition A fuzzy subset A of a universe of discourse U is characterized by a membership function which associate with each element u of U a number in the interval [0,1]. Set Theory: A={a, b, c}.Subset of A: {a, c}. An element is either in a set of not in a set. is either 0 or 1.

Set Theory Let U be the set of all elements (universe) There are three basic operations: AB={elements in A or in B}. AB={elements in both A and B} Not A=U-A.

Definition Let U be the universe of discourse, A and B be two fussy subsets of U, and be the complement of A relative to U. Also, let u be an element of U. Then,

Fuzzy Information Retrieval We first set up term-term correlation matric: For terms ki and kl, Where ni is the number of documents containing ki , nl is the number of documents containing kl And ni,l is the number of documents containing both ki and kl. Note Ci,i=1.

Fuzzy Information Retrieval We define a fuzzy set for each term ki. In the fuzzy set for ki , a document dj has a degree of membership ij computed as Example: c1,2=0.1, c1,3=0.21. D1=(0, 1, 1, 0). 1,1= 1-0.9*0.79. D2=(1, 0, 0, 0). 1,2= 1-0. (since c1,1=1.) How is d3=(1, 0, 1,0)?

Fuzzy Information Retrieval Whenever, the document dj contains a term that is strongly related to ki, then the document dj is belong to the fuzzy set of term ki, i.e., i,j is very close to 1. Example, c1,2=0.9, d1=(0, 1, 0, 0). 1,1 =1-(1-0.9)=0.9

Query: Query is a Boolean formula, e.g., q=Ka and (Kb or not Kc). q= (1, 1, 1) or (1, 1, 0) or (1, 0, 0). Suppose q is

Figure 1. Fuzzy document sets for the query . Each is a conjunctive component. is the query fuzzy set.

Where is the membership of in the fuzzy set associated with . q,j is the membership of document j for query q.

Some changes in the last slide. Instead of max{ }, we use +. Instead of min{ }, we use . Exercise: suppose there are 3 doc. and 4 terms. d1=(1, 0, 1, 0), d2=(1, 1, 0, 0), and d3=(0, 1, 1, 0). (1) Compute the term-term correlation matrix ci,j. (2) Compute i,j (membership of document j in term i.) (3) If the query q=(1, 0, 0, 0) or (1, 1, 0, 0), compute q,k for each document dk.