1. Exercising these ideas  You have a description of each item in a small collection. (30 web sites)  Assume we are looking for information about boxers, a type of dog.  List the items that are relevant to this information need. (If it is impossible to tell, as with #9, mark it not relevant.)  Rq = { }  Assume that an unspecified search system has returned a ranked result as follows:  (1, 4, 6, 8, 10, 12, 15, 16, 17, 18, 20, 23,26, 28, 30)  Show the ranking for the query as done on page 76.  Plot the precision vs. recall curve

2. Exercise, continued  We do a second query against the same collection. This time we are interested in the Boxer rescue organizations.  Rq = { }  Ranked result = (4, 6, 7, 10, 11, 15, 16, 17, 18, 19, 22, 23, 25, 28, 29)  Calculate the average precision at recall level 3 for these two queries.

3. More exercise  We redo the first query, using a new search algorithm. The ranked result is now this:  (4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 20, 23, 28, 29)  Produce the average recall vs. precision figures for these two algorithms. How would you describe the performance?  We redo the second query using the new search algorithm. The ranked result is this:  (7, 10, 11, 15, 17, 18, 19, 21, 23, 24, 26, 28, 29, 30)

4. Precision Histogram

5.  For each of the four searches, what is the R- precision?  Use a precision histogram to compare these two algorithms for the two queries shown.  Calculate the harmonic mean at document 5 in the rankings, using the first query and repeating for each algorithm.  Calculate the E measure for algorithm 1 using query 1 and a moderate preference for recall rather than precision.

6. Relevant docs known to the User |U| Answer Set |A| Relevant docs known to the User which were retrieved |Rk| |Ru| |Ru| + |Rk| Novelty = --------------- Coverage = ------ |U| |Rk| Relevant docs unknown to the User which were retrieved |Ru| Relevant docs |R| Collection

7.  Assume that the even number elements in the collection are known to the user.  Calculate the coverage ratio for algorithm 1 on search 1  Calculate the novelty ratio for the same algorithm and search