1. Evaluating Retrieval Systems with Findability Measurement Shariq Bashir PhD-Student Technology University of Vienna

2. Agenda Document FindabilityDocument Findability Calculating Findability MeasureCalculating Findability Measure GINI CoefficientGINI Coefficient Queries Creation for Findability MeasureQueries Creation for Findability Measure ExperimentsExperiments

3. Document Findability Large Findability Legal or Patent RetrievalLarge Findability of each and every document in Collection is considered an important factor in Legal or Patent Retrieval Settings. For example, in Patent Retrieval Settings, un- accessibility of a single related Patent document can approve wrong Patent application.

4. Document Findability Easy vs. Hard Findability called easy Findable top rank results –A patent is called easy Findable, if it is accessible on top rank results of its several relevant queries. harder will be its Findability –More the Patent will far away from the top rank results, the harder will be its Findability. –Why?, because users are more interested on only top rank results (say top 30).

5. Document Findability Considered two Retrieval Systems (RS1, RS2) and three Patents (P1, P2, P3). The following table shows the Findability values of three Patents on top 30 results. It is clear, RS2 makes all Patents more Findable than RS1. P1P2P3 RS1027 RS2475

6. What Makes Hard to Find Documents System Bias preference to some features –Bias is a term used in IR, when retrieval system give preference to some features of documents when it rank results of queries. PageRank BM25, BM25F, TF-IDF –Example, PageRank is bias toward larger in-links, BM25, BM25F, TF-IDF are bias toward large terms frequencies. why? –Bias is dangerous, why?, since under Bias some documents will be more findable, while rest of others will be very hard to find.

7. Bias with Findability analysis capture the bias Findability analysisWe can capture the bias impact of different retrieval systems using Findability analysis. system has less bias more FindableIf a system has less bias, then it will make the individual documents more Findable. Findability evaluation vs. Precision based EvaluationFindability evaluation vs. Precision based Evaluation –We can’t use Findability evaluation at individual queries level. –It is just large scale evaluation, only use for capturing the bias of retrieval systems.

8. Findability Measure Given a collection of documents d  D, with large set of Queries Q. k dq is the rank of d  D in the result set of query q  Q, c denotes the maximum rank that a user is willing to proceed down. The function f(k dq,c) returns a value of 1 if k dq <= c, and 0 otherwise.

9. GINI Coefficient For viewing the Bias of Retrieval System in a single value, we can use GINI coefficient. For computing GINI index, r(d i ) should be sort in ascending order. N total number of documents. If G = 0, then no bias, because all document are equally Findable. If G = 1, then only one document is Findable, and all other document have r(d) = 0.

10. Bias with Findability (Example) r(d) with RS1 r(d) with RS2 d129 d207 d3612 d4514 D53418 D6411 d73919 GINI0.580.18 GINI Coefficient with Lorenz Curve

11. Bias of Retrieval Systems Experiment Setting –We used total Patents listed under United State Patent Classification (USPC) classes – 433 (Dentistry), 424 (Drug, Bio-affecting and body treating compositions), 422 (Chemical apparatus and process disinfecting, deodorizing, preserving, or sterilizing), and 423 (Chemistry of inorganic compounds).

12. Experiment Setting Retrieval Systems used: BM25 –The OKAPI retrieval function (BM25). –Exact –Exact match model. –TFIDF LM –Language Modeling with term smoothing for Pseudo Relevance Feedback selection (LM). KLD –Kullback-Leibler divergence (KLD). QE TS –Term selection value (Robertson and Walker) (QE TS). –Pseudo Relevance Feedback documents selection using clustering approach (Cluster). top 35 50 terms –For all Query Expansion models, we used top 35 documents for Pseudo relevance feedback and 50 terms for query expansion.

13. Experiment Setting Queries Creation for Findability analysis In query creation, we try to reflect the approach of Patent Examiners, how they create their query sets during “Patent Invalidity Search”.

14. Experiment Setting Approach 1: single frequent terms Claim sections –First, we extract all the single frequent terms from the Claim sections which have support greater than some threshold. –Then we combine these single frequent terms with two, three, and four terms combinations for construction longer queries. Patent (A)----------- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Patent ---------- ---------- ---------- --- Use Patent (A) as a query for searching related documents.

15. Experiment Setting Terms with Support >= 3

17. Experiment Setting Approach 2: –If patent contain many rare terms, –then we can’t search all of its similar Patents –using queries collected from only single Patent document, we can’t search all of its similar Patents. Patent relatedness –In this Query Creation approach, we construct queries with considering Patent relatedness.

18. Experiment Setting Approach 2 Steps: related Patents in set (R) –(Step 1): For each Patent, group all of its related Patents in set (R) using k-nearest neighbor approach. R –(Step 2): Then using this R, construct its language model, for finding dominant terms which can search the documents in R. –Where P jm (t|R) is the probability of term t in set R, and P jm (t|corpos) is the probability of term t in whole collection. –This is similar approach, as terms in Language Modeling (Query Expansion) are used for brining up relevant documents. –(Step 3): Combine single terms with two, three, and four terms combinations for constructing longer queries.

19. Experiment Setting Properties of Queries used in Experiments CQG 1: Approach 1 CQG 2: Approach 2

20. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 1

21. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 1

22. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 1

23. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 2

24. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 2

25. Bias of Retrieval Systems with Patent Collection (433, 424) With Query Creation Approach 2

26. GINI Index of Retrieval Systems with Patent Collection (433, 424)

27. GINI Index of Retrieval Systems with Patent Collection (422, 423)

28. Future Work We are working toward improving Findability of Patents using Query Expansion approach. We have results, in which selecting better documents for Pseudo Relevance Feedback can improve the Findability of documents. Considering external provided Ontology in Query Expansion, can also create its role in improving Findability of documents.

29. References Leif Azzopardi, Vishwa Vinay, Retrievability: an evaluation measure for higher order information access tasks, CIKM '08: Proceeding of the 17th ACM conference on Information and knowledge management, pages 561-- 570, October 26-30, 2008, Napa Valley, California, USA. Chris Jordan, Carolyn Wattters, Qigang Gao, Using controlled query generation to evaluate blind relevance feedback algorithms, JCDL '06: Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, 2006, Pages 286--295, Chapel Hill, NC, USA. Tonya Custis, Khalid Al-Kofahi, A new approach for evaluating query expansion: query-document term mismatch, SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 575--582, July 23-27, 2007, Amsterdam, The Netherlands.