1. Modeling term relevancies in information retrieval using Graph Laplacian Kernels Shuguang Wang Joint work with Saeed Amizadeh and Milos Hauskrecht

2. A Problem in Document Retrieval There is a ‘gap’ between search queries and documents. Query: car

3. A Problem in Document Retrieval There is a ‘gap’ between search queries and documents. Google.com Bing.com Yahoo.com … Query: car

4. A Problem in Document Retrieval There is a ‘gap’ between search queries and documents. Google.com Bing.com Yahoo.com … Query: car

5. A Problem in Document Retrieval There is a ‘gap’ between search queries and documents. Google.com Bing.com Yahoo.com … Query: car Good enough?

6. A Problem in Document Retrieval What about the documents about automobiles, BMW, Benz, …? There are various expressions for a same entities. One solution is to expand the original user queries with some ‘relevant’ terms.

7. Traditional Query Expansion Methods Human and/or computer generated thesauri – Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. Human Relevance feedback – Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). – User click information (Yin et al., ECIR 2009) Automatic query expansion – Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents.

8. Traditional Query Expansion Methods Human and/or computer generated thesauri – Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. Human Relevance feedback – Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). – User click information (Yin et al., ECIR 2009) Automatic query expansion – Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. – Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming

9. Traditional Query Expansion Methods Human and/or computer generated thesauri – Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. Human Relevance feedback – Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). – User click information (Yin et al., ECIR 2009) Automatic query expansion – Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. – Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming Human Input

10. Traditional Query Expansion Methods Human and/or computer generated thesauri – Zhou et al., SIGIR 2007 proposed to expand query with MeSH concepts. Human Relevance feedback – Implicit feedback from human such as tracking eye movement (Buscher et al., SIGIR 2009). – User click information (Yin et al., ECIR 2009) Automatic query expansion – Pseudo Relevance Feedback first proposed in (Xu and Croft, SIGIR 1996). Use top ‘n’ documents from the initial search as the implicit feedback and select ‘relevant’ terms from these ‘n’ documents. – Analyze query flow graph in (Bordino et al., SIGIR 2010) Expensive, and time consuming Human Input

11. A Different View What we really need here is a way to estimate term-term relevance. Problem of finding expansion terms for user queries  Problem of finding ‘relevant’ terms given a similarity metric. How to derive a term-term similarity metric?

12. Term-Term Similarity Hypothesis: the metric ‘d’ should be smooth, i.e., d(t1) ~ d(t2) if ‘t 1 ’ and ‘t 2 ’are similar/relevant. Why not graph Laplacian kernels?! – We can easily have smoothness property. – We can also define distance metrics with it.

13. Define Affinity Graph Nodes are terms Edges are co-occurrences Weights of the edges are the number of documents terms co-occur

14. Graph Laplacian Kernels General Form Definition: Resistance: Diffusion: P-step Random Walk: … Recall:

15. Graph Laplacian Kernels General Form Definition: Resistance: Diffusion: P-step Random Walk: … How to choose hyper parameters?

16. Recall: Graph Laplacian Kernels General Form Definition: Resistance: Diffusion: P-step Random Walk: … How to choose g(λ)? How to choose hyper parameters?

17. Non-parametric kernel Learn the transformation g(λ) directly from training data. – If we know some terms are similar, we want to maximize their similarities. – At the same time, we want to have a smoother metric.

18. An Optimization Problem : the set of eigenvalues of original Laplacian graph ‘‘ ‘ ‘‘ t in‘ and t jn’ are pair of similar terms in the training document n’

19. An Optimization Problem : the set of eigenvalues of original Laplacian graph ‘‘ ‘ ‘‘ t in‘ and t jn’ are pair of similar terms in the training document n’ Maximize for known similar terms t in and t jn

20. An Optimization Problem : the set of eigenvalues of original Laplacian graph ‘‘ ‘ ‘‘ t in‘ and t jn’ are pair of similar terms in the training document n’ Penalize more for large eigenvalues

21. Kernel to Distances Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ1µ1 µ2µ2 µnµn titi tjtj Recall: We define:

22. Kernel to Distances Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ1µ1 µ2µ2 µnµn titi tjtj Recall: We define: d(i,j) = K ii +K jj -2K ij Euclidean Distance

23. Kernel to Distances Given the kernel K, we can define distances between any pair of nodes, d(i,j), in the graph. µ1µ1 µ2µ2 µnµn titi tjtj Recall: We define: d(i,j) = K ii +K jj -2K ij Euclidean Distance The distance metric derived from graph Laplacian Kernel is the Euclidean distances in the kernel space

24. Using term-term similarity in IR Deal with similarity between sets and terms. – In query expansion tasks, set of query terms is ‘S’ and a candidate expansion term is ‘t’. Transform the pair-wise distances, ‘d’, to set- to-term similarity. – Naïve methods: d max =max(d(S,t)) d avg =avg(d(S,t)) d min =min(d(S,t))

25. Set-to-term Similarity Query collapsing

26. Query Collapsing We have to compute eigen-decompostion again for each query. – It is too expensive for the online task. Approximation is possible. – We want to approximate the projection of `new’ point ‘S’ in the kernel space. – We need to add one element in the original eigenvector. µ1µ1 µ2µ2 µnµn A E µ1µ1 µ2µ2 µnµn A E S

27. Nystrőm Approximation For all nodes in the graph Laplacian, we have If the new point s’ was in the graph, it would satisfy the above as well.

28. Nystrőm Approximation For all nodes in the graph Laplacian, we have If the new point s’ was in the graph, it would satisfy the above as well.

29. Nystrőm Approximation For all nodes in the graph Laplacian, we have If the new point s’ was in the graph, it would satisfy the above as well.

30. Evaluation Two tasks: – Term prediction (scientific publication) Give the terms in the abstracts, predict the possible terms in the full body Compare with TFIDF, PRF, PLSI – Query expansion Compare with Lemur/Indri + PRF and Terrier + PRF Kernels: – Diffusion (optimized by line search) – Resistance – Non-parametric (optimized by line search) Set-to-term: – Average – Query collapse

31. Term prediction 6000 articles about 10 cancers downloaded from PubMed. – 80% as training and 20% as testing Given the terms in abstracts, rank all the candidate terms using the distance metrics. – The smaller the distances between candidate terms and query terms, the higher rank these terms are. Use AUC to evaluate (Joachims, ICML 2005)

32. Results

33. Query Expansion Four TREC datasets: Genomic 03 & 04, Adhoc TREC 7 & 8. We built graph using different set of terms in these datasets: – genes/proteins on Genomic 03 data – 5000 terms with highest TFIDF scores on Genomic 04 data. – 25% subsamples from all (~100k) unique terms from TREC7 & 8. Use Mean Average Precision (MAP) to evaluate the performance. Only Resistance Kernel

34. Results