A Markov Random Field Model for Term Dependencies Chetan Mishra CS 6501 Paper Presentation Ideas, graphs, charts, and results from paper of same name by Metzler and Croft 2005 (SIGIR)

Agenda 1.Motivation behind the work 2.Background – What is a Markov Random Field (MRF)? 3.Research Insight – How did the authors use MRF to model term dependencies? Results? 4.Future Work – If you thought this was interesting, how could you build on this? 5.Conclusion 6501: Text Mining2 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Motivation Terms are not independently distributed – A model incorporating term dependencies should outperform a model that ignores them One problem: models incorporating term dependencies seemed no better or worse – Statistical models weren’t effectively modeling term dependencies – Why? 6501: Text Mining3 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Motivation Two Problems (perspective of authors): – Problem 1: Most models have taken bag of word- like approaches (which have tremendous data requirements) – Solution 1: We need a new type of model – Problem 2: Term dependency modeling (even with a reasonable model) requires a significant corpus – Solution 2: Add to research testing collections large, web-scraped corpuses 6501: Text Mining4 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Background What is a Markov random field (MRF) model? – Fancy name for a bidirectional graph-based model – Often used in machine learning to succinctly model joint distributions MRF models are used in the paper to tackle the problem of document retrieval with response to a query 6501: Text Mining5 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Model Overview 6501: Text Mining6 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Model Overview 6501: Text Mining7 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Model Overview 6501: Text Mining8 By the joint probability law AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

The Markov Random Field Model 6501: Text Mining9 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

The Markov Random Field Model 6501: Text Mining10 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

The Markov Random Field Model The paper looks at the performance of three general types of dependencies: – Independence – Sequential dependence – Full dependence Visual Depiction: 6501: Text Mining11 Metzler and Croft ‘05 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Potential Functions 6501: Text Mining12 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion All log scale!

Parameter Training 6501: Text Mining13 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Parameter Training What optimization technique do we use? – Authors found a shape common to the metric surface via parameter sweep A hill-climbing search should work well 6501: Text Mining14 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Results Did MRF’s help? – I’d say so. Significant gains across data sets 6501: Text Mining15 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion IndependentSequential DependenceFull Dependence

Future Work 6501: Text Mining16 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

1.Motivation behind the work 2.Background – What is a Markov Random Field Model (MRF)? 3.Research Insight – How did the authors use MRF to model term dependencies? Results? 4.Future Work – If you thought this was interesting, how could you build on this? 5.Conclusion 6501: Text Mining17 AgendaMotivationBackgroundResearch InsightFuture WorkConclusion

Questions? 6501: Text Mining18