Evaluating content-oriented XML retrieval: The INEX initiative Mounia Lalmas Queen Mary University of London

Slides:



Advertisements
Similar presentations
XIRQL: Eine Anfragesprache für Information Retrieval in XML-Dokumenten
Advertisements

XML - Beyond the Hype Everything you must know to succeed in implementing XML Lisa Shreve - Syscom Strategies Mike Rawlins - Rawlins EDI Consulting.
CSCI N241: Fundamentals of Web Design Copyright ©2004 Department of Computer & Information Science Introducing XHTML: Module B: HTML to XHTML.
COUNTER Update Peter Shepherd Project Director COUNTER STM Innovations Seminar, 2 December 2005.
INEX: Evaluating content-oriented XML retrieval Mounia Lalmas Queen Mary University of London
Evaluating XML retrieval: The INEX initiative Mounia Lalmas Queen Mary University of London
XML Retrieval: from modelling to evaluation Mounia Lalmas Queen Mary University of London qmir.dcs.qmul.ac.uk.
Even More TopX: Relevance Feedback Ralf Schenkel Joint work with Osama Samodi, Martin Theobald.
DOCUMENT TYPES. Digital Documents Converting documents to an electronic format will preserve those documents, but how would such a process be organized?
Retrieval of Information from Distributed Databases By Ananth Anandhakrishnan.
XML Craig Stewart Dr. Alexandra I. Cristea
Chapter 5: Introduction to Information Retrieval
Modern information retrieval Modelling. Introduction IR systems usually adopt index terms to process queries IR systems usually adopt index terms to process.
Multimedia Database Systems
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
XML Ranking Querying, Dagstuhl, 9-13 Mar, An Adaptive XML Retrieval System Yosi Mass, Michal Shmueli-Scheuer IBM Haifa Research Lab.
Overview of Collaborative Information Retrieval (CIR) at FIRE 2012 Debasis Ganguly, Johannes Leveling, Gareth Jones School of Computing, CNGL, Dublin City.
1 Entity Ranking Using Wikipedia as a Pivot (CIKM 10’) Rianne Kaptein, Pavel Serdyukov, Arjen de Vries, Jaap Kamps 2010/12/14 Yu-wen,Hsu.
Search Engines and Information Retrieval
Structure/XML Retrieval Mounia Lalmas Department of Computer Science Queen Mary University of London.
DYNAMIC ELEMENT RETRIEVAL IN A STRUCTURED ENVIRONMENT MAYURI UMRANIKAR.
Modern Information Retrieval
T.Sharon - A.Frank 1 Internet Resources Discovery (IRD) IR Queries.
Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.
A Mobile World Wide Web Search Engine Wen-Chen Hu Department of Computer Science University of North Dakota Grand Forks, ND
Evaluating the Performance of IR Sytems
ReQuest (Validating Semantic Searches) Norman Piedade de Noronha 16 th July, 2004.
1 CS 430: Information Discovery Lecture 20 The User in the Loop.
INEX 2003, Germany Searching in an XML Corpus Using Content and Structure INEX 2003, Germany Yiftah Ben-Aharon, Sara Cohen, Yael Grumbach, Yaron Kanza,
Modern Information Retrieval Chapter 2 Modeling. Can keywords be used to represent a document or a query? keywords as query and matching as query processing.
Web Search – Summer Term 2006 II. Information Retrieval (Basics Cont.) (c) Wolfgang Hürst, Albert-Ludwigs-University.
Chapter 4 Database Management Systems. Chapter 4Slide 2 What is a Database Management System (DBMS)?  Database An organized collection of related data.
Overview of Search Engines
CIS101 Introduction to Computing Week 06. Agenda Your questions Excel Exam during second hour Our status after the snow day Introduction to the Internet.
XML Information Retrieval and INEX Norbert Fuhr University of Duisburg-Essen.
INEX : Understanding XML Retrieval Evaluation Mounia Lalmas and Anastasios Tombros Queen Mary, University of London Norbert Fuhr University.
Creating a Basic Web Page
Search Engines and Information Retrieval Chapter 1.
XML Retrieval: A content-oriented perspective Mounia Lalmas Department of Computer Science Queen Mary, University of London.
1 Web Basics Section 1.1 Compare the Internet and the Web Compare Web sites and Web pages Identify Web browser components Describe types of Web sites Section.
Chapter 7 Web Content Mining Xxxxxx. Introduction Web-content mining techniques are used to discover useful information from content on the web – textual.
INEX – a broadly accepted data set for XML database processing? Pavel Loupal, Michal Valenta.
1 Formal Models for Expert Finding on DBLP Bibliography Data Presented by: Hongbo Deng Co-worked with: Irwin King and Michael R. Lyu Department of Computer.
A Simple Unsupervised Query Categorizer for Web Search Engines Prashant Ullegaddi and Vasudeva Varma Search and Information Extraction Lab Language Technologies.
Chapter 2 Architecture of a Search Engine. Search Engine Architecture n A software architecture consists of software components, the interfaces provided.
Semantic Learning Instructor: Professor Cercone Razieh Niazi.
April 14, 2003Hang Cui, Ji-Rong Wen and Tat- Seng Chua 1 Hierarchical Indexing and Flexible Element Retrieval for Structured Document Hang Cui School of.
1 Information Retrieval Acknowledgements: Dr Mounia Lalmas (QMW) Dr Joemon Jose (Glasgow)
ISP 433/533 Week 11 XML Retrieval. Structured Information Traditional IR –Unit of information: terms and documents –No structure Need more granularity.
Controlling Overlap in Content-Oriented XML Retrieval Charles L. A. Clarke School of Computer Science University of Waterloo Waterloo, Canada.
Chapter 6: Information Retrieval and Web Search
Introduction to Digital Libraries hussein suleman uct cs honours 2003.
1 01/10/09 1 INFILE CEA LIST ELDA Univ. Lille 3 - Geriico Overview of the INFILE track at CLEF 2009 multilingual INformation FILtering Evaluation.
Users and Assessors in the Context of INEX: Are Relevance Dimensions Relevant? Jovan Pehcevski, James A. Thom School of CS and IT, RMIT University, Australia.
Towards Contextual and Structural Relevance Feedback in XML Retrieval Lobna Hlaoua IRIT (Institut de Recherche en Informatique de Toulouse) Equipe SIG-RI.
Measuring How Good Your Search Engine Is. *. Information System Evaluation l Before 1993 evaluations were done using a few small, well-known corpora of.
Digital Libraries1 David Rashty. Digital Libraries2 “A library is an arsenal of liberty” Anonymous.
Information Retrieval
Comparing Document Segmentation for Passage Retrieval in Question Answering Jorg Tiedemann University of Groningen presented by: Moy’awiah Al-Shannaq
1 13/05/07 1/20 LIST – DTSI – Interfaces, Cognitics and Virtual Reality Unit The INFILE project: a crosslingual filtering systems evaluation campaign Romaric.
Information Retrieval Quality of a Search Engine.
General Architecture of Retrieval Systems 1Adrienn Skrop.
Using Blog Properties to Improve Retrieval Gilad Mishne (ICWSM 2007)
Information Retrieval in Practice
Information Storage and Retrieval Fall Lecture 1: Introduction and History.
Multimedia Information Retrieval
“INEX 2005: Playground for XML-retrieval” Sergey Chernov
Modern Information Retrieval
COUNTER Update February 2006.
Introduction to Information Retrieval
Presentation transcript:

Evaluating content-oriented XML retrieval: The INEX initiative Mounia Lalmas Queen Mary University of London

Outline Information retrieval Information retrieval XML retrieval XML retrieval Evaluating information retrieval Evaluating information retrieval Evaluating XML retrieval: INEX Evaluating XML retrieval: INEX

Information retrieval Example of a user information need (e.g. on the WWW): Find all documents about sailing charter agencies that (1) offer sailing boats in the Greek islands, and (2) are registered with the RYA. The documents should contain boat specification, price per week, and other contact details. A formal representation of an information need constitutes a query

Information retrieval IR is concerned with the representation, storage, organisation, and access to repositories of information, usually under the form of documents. Primary goal of an IR system Retrieve all the documents which are relevant (useful) to a user query, while retrieving as few non-relevant documents as possible.

DocumentsQuery Document representation Retrieval results Query representation IndexingFormulation Retrieval function Relevance feedback Conceptual model for IR

XML Retrieval Traditional IR is about finding relevant documents to a users information need, e.g. entire book. Traditional IR is about finding relevant documents to a users information need, e.g. entire book. XML allows users to retrieve document components that are more focussed to their information needs, e.g a chapter of a book instead of an entire book. XML allows users to retrieve document components that are more focussed to their information needs, e.g a chapter of a book instead of an entire book. The structure of documents is exploited to identify which document components (XML elements) to retrieve. The structure of documents is exploited to identify which document components (XML elements) to retrieve.

XML: eXtensible Mark-up Language Meta-language (user-defined tags) currently being adopted as the document format language by W3C Meta-language (user-defined tags) currently being adopted as the document format language by W3C Used to describe content and structure (and not layout) Used to describe content and structure (and not layout) Grammar described in DTD ( used for validation) Grammar described in DTD ( used for validation) Structured Document Retrieval Smith John Introduction into XML retrieval …. … …

XML: eXtensible Mark-up Language Use of XPath notation to refer to the XML structure Use of XPath notation to refer to the XML structure chapter/title: title is a direct sub-component of chapter //title: any title chapter//title: title is a direct or indirect sub-component of chapter chapter/paragraph[2]: any direct second paragraph of any chapter chapter/*: all direct sub-components of a chapter Structured Document Retrieval Smith John Introduction into SDR …. …

Queries Content-only (CO) queries Content-only (CO) queries Standard IR queries but here we are retrieving document components Standard IR queries but here we are retrieving document components London tube strikes London tube strikes Content-and-structure (CAS) queries Content-and-structure (CAS) queries Put on constraints on which types of components are to be retrieved Put on constraints on which types of components are to be retrieved E.g. Sections of an article in the Times about congestion charges E.g. Sections of an article in the Times about congestion charges E.g. Articles that contain sections about congestion charges in London, and that contain a picture of Ken Livingstone E.g. Articles that contain sections about congestion charges in London, and that contain a picture of Ken Livingstone

Conceptual model for XML retrieval Structured documentsContent + structure Inverted file + structure index tf, idf, acc Matching content + structure Presentation of related components DocumentsQuery Document representation Retrieval results Query representation IndexingFormulation Retrieval function Relevance feedback

Example of XML approaches The representation of a composite element (e.g. article and section) is defined as the aggregated representation of its sub-elements section p1 is about XML retrieval p2 is about XML, authoring paragraph article Sec3 is then also about XML (in fact very much about XML), retrieval, authoring

Example of XML approaches Document {?t 1, ?t 2, ?t 3 } Document {?t 1, ?t 2, ?t 3 } Title Section_1 Section_2 Title Section_1 Section_2 {0.9 t 1, 0.4 t 2 } {0.5 t 1 } {0.2 t 1, 0.7 t 3 } {0.9 t 1, 0.4 t 2 } {0.5 t 1 } {0.2 t 1, 0.7 t 3 } ? = Aggregated weight of t i in Document based on the instances of t i in the sub-elements (Title, Section_1 and Section_2)

Evaluation The goal of an IR system The goal of an IR system retrieve as many relevant documents as possible and as few non-relevant documents as possible retrieve as many relevant documents as possible and as few non-relevant documents as possible Comparative evaluation of technical performance of IR systems = effectiveness Comparative evaluation of technical performance of IR systems = effectiveness ability of the IR system to retrieve relevant documents and suppress non-relevant documents ability of the IR system to retrieve relevant documents and suppress non-relevant documents Effectiveness Effectiveness combination of recall and precision combination of recall and precision

Relevance A document is relevant if it has significant and demonstrable bearing on the matter at hand. A document is relevant if it has significant and demonstrable bearing on the matter at hand. Common assumptions: Common assumptions: Objectivity Objectivity Topicality Topicality Binary nature Binary nature Independence Independence

Recall / Precision Document collection Retrieved Relevant Retrieved and relevant

Recall / Precision relevant documents for a given query {d3, d5, d9, d25, d39, d44, d56, d71, d89, d123} rankdocprecisionrecallrankdocprecisionrecall d123d84d56D6d8d9d5111/12/33/61/102/103/ d129d187d25d48d250d113d34/105/144/105/10

Comparison of systems

Test collection Document collection = document themselves Document collection = document themselves depend on the task, e.g. evaluating web retrieval requires a collection of HTML documents. depend on the task, e.g. evaluating web retrieval requires a collection of HTML documents. Queries / requests Queries / requests simulate real user information needs. simulate real user information needs. Relevance judgements Relevance judgements stating for a query the relevant documents. stating for a query the relevant documents. See TREC See TREC

Evaluation of XML retrieval: INEX Evaluating the effectiveness of content-oriented XML retrieval approaches Evaluating the effectiveness of content-oriented XML retrieval approaches Collaborative effort = participants contribute to the development of the collection Collaborative effort = participants contribute to the development of the collection queries queries relevance assessments relevance assessments Similar methodology as for TREC, but adapted to XML retrieval. Similar methodology as for TREC, but adapted to XML retrieval.

INEX Test Collection The INEX test collection (2002) The INEX test collection (2002) Documents (~500MB), which consist of 12,107 articles in XML format from the IEEE Computer Society Documents (~500MB), which consist of 12,107 articles in XML format from the IEEE Computer Society 30 CO and 30 CAS queries 30 CO and 30 CAS queries Relevance assessments per retrieved components, by participating groups Relevance assessments per retrieved components, by participating groups Relevance defined in terms of relevance and coverage Relevance defined in terms of relevance and coverage Participants: 36 active groups worldwide Participants: 36 active groups worldwide In 2003, INEX has 36 CO and 30 CAS queries In 2003, INEX has 36 CO and 30 CAS queries Same document collections Same document collections CAS queries are defined according to a subset of XPath. CAS queries are defined according to a subset of XPath. Relevance assessments per retrieved components, by participating group Relevance assessments per retrieved components, by participating group Relevance defined in terms of exhaustivity and specificity Relevance defined in terms of exhaustivity and specificity Participants: 40 active groups worldwide Participants: 40 active groups worldwide INEX 2004 is just starting INEX 2004 is just starting

Example of CO topic Open standards for digital video in distance learning Open technologies behind media streaming in distance learning projects I am looking for articles/components discussing methodologies of digital video production and distribution that respect free access to media content through internet or via CD-ROMs or DVDs in connection to the learning process. Discussions of open versus proprietary standards of storing and sending digital video will be appreciated. media streaming,video streaming,audio streaming, digital video,distance learning,open standards,free access

Example of CAS topic //article[about(.,'formal methods verify correctness aviation systems')]/body//*[about(.,'case study application model checking theorem proving')] Find documents discussing formal methods to verify correctness of aviation systems. From those articles extract parts discussing a case study of using model checking or theorem proving for the verification. To be considered relevant a document must be about using formal methods to verify correctness of aviation systems, such as flight traffic control systems, airplane- or helicopter- parts. From those documents a body-part must be returned (I do not want the whole body element, I want something smaller). That part should be about a case study of applying a model checker or a theorem proverb to the verification. SPIN, SMV, PVS, SPARK, CWB

Relevance in XML A element is relevant if it has significant and demonstrable bearing on the matter at hand A element is relevant if it has significant and demonstrable bearing on the matter at hand Common assumptions in IR Common assumptions in IR Objectivity Objectivity Topicality Topicality Binary nature Binary nature Independence Independence section paragraph article

Relevance in XML Exhaustivity Exhaustivity how exhaustively a document component discusses the topic of request how exhaustively a document component discusses the topic of request Specificity Specificity how focused the component is on the topic of request (i.e. discusses no other, irrelevant topics) how focused the component is on the topic of request (i.e. discusses no other, irrelevant topics) 4-graded: 0, 1, 2, 3 4-graded: 0, 1, 2, 3 needed because of the structure needed because of the structure Relevance: (3,3), (2,3), (1,1), (0,0), etc Relevance: (3,3), (2,3), (1,1), (0,0), etc

Relevance assessment task Exhaustivity Exhaustivity Element parent element, children element Element parent element, children element Consistency Consistency Parent of a relevant element must also be relevant, although to a different extent Parent of a relevant element must also be relevant, although to a different extent Exhaustivity increase going Exhaustivity increase going Specificity decrease going Specificity decrease going Use of an online interface Use of an online interface Assessing a query takes a week! Assessing a query takes a week! Average 2 topics per participants Average 2 topics per participants Only participants that complete the assessment task have access to the collection Only participants that complete the assessment task have access to the collection section paragraph article

Metrics Recall/precision can used but must take into consideration: near misses (we do not retrieve the best component e.g. p[4] but one near enough e.g. p[2]) near misses (we do not retrieve the best component e.g. p[4] but one near enough e.g. p[2]) overlap (we retrieve a component e.g. doc[23] and one of its sub-components e.g. sec[3]) overlap (we retrieve a component e.g. doc[23] and one of its sub-components e.g. sec[3]) doc[23] sec[3] p[2] p[4]

Conclusion XML retrieval is not just about the effective retrieval of XML documents, but also how to evaluate the effectiveness XML retrieval is not just about the effective retrieval of XML documents, but also how to evaluate the effectiveness INEX 2004 INEX 2004 More rigorous query topic format (e.g. parser) More rigorous query topic format (e.g. parser) New metrics (e.g. not based on precision/recall) New metrics (e.g. not based on precision/recall) Tracks Tracks Relevance feedbackRelevance feedback InteractiveInteractive Heterogeneous collectionHeterogeneous collection Natural language queryNatural language query

Thank you