XML Information Retreival Hui Fang Department of Computer Science University of Illinois at Urbana-Champaign Some slides are borrowed from Nobert Fuhr’s XML Tutorial.

Outline XML basics Research Topics XML IR –Tasks –Retrieval methods –Clustering XML documents

XML standards

Basic XML Hierarchical document format for information exchange in WWW Self describing data (tags) Nested element structure having a root Element data can have –Attributes –Sub-elements (Slides from Jayavel Shanmugasundaram )

Attribute Element Example XML document - Finding Out About: A Cognitive Perspective on Search Engine Technology and the WWW Richard Belew San Diego 92093

Tree structure of XML documents book id=“rbelew” authortitle nameaddress First nameLast name cityZip code Finding…. RichardBelewSan Diego 92093

Basic XML standard does not deal with … Standardization of element names  XML namespaces Structure of element content  XML DTDs Data types of element content  XML schema

XML namespace Apples Bananas GPA Table Provide a method to avoid element name conflicts

XML namespace(Cont.) Apples Bananas GPA Table Provide a method to avoid element name conflicts

XML Document Type Definition Define the document structure with a list of legal elements Tove Jani Reminder Have a rest!

Research Topics related to XML

Research Topics IR areas –Retrieval Models –Query Languages –… DB areas –Query Languages –System architecture –Apply relational DB technology to XML data –Streaming XML –XML Query Processing –XML indexing and compression –……

XML IR

INEX: Initiative for the Evaluation for XML Retrieval Documents: 12,107 articles in XML format Queries: 30 Content-only; 30 Content and structure Relevance Assessments: by participating groups Participants: 36 active groups in 2003

CO search task Document as hierarchical structure of nested elements Type of elements is not considered Query refers to content only Query syntax as in standard text retrieval Task: Find smallest subtree(element) satisfying the query

Example of CO Topic augmented reality and medicine How virtual (or augmented )reality can contribute to improve the medical and surgical practice. In order to be considered relevant, a document/component must include considerations about applications of computer graphics and especially augmented (or virtual) reality to medice(including surgery). Augmented virtual reality medicine surgery improve computer assisted aided image

CAS search Task Queries contain explicit references to the XML structure, by restricing –The context of interest :target element –The context of certain search concepts (, ) pairs

Example of CAS topic article non-monotonic reasoning bdy/sec hdr//yr -calendar belief revision Retrieve all articles from the years that deal with works on non-monotonic reaonsing. Do not retrieve CfPs/calendar entries non-monotonic reasoning belief revision

XML Retrieval Methods XIRQL –XML query languages with IR-related features Language models JuruXML

XIRQL(I) CO Approaches : –Split document text into disjoint nodes –Index nodes separately –Aggregate indexing weights for higher- level elements (subtrees)

Index nodes as units for term weighting Application of known indexing functions (e.g. tf*idf)

Index nodes for relevance-oriented search document class="H.3.3" author John Smith title XML Retrieval Introduction chapter headingThis... heading SyntaxExamples heading sectionheading XML Query Lang. XQL section We describe syntax of XQL chapter Q1: syntax  example Q2: XQL

Combining weights …by disjunction Q1: syntax  example Q2: XQL 0.5 example0.8 XQL 0.7 syntax section1section2 0.3 XQL chapter 0.5 example 0.7 syntax *0.3=0.86 Need to return most specific element satisfying the query! 0.7*0.5=0.35

Combining weights … with augmentation weight Q2: XQL 0.5 example0.8 XQL 0.7 syntax section1section2 0.3 XQL chapter 0.30 example 0.42 syntax *0.3=

XIRQL(II) CAS approaches –Extension of XQL by Weighting and ranking Data types with vague predicates Structural relativism

XQL Expressions Path condition –search for single elements heading –parent-child: chapter/heading –ancestor-descendant: chapter//section –document root: /book/* Filter wrt. structure: //chapter[heading] Filter wrt. content: $and$ author=“John Smith”]

Data types with vague predicates Compares two values of a specific data-type –E.g. Near, broader, narrower Returns (probabilistic) matching value –E.g. “Search for an artist named Ulbrich, living in Frankfurt, Germany about 100 years ago”  Ernst Olbrich, Darmstadt, 1899 P(OlbrichUlbrich)=0.8 (phonetic similarity) P( )=0.9 (numeric similarity) P(Darmstadt Frankfurt)=0.7 (geographic distance)

Semantic Relativism Drop distinction attribute/element: ~author searches for attribute or element Generalize to data types: #personname searches for attribute/elements of specific data type

Language models Generate language models for each node in the tree Combine the children language models using linear interpolation Use EM approach to train the linear interpolation parameters

Element-specific language models ---CO Approaches

Higher level nodes: mixture of language models Query: dog and cat 0.5

Type-specific language models --- CAS approaches

0.5 “Return components of type x where it has component y that contains the query term w” e.g. return documents where the title is contains the word “bird” e.g. return documents where the body’s first section is contains the word “dog”

Juru-XML Element-specific indexing+vector space model: –Transform query into set of (term,path)- conditions –Vague matching of path conditions –Modified cosine similarity as retrieval function

JuruXML(1) ---Transform Query

JuruXML(2) ---Vague matching of path conditions

JuruXML(3) ---Retrieval function Standard cosine similarity –w Q (t i ): query term weight of term t i –w D (t i ): indexing weight of term t i in the document Modified cosine similarity –w Q (t i,c i Q ): query term weight of pair (t i,c i Q ) –w D (t i,c i D ): indexing weight of pair (t i,c i D ) in the document

For each query term (t i,c i Q ) treat all matched document terms (t i,c j D ) equally from the user perspective. Define a weight function w(c i Q ) –E.g. JuruXML(4) ---Alternative approach (Merging contexts)

Clustering XML documents

Document similarity Document representation: document  N-dimensional vector –N= # document features –Feature sets Text only Tags only Text + Tags Feature weighting in the document vector Similarity measure--- vector similarity –E.g. cosine measure

Clustering methods Hierarchical clustering: –Main weakness: quadratic complexity Partitional clustering: –K-means Linear time complexity Simplicity of its algorithm

K-Means clustering algorithm

Measuring clustering quality External quality: comparison of clusters with external classification –Entropy distribution of classes within clusters –Purity largest class in a cluster/cluster size Internal quality: calculate average inter- and intra- cluster similarities. –cohesiveness ( overall similarity)

Discussion Text alone give best results Text+tags: problem with weighting of tags vs. terms

Conclusion XML basics XML Retrieval Tasks and methods Clustering XML documents

Bayesian Networks

Context-dependent Retrieval The score of one element is given by RSV(Retrieval Status Value). RSV of node depends on RSVs of nodes in the context(parent nodes) Elements with highest values are then presented to the user.

Bayesian Networks

Bayesian Networks(Cont.)