Introduction to Semistructured Data and XML

How the Web is Today HTML documents – often generated by applications – consumed by humans only – easy access: across platforms, across organizations No application interoperability: – HTML not understood by applications – Database technology: client-server

New Universal Data Exchange Format: XML A recommendation from the W3C XML = data XML generated by applications XML consumed by applications Easy access: across platforms, organizations

Paradigm Shift on the Web From documents (HTML) to data (XML) From information retrieval to data management For databases, also a paradigm shift: – from relational model to semistructured data – from data processing to data/query translation – from storage to transport

Semistructure data 1.Information integration: important new application that motivates what follows. 2.Semistructured data: a new data model designed to cope with problems of information integration. 3.XML (Extensible Markup Language) : a new Web standard that is essentially semistructured data. 4.XQUERY: an emerging standard query language for XML data.

Information Integration Problem: related data exists in many places. They talk about the same things, but differ in model, schema, conventions (e.g., terminology). Example: In the real world, every bar has its own database. Some may have relations like beer-price; others have an Microsoft Word file from which the menu is printed. Some keep phones of manufacturers but not addresses. Some distinguish beers and ales; others do not.

The Semistructured Data Model &o1 &o12&o24&o29 &o43 &96 &243 &206 &25 “Serge” “Abiteboul” 1997 “Victor” “Vianu” paper book paper references author title year http author title publisher author title page firstname lastname firstnamelastnamefirst last Bib Object Exchange Model (OEM) complex object atomic object

Characteristics of Semistructured Data Missing or additional attributes Multiple attributes Different types in different objects Heterogeneous collections Self-describing, irregular data, no a priori structure

Comparison with Relational Data { row: { name: “John”, phone: 3634 }, row: { name: “Sue”, phone: 6343 }, row: { name: “Dick”, phone: 6363 } } row name phone “John”3634“Sue”“Dick”

XML (Extensible Markup Language) A W3C standard to complement HTML Origins: Structured text SGML – Large-scale electronic publishing – Data exchange on the web Motivation: – HTML describes presentation – XML describes content

From HTML to XML HTML describes the presentation

HTML Bibliography Foundations of Databases Abiteboul, Hull, Vianu Addison Wesley, 1995 Data on the Web Abiteboul, Buneman, Suciu Morgan Kaufmann, 1999

XML Foundations… Abiteboul Hull Vianu Addison Wesley 1995 … XML describes the content

Why are we DB’ers interested? It’s data. That’s us. Database issues: – How are we going to model XML? (graphs). – How are we going to query XML? (XQuery) – How are we going to store XML (in a relational database? object-oriented? native?) – How are we going to process XML efficiently? (many interesting research questions!)

XML Terminology Tags: book, title, author, … – start tag:, end tag: Elements: …, … – elements can be nested – empty element: (Can be abbrv. ) XML document: Has a single root element Well-formed XML document: Has matching tags Valid XML document: conforms to a schema

Well-Formed XML 1. Declaration =. – Normal declaration is – “Standalone” means that there is no DTD specified. 2. Root tag surrounds the entire balance of the document.  is balanced by, as in HTML. 3. Any balanced structure of tags OK. – Option of tags that don’t require balance, like in HTML.

XML: An Example Richard Feynman The Character of Physical Law 1980 R.K. Narayan Waiting for the Mahatma 1981 R.K. Narayan The English Teacher 1980

XML – Elements … Xml is case and space sensitive Element opening and closing tag names must be identical Opening tags: “ ” Closing tags: “ ” Empty Elements have no data and no closing tag: – They begin with a “ ” closing tag attribute attribute valuedata open tag element name

XML – Attributes … Attributes provide additional information for element tags. There can be zero or more attributes in every element; each one has the form: attribute_name=“attribute_value” - There is no space between the name and the “=” - Attribute values must be surrounded by “ or ‘ characters Multiple attributes are separated by white space (one or more spaces or tabs). closing tag attribute attribute valuedata open tag element name

Elements The segment of an XML document between an opening and a corresponding closing tag is called an element. Malcolm Atchison (215) element not an element element, a sub-element of

XML – Data and Comments … Xml data is any information between an opening and closing tag Xml data must not contain the ‘ ’ characters Comments: closing tag attribute attribute valuedata open tag element name

XML text XML has only one “basic” type -- text. It is bounded by tags, e.g. The Big Sleep is still text XML text is called PCDATA (for parsed character data). It uses a 16-bit encoding.

XML – Nesting & Hierarchy Xml tags can be nested in a tree hierarchy Xml documents can have only one root tag Between an opening and closing tag you can insert: 1. Data 2. More Elements 3. A combination of data and elements Some Text More

Representing relational DBs: Two ways projects: title budget managedBy employees: name ssn age

Project and Employee relations in XML Pattern recognition Joe Joe Sandra Auto guided vehicle Sandra : Projects and employees are intermixed

Pattern recognition Joe Auto guided vehicles Sandra : Project and Employee relations in XML (cont’d) Joe Sandra : Employees follows projects

More XML: Oids and References Jane Mary John oids and references in XML are just syntax

XML Data Model (Graph)

Document Type Descriptors Sort of like a schema but not really. Inherited from SGML DTD standard BNF grammar establishing constraints on element structure and content Definitions of entities

DTD – An Example

DTD - !ELEMENT !ELEMENT declares an element name, and what children elements it should have Content types: – Other elements – #PCDATA (parsed character data) – EMPTY (no content) – ANY (no checking inside this structure) – A regular expression NameChildren

DTD - !ELEMENT (Contd.) A regular expression has the following structure: – exp 1, exp 2, exp 3, …, exp k : A list of regular expressions – exp*: An optional expression with zero or more occurrences – exp+: An optional expression with one or more occurrences – exp 1 | exp 2 | … | exp k : A disjunction of expressions

DTD - !ATTLIST <!ATTLIST Orange location CDATA #REQUIRED color ‘orange’> !ATTLIST defines a list of attributes for an element Attributes can be of different types, can be required or not required, and they can have default values. ElementAttributeTypeFlag

DTD – Well-Formed and Valid Well-Formed and Valid Not Well-Formed Well-Formed but Invalid Home

Example: An Address Book MacNiel, John Dr. John MacNiel 1234 Huron Street Rome, OH (321) Exactly one name At most one greeting As many address lines as needed (in order) Mixed telephones and faxes As many as needed

Specifying the structure name to specify a name element greet? to specify an optional (0 or 1) greet elements name,greet? to specify a name followed by an optional greet

Specifying the structure (cont) addr* to specify 0 or more address lines tel | fax a tel or a fax element (tel | fax)* 0 or more repeats of tel or fax * 0 or more elements

A DTD for the address book <!DOCTYPE addressbook [ <!ELEMENT person (name, greet?, address*, (fax | tel)*, *)> ]>

DTD for the example relational DB <!DOCTYPE db [... ]>

Summary of XML regular expressions Each element name is a tag. Its components are the tags that appear nested within, in the order specified. AThe tag A occurs e1,e2The expression e1 followed by e2 e*0 or more occurrences of e e?Optional -- 0 or 1 occurrences e+1 or more occurrences e1 | e2either e1 or e2 (e)grouping

41 XML Schema In XML format Includes primitive data types (integers, strings, dates, etc.) Supports value-based constraints (integers > 100) User-definable structured types Inheritance (extension or restriction) Foreign keys Element-type reference constraints

42 Sample XML Schema …

43 Subtyping in

XML Namespaces Namespaces are a simple and straightforward way to distinguish names used in XML documents, no matter where they come from The only reason namespaces exist, is to give elements and attributes programmer-friendly names that will be unique across the whole Internet 44

Example <h:html xmlns:xdc=" xmlns:h=" Book Review XML: A Primer Author Price Pages Date Simon St. Laurent /01 45

XML Namespaces The prefixes are linked to the full names using the attributes on the top element whose names begin xmlns : The prefixes are just shorthand placeholders for the full names Those full names are URIs, i.e. Web addresses 46

Extensibility in XML Anyone can invent new tags and attach a meaning to those tags But if every user creates its own XML definition for describing his data, it is not possible to achieve interoperability For example, one may prefer to use the tag name “POR”, while another prefers using the tag name “PurchaseOrderReq” In other words, a tagged document is not very useful without some kind of agreement on the tags among inter- operating applications 47

Many Efforts for Standardized Tags… HL7 for healthcare RosettaNet for supply chain integration in Information Technology and Electronic Components domain MPEG7 in multimedia applications ebXML for eBusiness Common Business Library (CBL) for electronic catalogs, purchase orders, etc. … 48

XML Parsers A parser takes an XML document and makes its structure and content available to an application through an API There are two main Application Programming Interfaces (APIs) for writing parsers: – Document Object Model (DOM) and – Simple API for XML (SAX) Today, many parsers are both DOM and SAX compliant 49

XML DOM Parser 50 Application Code Initialize Parser In memory DOM: Perform Processing XML Parser XML Document Begin parsing Parsing complete A parser validates and makes the data contained in an XML document available to the application

XSLT Processor 51 XML Document XSL Style Sheet 2 Output from Style Sheet 1 Output from Style Sheet 2 XSL Style Sheet 1 Parser XSLT Processor Converts an XML document to another form An XSL style sheet is a set of transformation instructions for converting a source XML document to a target document

52 table.xsl bar.xsl art.xsl

XML Querying Path Expressions : Bib.paper Bib.book.publisher Bib.paper.author.lastname Given an OEM instance, the value of a path expression p is a set of objects

Path Expressions Examples: DB = &o1 &o12&o24&o29 &o43 &o70&o71 &96 &243 &206 &25 “Serge” “Abiteboul” 1997 “Victor” “Vianu” paper book paper references author title year http author title publisher author title page firstname lastname firstnamelastnamefirst last Bib &o44&o45&o46 &o47&o48 &o49 &o50 &o51 &o52 Bib.paper={&o12,&o29} Bib.book.publisher={&o51} Bib.paper.author.lastname={&o71,&206} Bib.paper={&o12,&o29} Bib.book.publisher={&o51} Bib.paper.author.lastname={&o71,&206}

XQuery Emerging standard for querying XML documents. Basic form: FOR WHERE RETURN ; Sets of elements described by paths, consisting of: 1.URL, if necessary. 2.Element names forming a path in the semistructured data graph, e.g., //BAR/NAME = “start at any BAR node and go to a NAME child.” 3.Ending condition of the form [ ]

XQuery Overview: FOR-LET-WHERE-ORDERBY-RETURN = FLWOR FOR/LET Clauses WHERE Clause ORDERBY/RETURN Clause List of tuples Instance of Xquery data model

XQuery FOR $x in expr -- binds $x to each value in the list expr LET $x := expr -- binds $x to the entire list expr – Useful for common subexpressions and for aggregations

FOR v.s. LET FOR $x IN document("bib.xml") /bib/book RETURN $x FOR $x IN document("bib.xml") /bib/book RETURN $x Returns:... LET $x := document("bib.xml") /bib/book RETURN $x LET $x := document("bib.xml") /bib/book RETURN $x Returns:...

XQuery Find all book titles published after 1995: FOR $x IN document("bib.xml") /bib/book WHERE $x/year gt 1995 RETURN $x/title FOR $x IN document("bib.xml") /bib/book WHERE $x/year gt 1995 RETURN $x/title Result: abc def ghi

XQuery For each author of a book by Morgan Kaufmann, list all books s/he published: FOR $a IN distinct( document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN $a, FOR $t IN /bib/book[author=$a]/title RETURN $t FOR $a IN distinct( document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN $a, FOR $t IN /bib/book[author=$a]/title RETURN $t distinct = a function that eliminates duplicates

XQuery Result: Jones abc def Smith ghi

XQuery count = a (aggregate) function that returns the number of elms FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) gt 100 RETURN $p FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) gt 100 RETURN $p

XQuery Find books whose price is larger than average: LET $a:=avg( document("bib.xml") /bib/book/price) FOR $b in document("bib.xml") /bib/book WHERE $b/price gt $a RETURN $b LET $a:=avg( document("bib.xml") /bib/book/price) FOR $b in document("bib.xml") /bib/book WHERE $b/price gt $a RETURN $b

Examples for XQuery queries FOR $x IN doc( //employee [employeeSalary gt 70000]/employeeName RETURN $x/firstName, $x/lastName FOR $x IN doc( WHERE $x/employeeSalary gt RETURN $x/employeeName/firstName, $x/employeeName/lastName FOR $x IN doc( /project [projectNumber = 5]/projectWorker, $y IN doc( WHERE $x/hours gt 20.0 AND $y.ssn = $x.ssn RETURN $x/EmployeeName/firstName, $y/employeeName/lastName, $x/hours

transcript.xml … … cont’d … …

transcript.xml (cont’d)

XQuery Example: FOR $t IN document(“ WHERE = “MAT123” RETURN $t/Student Result:

XQuery (cont’d) Previous query doesn’t produce a well-formed XML document; the following does: { FOR $t IN document(“transcript.xml”)/Transcript WHERE = “MAT123” RETURN $t/Student } Transcript StudentStudentList FOR binds $t to Transcript elements one by one, filters using WHERE, then places Student-children as children of StudentList using RETURN FLWR inside XML

Document Restructuring with XQuery Transcript Reconstruct lists of students taking each class using the Transcript records: FOR $c IN distinct(document(“transcript.xml”)/CrsTaken) RETURN { FOR $t IN document(“transcript.xml”)/Transcript WHERE = $c/CrsCode AND = RETURN $t/Student ORDERBY } ORDERBY FLWOR inside RETURN

Document Restructuring (cont’d) Output elements have the form: Problem: the above element will be output twice – once when $c is bound to and once when it is bound to Note: grades are different – distinct( ) won’t eliminate transcript records that refer to same class! John Doe’s Bart Simpson’s

Document Restructuring (cont’d) Solution: instead of FOR $c IN distinct(document(“transcript.xml”)/CrsTaken) use FOR $c IN document(“classes.xml”)//Class where classes.xml lists course offerings (course code/semester) explicitly (no need to extract them from transcript records). Then $c is bound to each class exactly once, so each class roster will be output exactly once Document on next slide

SE Adrian Jones Circuits David Jones Databases Mary Doe TP John Smyth Algebra Ann White

XQuery Semantics So far the discussion was informal XQuery semantics defines what the expected result of a query is Defined analogously to the semantics of SQL

XQuery Semantics (cont’d) Step 1: Produce a list of bindings for variables – The FOR clause binds each variable to an ordered list of nodes specified by an XQuery expression. The expression can be: An XQuery query A function that returns a list of nodes – End result of a FOR clause: Ordered list of tuples of document nodes Each tuple is a binding for the variables in the FOR clause

XQuery Semantics (cont’d) Example (bindings): – Let FOR declare $A and $B – Bind $A to document nodes {v,w}; $B to {x,y,z} – Then FOR clause produces the following list of bindings for $A and $B: $A/v, $B/x $A/v, $B/y $A/v, $B/z $A/w, $B/x $A/w, $B/y $A/w, $B/z

XQuery Semantics (cont’d) Step 2: filter the bindings via the WHERE clause – Use each tuple-binding to substitute its components for variables; retain those bindings that make WHERE true – Example: WHERE = Binding: $A/w, where w = $B/x, where x = Then = so the WHERE condition is satisfied & binding retained

XQuery Semantics (cont’d) Step 3: Construct result – For each retained tuple of bindings, instantiate the RETURN clause – This creates a fragment of the output document – Do this for each retained tuple of bindings in sequence

78 XML Storage Approach 1: store XML in a relational database. Translate XQuery into a set of SQL queries. – Leverage 20 years of research & development. Approach 2: store XML in an object-oriented database system. – OO model is closest to XML, but systems do not perform well and are not well accepted. Approach 3: build an entire DBMS tailored to XML. – Still in the research phase.

79 &o1 &o3 &o2 &o4&o5 paper title author year &o6 “The Calculus”“…” “1986” Store XML in Ternary Relation [Florescu, Kossman 1999] Ref Val

80 Use DTD to derive Schema DTD: ODMG classes: [Christophides et al. 1994, Shanmugasundaram et al. 1999] class Employee public type tuple (name:string, address:Address, project:List(Project)) class Address public type tuple (street:string, …)

81 The Future Many research problems remain: – Efficient storage of XML – How to leverage relational DBMS – Update formalisms – Processing streaming data – Transactions – Everything else we think about in databases.