1 CSE544 XML, XQuery Wednesday, April 5, 2006
2 Announcements Project Ideas are posted (to discuss in class) Groups due today Proposals due Monday Two paper reviews are due on Monday Please read about E/R diagrams and OQL somewhere (e.g. in the textbook)
3 Outline XML: syntax, semantics, data, DTDs XPath Xquery Storage Chamberlin: XQuery: An XML query language. IBM Systems Journal 2002
4 XML Syntax tags: book, title, author, … start tag:, end tag: elements: …, … elements are nested empty element: abbrv. an XML document: single root element well formed XML document: if it has matching tags
5 XML Syntax Foundations of Databases Abiteboul … 1995 Foundations of Databases Abiteboul … 1995 attributes are alternative ways to represent data
6 XML Semantics: a Tree ! Mary Maple 345 Seattle John Thailand Mary Maple 345 Seattle John Thailand data Mary person name address name address streetnocity Maple345 Seattle John Thai phone id o555 Element node Text node Attribute node Order matters !!!
7 XML Data XML is self-describing Schema elements become part of the data –Reational schema: persons(name,phone) –In XML,, are part of the data, and are repeated many times Consequence: XML is much more flexible XML = semistructured data
8 Relational Data as XML John 3634 Sue 6343 Dick 6363 John 3634 Sue 6343 Dick 6363 row name phone “John”3634“Sue”“Dick” person XML: person namephone John3634 Sue6343 Dick6363
9 What is “Semi-Structured” ? Missing attributes: Could represent in a table with nulls John 1234 Joe John 1234 Joe no phone ! namephone John1234 Joe-
10 What is “Semi-Structured” ? Repeated attributes Impossible in tables: Mary Mary two phones ! namephone Mary ???
11 What is “Semi-Structured” ? Attributes with different types in different objects Nested collections (non 1NF) Heterogeneous collections: – contains both s and s John Smith 1234 John Smith 1234 structured name !
12 Document Type Definitions DTD DTD’s = old and archaic –You’ll hate it XML Schema = new, baroque, and horrible –Won’t discuss, or you’ll hate me XML document: well-formed = if tags are correctly closed Valid = if it has a DTD and conforms to it Validation is useful in data exchange
13 Very Simple DTD <!DOCTYPE company [ ]> <!DOCTYPE company [ ]>
14 Very Simple DTD John B Jim B John B Jim B Example of valid XML document:
15 DTD: The Content Model Content model: –Complex = a regular expression over other elements –Text-only = #PCDATA –Empty = EMPTY –Any = ANY –Mixed content = (#PCDATA | A | B | C)* content model
16 DTD: Regular Expressions <!ELEMENT name (firstName, lastName)) <!ELEMENT name (firstName?, lastName)) DTDXML <!ELEMENT person (name, phone*)) sequence optional <!ELEMENT person (name, (phone| ))) Kleene star alternation
17 Very Simple DTD <!DOCTYPE company [ ]> <!DOCTYPE company [ ]>
18 Very Simple DTD John B Jim B John B Jim B Example of valid XML document:
19 Attributes in DTDs
20 Attributes in DTDs <!ATTLIST person age CDATA #REQUIRED id ID #REQUIRED manager IDREF #REQUIRED manages IDREFS #REQUIRED > <!ATTLIST person age CDATA #REQUIRED id ID #REQUIRED manager IDREF #REQUIRED manages IDREFS #REQUIRED > <person age=“25” id=“p29432” manager=“p48293” manages=“p34982 p423234”> <person age=“25” id=“p29432” manager=“p48293” manages=“p34982 p423234”>
21 Attributes in DTDs Types: CDATA = string ID = key IDREF = foreign key IDREFS = foreign keys separated by space (Monday | Wednesday | Friday) = enumeration NMTOKEN = must be a valid XML name NMTOKENS = multiple valid XML names ENTITY = you don’t want to know this
22 Attributes in DTDs Kind: #REQUIRED #IMPLIED = optional value = default value value #FIXED = the only value allowed
23 Using DTDs Must include in the XML document Either include the entire DTD: – Or include a reference to it: – Or mix the two... (e.g. to override the external definition)
24 Querying XML Data XPath = simple navigation through the tree XQuery = the SQL of XML XSLT = recursive traversal –will not discuss in class
25 Sample Data for Queries Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998 Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998
26 Data Model for XPath bib book publisherauthor.. Addison-WesleySerge Abiteboul The root The root element
27 XPath: Simple Expressions Result: Result: empty (there were no papers) /bib/book/year /bib/paper/year
28 XPath: Restricted Kleene Closure Result: Serge Abiteboul Rick Hull Victor Vianu Jeffrey D. Ullman Result: Rick //author /bib//first-name
29 Xpath: Text Nodes Result: Serge Abiteboul Victor Vianu Jeffrey D. Ullman Rick Hull doesn’t appear because he has firstname, lastname Functions in XPath: –text() = matches the text value –node() = matches any node (= * or text()) –name() = returns the name of the current tag /bib/book/author/text()
30 Xpath: Wildcard Result: Rick Hull * Matches any element //author/*
31 Xpath: Attribute Nodes Result: means that price is has to be an attribute
32 Xpath: Predicates Result: Rick Hull /bib/book/author[firstname]
33 Xpath: More Predicates Result: … … /bib/book/author[firstname][address[.//zip][city]]/lastname
34 Xpath: More Predicates < 60] < 25] /bib/book[author/text()]
35 Xpath: Summary bibmatches a bib element *matches any element /matches the root element /bibmatches a bib element under root bib/papermatches a paper in bib bib//papermatches a paper in bib, at any depth //papermatches a paper at any depth paper|bookmatches a paper or a a price attribute price attribute in book, in bib matches…
36 XQuery Based on Quilt, which is based on XML-QL Uses XPath to express more complex queries
37 FLWR (“Flower”) Expressions FOR... LET... WHERE... RETURN... FOR... LET... WHERE... RETURN...
38 FOR-WHERE-RETURN Find all book titles published after 1995: FOR $x IN document("bib.xml")/bib/book WHERE $x/year/text() > 1995 RETURN $x/title FOR $x IN document("bib.xml")/bib/book WHERE $x/year/text() > 1995 RETURN $x/title Result: abc def ghi
39 FOR-WHERE-RETURN Equivalently (perhaps more geekish) FOR $x IN document("bib.xml")/bib/book[year/text() > 1995] /title RETURN $x FOR $x IN document("bib.xml")/bib/book[year/text() > 1995] /title RETURN $x And even shorter: document("bib.xml")/bib/book[year/text() > 1995] /title
40 FOR-WHERE-RETURN Find all book titles and the year when they were published: FOR $x IN document("bib.xml")/bib/book RETURN { $x/title/text() } { $x/year/text() } We can construct whatever XML results we want !
41 Answer How to cook a Turkey 2003 Cooking While Watching TV 2004 Turkeys on TV
42 FOR-WHERE-RETURN Notice the use of “{“ and “}” What is the result without them ? FOR $x IN document("bib.xml")/bib/book RETURN $x/title/text() $x/year/text()
43 XQuery: Nesting For each author of a book by Morgan Kaufmann, list all books she published: FOR $b IN document(“bib.xml”)/bib, $a IN $b/book[publisher /text()=“Morgan Kaufmann”]/author RETURN { $a, FOR $t IN $b/book[author/text()=$a/text()]/title RETURN $t } FOR $b IN document(“bib.xml”)/bib, $a IN $b/book[publisher /text()=“Morgan Kaufmann”]/author RETURN { $a, FOR $t IN $b/book[author/text()=$a/text()]/title RETURN $t } In the RETURN clause comma concatenates XML fragments
44 XQuery Jones abc def Smith ghi Jones abc def Smith ghi Result:
45 Aggregates Find all books with more than 3 authors: count = a function that counts avg = computes the average sum = computes the sum distinct-values = eliminates duplicates FOR $x IN document("bib.xml")/bib/book WHERE count($x/author)>3 RETURN $x
46 Aggregates Same thing: FOR $x IN document("bib.xml")/bib/book[count(author)>3] RETURN $x
47 XQuery Find books whose price is larger than average: FOR $b in document(“bib.xml”)/bib LET $a:=avg($b/book/price/text()) FOR $x in $b/book WHERE $x/price/text() > $a RETURN $x FOR $b in document(“bib.xml”)/bib LET $a:=avg($b/book/price/text()) FOR $x in $b/book WHERE $x/price/text() > $a RETURN $x LET binds a variable to one value; FOR iterates a variable over a list of values We will come back to that
48 FOR-WHERE-RETURN “Flatten” the authors, i.e. return a list of (author, title) pairs FOR $b IN document("bib.xml")/bib/book, $x IN $b/title/text(), $y IN $b/author/text() RETURN { $x } { $y } Answer: abc efg abc hkj
49 FOR-WHERE-RETURN For each author, return all book titles he/she wrote FOR $b IN document("bib.xml")/bib, $x IN $b/book/author/text() RETURN { $x } { FOR $y IN $b/book[author/text()=$x]/title RETURN $y } What about duplicate authors ? Answer: efg abc klm....
50 FOR-WHERE-RETURN Same, but eliminate duplicate authors: FOR $b IN document("bib.xml")/bib LET $a := distinct-values($b/book/author/text()) FOR $x IN $a RETURN $x { FOR $y IN $b/book[author/text()=$x]/title RETURN $y }
51 FOR-WHERE-RETURN Same thing: FOR $b IN document("bib.xml")/bib, $x IN distinct-values($b/book/author/text()) RETURN $x { FOR $y IN $b/book[author/text()=$x]/title RETURN $y }
52 SQL and XQuery Side-by-side Product(pid, name, maker, price) Find all product names, prices, sort by price SELECT x.name, x.price FROM Product x ORDER BY x.price SQL FOR $x in document(“db.xml”)/db/Product/row ORDER BY $x/price/text() RETURN { $x/name, $x/price } XQuery
53 abc 7 def Answers Notice: this is NOT a well-formed document ! (WHY ???) NamePrice abc7 def23...
54 Producing a Well-Formed Answer { FOR $x in document(“db.xml”)/db/Product/row ORDER BY $x/price/text() RETURN { $x/name, $x/price } }
55 abc 7 def Xquery’s Answer Now it is well-formed !
56 SQL and XQuery Side-by-side Product(pid, name, maker, price) Company(cid, name, city, revenues) Find all products made in Seattle SELECT x.name FROM Product x, Company y WHERE x.maker=y.cid and y.city=“Seattle” SQL FOR $r in document(“db.xml”)/db, $x in $r/Product/row, $y in $r/Company/row WHERE $x/maker/text()=$y/cid/text() and $y/city/text() = “Seattle” RETURN { $x/name } XQuery FOR $y in /db/Company/row[city/text()=“Seattle”], $x in /db/Product/row[maker/text()=$y/cid/text()] RETURN { $x/name } Cool XQuery
abc efg …. …
58 SQL and XQuery Side-by-side For each company with revenues < 1M count the products over $100 SELECT y.name, count(*) FROM Product x, Company y WHERE x.price > 100 and x.maker=y.cid and y.revenue < GROUP BY y.cid, y.name FOR $r in document(“db.xml”)/db, $y in $r/Company/row[revenue/text() { $y/name/text() } { count($r/Product/row[maker/text()=$y/cid/text()][price/text()>100]) }
59 SQL and XQuery Side-by-side Find companies with at least 30 products, and their average price SELECT y.name, avg(x.price) FROM Product x, Company y WHERE x.maker=y.cid GROUP BY y.cid, y.name HAVING count(*) > 30 FOR $r in document(“db.xml”)/db, $y in $r/Company/row LET $p := $r/Product/row[maker/text()=$y/cid/text()] WHERE count($p) > 30 RETURN { $y/name/text() } avg($p/price/text()) A collection An element
60 FOR v.s. LET FOR Binds node variables iteration LET Binds collection variables one value
61 FOR v.s. LET FOR $x IN /bib/book RETURN { $x } FOR $x IN /bib/book RETURN { $x } Returns:... LET $x := /bib/book RETURN { $x } LET $x := /bib/book RETURN { $x } Returns:...
62 XML from/to Relational Data XML publishing: –relational data XML XML storage: –XML relational data
63 XML Publishing Relational Database Application Web XML publishing Tuple streams XML SQL Xpath/ XQuery
64 XML Publishing Relational schema: Student(sid, name, address) Course(cid, title, room) Enroll(sid, cid, grade) StudentCourse Enroll
65 XML Publishing Operating Systems MGH084 John Seattle 3.8 … Database EE045 Mary Shoreline 3.9 … … Operating Systems MGH084 John Seattle 3.8 … Database EE045 Mary Shoreline 3.9 … … Other representations possible too Group by courses: redundant representation of students
66 XML Publishing First thing to do: design the DTD:
67 { FOR $x IN /db/Course/row RETURN { $x/title/text() } { $x/room/text() } { FOR $y IN /db/Enroll/row[cid/text() = $x/cid/text()] $z IN /db/Student/row[sid/text() = $y/sid/text()] RETURN { $z/name/text() } { $z/address/text() } { $y/grade/text() } } { FOR $x IN /db/Course/row RETURN { $x/title/text() } { $x/room/text() } { FOR $y IN /db/Enroll/row[cid/text() = $x/cid/text()] $z IN /db/Student/row[sid/text() = $y/sid/text()] RETURN { $z/name/text() } { $z/address/text() } { $y/grade/text() } } Now we write an XQuery to export relational data XML Note: result is is the right DTD
68 XML Publishing Query: find Mary’s grade in Operating Systems FOR $x IN /xmlview/course[title/text()=“Operating Systems”], $y IN $x/student/[name/text()=“Mary”] RETURN $y/grade/text() FOR $x IN /xmlview/course[title/text()=“Operating Systems”], $y IN $x/student/[name/text()=“Mary”] RETURN $y/grade/text() XQuery SELECT Enroll.grade FROM Student, Enroll, Course WHERE Student.name=“Mary” and Course.title=“OS” and Student.sid = Enroll.sid and Enroll.cid = Course.cid SQL Can be done automatically
69 XML Publishing How do we choose the output structure ? Determined by agreement with partners/users Or dictated by committees –XML dialects (called applications) = DTDs XML Data is often nested, irregular, etc No normal forms for XML
70 XML Storage Most often the XML data is small –E.g. a SOAP message –Parsed directly into the application (DOM API) Sometimes XML data is large –need to store/process it in a database The XML storage problem: –How do we choose the schema of the database ?
71 XML Storage Three solutions: Schema derived from DTD Storing XML as a graph: “Edge relation” Store it as a BLOB –Simple, boring, inefficient –Won’t discuss in class
72 Designing a Schema from DTD Design a relational schema for: <!DOCTYPE company [ ]> <!DOCTYPE company [ ]>
73 Designing a Schema from DTD First, construct the DTD graph: company personproduct ssnname officephonepidpriceavail.descr. * * * We ignore the order
74 Designing a Schema from DTD Next, design the relational schema, using common sense. company personproduct ssnname officephonepidpriceavail.descr. * * * Person(ssn, name, office) Phone(ssn, phone) Product(pid, name, price, avail., descr.) Which attributes may be NULL ? (Look at the DTD)
75 Designing a Schema from DTD What happens to queries: FOR $x IN /company/product[description] RETURN { $x/name, $x/description } FOR $x IN /company/product[description] RETURN { $x/name, $x/description } SELECT Product.name, Product.description FROM Product WHERE Product.description IS NOT NULL SELECT Product.name, Product.description FROM Product WHERE Product.description IS NOT NULL
76 Storing XML as a Graph Sometimes we don’t have a DTD: How can we store the XML data ? Every XML instance is a tree Store the edges in an Edge table Store the #PCDATA in a Value table
77 Storing XML as a Graph db book publisher titleauthor titleauthor titlestate “Complete Guide to DB2” “Chamberlin”“Transaction Processing” “Bernstein”“Newcomer” “Morgan Kaufman” “CA” SourceTagDest 0db1 1book2 2title3 2author4 1book5 5title6 5author7... SourceVal 3Complete guide... 4Chamberlin 6... Edge Value Can be ANY XML data (don’t know DTD)
78 Storing XML as a Graph What happens to queries: FOR $x IN /db/book[author/text()=“Chamberlin”] RETURN $x/title FOR $x IN /db/book[author/text()=“Chamberlin”] RETURN $x/title db book author title “Chamberlin”Return value xdb xbook xauthorxtitle vauthor vtitle
79 Storing XML as a Graph What happens to queries: SELECT vtitle.value FROM Edge xdb, Edge xbook, Edge xauthor, Edge xtitle, Value vauthor, Value vtitle WHERE xdb.source = 0 and xdb.tag = ‘db’ and xdb.dest = xbook.source and xbook.tag = ‘book’ and xbook.dest = xauthor.source and xauthor.tag = ‘author’ and xbook.dest = xtitle.source and xtitle.tag = ‘title’ and xauthor.dest = vauthor.source and vauthor.value = ‘Chamberlin” and xtitle.dest = vtitle.source SELECT vtitle.value FROM Edge xdb, Edge xbook, Edge xauthor, Edge xtitle, Value vauthor, Value vtitle WHERE xdb.source = 0 and xdb.tag = ‘db’ and xdb.dest = xbook.source and xbook.tag = ‘book’ and xbook.dest = xauthor.source and xauthor.tag = ‘author’ and xbook.dest = xtitle.source and xtitle.tag = ‘title’ and xauthor.dest = vauthor.source and vauthor.value = ‘Chamberlin” and xtitle.dest = vtitle.source A 6-way join !!!
80 XML in SQL Server 2005 Create tables with attributes of type XML Use Xquery in SQL queries Rest of the slides are from: Shankar Pal et al., Indexing XML data stored in a relational database, VLDB’2004
81 CREATE TABLE DOCS ( ID int primary key, XDOC xml) SELECT ID, XDOC.query(’ for $s in “ ”]//SECTION return {data($s/TITLE)} ') FROM DOCS SELECT ID, XDOC.query(’ for $s in “ ”]//SECTION return {data($s/TITLE)} ') FROM DOCS
82 XML Methods in SQL Query() = returns XML data type Value() = extracts scalar values Exist() = checks conditions on XML nodes Nodes() = returns a rowset of XML nodes that the Xquery expression evaluates to
83 Examples From here: p?url=/library/en- us/dnsql90/html/sql2k5xml.asp p?url=/library/en- us/dnsql90/html/sql2k5xml.asp
84 XML Type CREATE TABLE docs ( pk INT PRIMARY KEY, xCol XML not null ) CREATE TABLE docs ( pk INT PRIMARY KEY, xCol XML not null )
85 Inserting an XML Value INSERT INTO docs VALUES (2, ' XML Schema Benefits Features ') INSERT INTO docs VALUES (2, ' XML Schema Benefits Features ')
86 Query( ) SELECT pk, = 123]//section') FROM docs SELECT pk, = 123]//section') FROM docs
87 Exists( ) SELECT = 123]//section') FROM docs WHERE xCol.exist = 123]') = 1 SELECT = 123]//section') FROM docs WHERE xCol.exist = 123]') = 1
88 Value( ) SELECT xCol.value( = 3]/title)[1])', 'nvarchar(max)') FROM docs SELECT xCol.value( = 3]/title)[1])', 'nvarchar(max)') FROM docs
89 Nodes( ) SELECT nref.value('first-name[1]', 'nvarchar(50)') FirstName, nref.value('last-name[1]', 'nvarchar(50)') LastName AS R(nref) WHERE nref.exist('.[first-name != "David"]') = 1 SELECT nref.value('first-name[1]', 'nvarchar(50)') FirstName, nref.value('last-name[1]', 'nvarchar(50)') LastName AS R(nref) WHERE nref.exist('.[first-name != "David"]') = 1
90 Nodes( ) SELECT 'varchar(max)') LastName FROM docs CROSS APPLY xCol.nodes('//book') AS R(nref) SELECT 'varchar(max)') LastName FROM docs CROSS APPLY xCol.nodes('//book') AS R(nref)
91 Internal Storage XML is “shredded” as a table A few important ideas: –Dewey decimal numbering of nodes; store in clustered B-tree indes –Use only odd numbers to allow insertions –Reverse PATH-ID encoding, for efficient processing of postfix expressions like //a/b/c –Add more indexes, e.g. on data values
92 Bad Bugs Nobody loves bad bugs. Tree Frogs All right-thinking people love tree frogs.
93
94 Infoset Table
95 = “ ”]/SECTION SELECT SerializeXML (N2.ID, N2.ORDPATH) FROM infosettab N1 JOIN infosettab N2 ON (N1.ID = N2.ID) WHERE N1.PATH_ID = AND N1.VALUE = ' ' AND N2.PATH_ID = PATH_ID(BOOK/SECTION) AND Parent (N1.ORDPATH) = Parent (N2.ORDPATH) SELECT SerializeXML (N2.ID, N2.ORDPATH) FROM infosettab N1 JOIN infosettab N2 ON (N1.ID = N2.ID) WHERE N1.PATH_ID = AND N1.VALUE = ' ' AND N2.PATH_ID = PATH_ID(BOOK/SECTION) AND Parent (N1.ORDPATH) = Parent (N2.ORDPATH)