1. 1 Part 3: Query Languages Managing XML and Semistructured Data

2. 2 In this section…  Lorel (A Lightweight Object REpository Language - developed at Standford) Lorel  XPath specification data model Examples [xpath, axis]xpathaxis syntax  XQuery  FLWR expressions  FOR and LET expressions  Collections and sorting  (XML-QL the earlier version in AT&T Labs)XML-QL  Resources: The Lorel Query Language for Semistructured DataThe Lorel Query Language for Semistructured Data by Abiteboul, Quass, McHugh, Widom, Wiener, in International Journal on Digital Libraries, 1997. A formal semantics of patterns in XSLT A formal semantics of patterns in XSLT by Phil Wadler. XML Path Language (XPath) www.w3.org/TR/xpathwww.w3.org/TR/xpath XQuery: A Query Language for XMLXQuery: A Query Language for XML Chamberlin, Florescu, et al. W3C recommendation: www.w3.org/TR/xquery/

3. 3 Querying XML Data  A core query language (extracting + restructuring)  XPath (core expressions) allows simple navigation through the tree  XQuery is used as the SQL of XML  XSLT (Extensible Stylesheet Language Transformation) = recursive traversal based on pattern matching - will not discuss here

4. 4 Sample Data for Queries … Smith 1999 Database Systems Roux Combalusier 1976 Database Systems … Smith 1999 Database Systems Roux Combalusier 1976 Database Systems

5. 5 Will illustrate with: XML DB = &o1 &o12&o24&o29 &96 &30 paper book author date title author date title biblio &o47&o48 &o50 &o52 &25 Smith1999 Database Systems Roux Combalusier 1976 Database Systems... A Core Query Language A SQL-like language for querying semi-structured data

6. 6 Query 1: SELECT author: X FROM biblio.book.author X &o1 &o12&o24&o29 &96 &30 paper book author date title author date title biblio &o47&o48 &o50 &o52 &25 Smith 1999Database Systems Roux Combalusier 1976 Database Systems... answer author Answer = {author: “Smith”, author: “Roux”, author: “Combalusier”} Answer = {author: “Smith”, author: “Roux”, author: “Combalusier”}

7. 7 Query 2: SELECT row: X FROM biblio._ X WHERE “Smith” in X.author &o1 &o12&o24&o29 &96 &30 paper book author date title author date title biblio &o47&o48 &o50 &o52 &25 Smith 1999Database Systems Roux Combalusier 1976 Database Systems... answer row... Answer = {row: {author:“Smith”, date: 1999, title: “Database…”}, row: … } Answer = {row: {author:“Smith”, date: 1999, title: “Database…”}, row: … }

8. 8 Query 3: SELECT row: ( SELECT author: Y FROM X.author Y) FROM biblio.book X &o1 &o12&o24&o29 &96 &30 paper book author date title author date title biblio &o47&o48 &o50 &o52 &25 Smith 1999Database Systems Roux Combalusier 1976 Database Systems... answer row &a1 &a2 author Answer = {row: {author:“Smith”}, row: {author:“Roux”, author:“Combalusier”,}, }

9. 9 Query 4: SELECT ( SELECT row: {author: Y, title: T} FROM X.author Y, X.title T) FROM biblio.book X WHERE “Roux” in X.author &o1 &o12&o24&o29 &96 &30 paper book author date title author date title biblio &o47&o48 &o50 &o52 &25 Smith 1999Database Systems Roux Combalusier 1976 Database Systems... answer row &a1 &a2 author title Answer = {row: {author:“Roux”, title: “Database…”}, row: {author:“ Combalusier ”, title: “Database…”}, } Answer = {row: {author:“Roux”, title: “Database…”}, row: {author:“ Combalusier ”, title: “Database…”}, } title

10. 10 Lorel  Minor syntactic differences in regular path expressions (% instead of _, # instead of _*)  Common path convention: becomes: SELECT biblio.book.author FROM biblio.book WHERE biblio.book.year = 1999 SELECT X.author FROM biblio.book X WHERE X.year = 1999

11. 11 Lorel  Existential variables: What happens with books having multiple authors ? Author is existentially quantified: SELECT biblio.book.year FROM biblio.book WHERE biblio.book.author = “Roux” SELECT X.year FROM biblio.book X, X.author Y WHERE Y = “Roux”

12. 12 Lorel  Path variables. @P in: What happens on graphs with cycles ?  Constructing new results Several default rules  Casting between datatypes Very useful in practice SELECT @P FROM biblio.# @P X

13. 13 XPath  http://www.w3.org/TR/xpath (11/99) http://www.w3.org/TR/xpath  Building block for other W3C standards: XSL Transformations (XSLT) XML Link (XLink) XML Pointer (XPointer) XML Query  Was originally part of XSL

14. 14 XPath: Summary bibmatches a bib element *matches any element /matches the root element /bibmatches a bib element under root bib/papermatches a paper in bib bib//papermatches a paper in bib, at any depth //papermatches a paper at any depth paper|bookmatches a paper or a book @pricematches a price attribute bib/book/@pricematches price attribute in book, in bib bib/book/[@price<“55”]/author/lastname matches…

15. 15 Example for XPath Queries Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998 Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998

16. 16 Data Model for XPath bib book publisherauthor.. Addison-WesleySerge Abiteboul The root The root element

17. 17 XPath: Simple Expressions Result: 1995 1998 Result: empty (there were no papers) /bib/book/year /bib/paper/year

18. 18 XPath: Restricted Kleene Closure Result: Serge Abiteboul Rick Hull Victor Vianu Jeffrey D. Ullman Result: Rick //author /bib//first-name

19. 19 XPath: Text Nodes Result: Serge Abiteboul Jeffrey D. Ullman !Rick Hull doesn’t appear because he has firstname, lastname Functions in XPath: text() = matches the text value node() = matches any node (= * or @* or text()) name() = returns the name of the current tag /bib/book/author/text()

20. 20 XPath: Wildcard Result: Rick Hull * Matches any element //author/*

21. 21 XPath: Attribute Nodes Result: “55” @price means that price is has to be an attribute /bib/book/@price

22. 22 XPath: Predicates Result: Rick Hull /bib/book/author[firstname]

23. 23 XPath: More Predicates Result: … … /bib/book/author[firstname][address[//zip][city]]/lastname

24. 24 XPath: More Predicates /bib/book[@price < “60”] /bib/book[author/@age < “25”] /bib/book[author/text()]

25. 25 XQuery  Based on Quilt (which is based on XML-QL)Quilt  http://www.w3.org/ TR/xquery/ 2/2001 http://www.w3.org/ TR/xquery/  XML Query data model Ordered ! FLWOR (flower) Expressions FOR... LET... WHERE... ORDER BY… RETURN... FOR... LET... WHERE... ORDER BY… RETURN...

26. 26 XQuery Query: Find all book titles published after 1995: FOR $x IN document("bib.xml")/bib/book WHERE $x/year > 1995 RETURN $x/title FOR $x IN document("bib.xml")/bib/book WHERE $x/year > 1995 RETURN $x/title * bib.xml is shown on slide 15 Result: Principles of Database… Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998 Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998

27. 27 XQuery Query: Find book titles by the coauthors of “Foundations of Databases”: FOR $x IN bib/book[title/text() = “Foundations …”]/author $y IN bib/book[author/text() = $x/text()]/title RETURN $y/text() FOR $x IN bib/book[title/text() = “Foundations …”]/author $y IN bib/book[author/text() = $x/text()]/title RETURN $y/text() Result: Foundations … The answer will contain duplicates !

28. 28 XQuery Same as before, but eliminate duplicates: FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN distinct(bib/book[author/text() = $x/text()]/title) RETURN $y/text() FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN distinct(bib/book[author/text() = $x/text()]/title) RETURN $y/text() Result: Foundations … distinct = a function that eliminates duplicates

29. 29 SQL and XQuery Side-by-side Product(pid, name, maker) Company(cid, name, city) Query: Find all products made in Seattle SELECT x.name FROM Product x, Company y WHERE x.maker=y.cid and y.city=“Seattle” FOR $x IN /db/Product/row $y IN /db/Company/row WHERE $x/maker/text()=$y/cid/text() and $y/city/text() = “Seattle” RETURN $x/name SQL XQuery FOR $y IN /db/Company/row[city/text()=“Seattle”] $x IN /db/Product/row[maker/text()=$y/cid/text()] RETURN $x/name Cool XQuery

30. 30 XQuery: Nesting Query: For each author of a book by Morgan Kaufmann, list all books s/he published: FOR $a IN distinct(document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN { $a, FOR $t IN /bib/book[author=$a]/title RETURN $t } FOR $a IN distinct(document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN { $a, FOR $t IN /bib/book[author=$a]/title RETURN $t } Jones abc def Smith ghi Jones abc def Smith ghi Result:

31. 31 XQuery  FOR $x IN expr -- binds $x to each value in the list expr  LET $x = expr -- binds $x to the entire list expr Useful for common subexpressions and for aggregations FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN $p FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN $p count = a (aggregate) function that returns the number of elms

32. 32 XQuery Query: Find books whose price is larger than average: FOR $a IN /bib/book LET $b:=avg(/bib/book/price/text()) WHERE $a/price/text() > $b RETURN $a FOR $a IN /bib/book LET $b:=avg(/bib/book/price/text()) WHERE $a/price/text() > $b RETURN $a

33. 33 XQuery $b is a collection of elements, not a single element count = a (aggregate) function that returns the number of elements { FOR $p IN distinct(//publisher/text()) LET $b := document("bib.xml")/book[publisher/text() = $p] WHERE count($b) > 100 RETURN $p } { FOR $p IN distinct(//publisher/text()) LET $b := document("bib.xml")/book[publisher/text() = $p] WHERE count($b) > 100 RETURN $p } Query: Find all publishers that published more than 100 books:

34. 34 FOR v.s. LET FOR  Binds node variables  iteration LET  Binds collection variables  one value Examples FOR $x IN document("bib.xml") /bib/book RETURN $x FOR $x IN document("bib.xml") /bib/book RETURN $x Returns:... LET $x := document("bib.xml") /bib/book RETURN $x LET $x := document("bib.xml") /bib/book RETURN $x Returns:...

35. 35 Sorting in XQuery FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN {$b/title, $b/@price} SORTBY(price DESCENDING) } SORTBY(name) FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN {$b/title, $b/@price} SORTBY(price DESCENDING) } SORTBY(name)

36. 36 Sorting in XQuery  Sorting arguments: refer to the name space of the RETURN clause, not the FOR clause  To sort on an element you don’t want to display, first return it, then remove it with an additional query. FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN { $b/title, $b/price } ORDER BY price DESCENDING } ORDER BY name FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text(), FOR $b IN document("bib.xml")//book[publisher = $p] RETURN { $b/title, $b/price } ORDER BY price DESCENDING } ORDER BY name

37. 37 Collections in XQuery  Ordered and unordered collections /bib/book/author = an ordered collection Distinct(/bib/book/author) = an unordered collection  LET $b = /bib/book  $b is a collection  $b/author  a collection (several authors...) RETURN $b/author Returns:...

38. 38 If-Then-Else FOR $h IN //holding RETURN { $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author } ORDER BY title FOR $h IN //holding RETURN { $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author } ORDER BY title

39. 39 Quantifiers FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN $b/title FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN $b/title Existential Quantifiers FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN $b/title FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN $b/title Universal Quantifiers

40. 40 Other Stuff in XQuery  BEFORE and AFTER for dealing with order in the input  FILTER deletes some edges in the result tree  Recursive functions Currently: arbitrary recursion Perhaps more restrictions in the future ?

41. 41 Group-By in XQuery ??  No GROUPBY currently in XQuery  A recent proposal (next) What do YOU think ?

42. 42 Group-By in XQuery ?? FOR $b IN document("http://www.bn.com")/bib/book, $y IN $b/@year WHERE $b/publisher="Morgan Kaufmann" RETURN GROUPBY $y WHERE count($b) > 10 IN $y FOR $b IN document("http://www.bn.com")/bib/book, $y IN $b/@year WHERE $b/publisher="Morgan Kaufmann" RETURN GROUPBY $y WHERE count($b) > 10 IN $y SELECT year FROM Bib WHERE Bib.publisher="Morgan Kaufmann" GROUPBY year HAVING count(*) > 10 SELECT year FROM Bib WHERE Bib.publisher="Morgan Kaufmann" GROUPBY year HAVING count(*) > 10  with GROUPBY Equivalent SQL 

43. 43 Group-By in XQuery ?? FOR $b IN document("http://www.bn.com")/bib/book, $a IN $b/author, $y IN $b/@year RETURN GROUPBY $a, $y IN $a, $y, count($b) FOR $Tup IN distinct(FOR $b IN document("http://www.bn.com")/bib, $a IN $b/author, $y IN $b/@year RETURN $a $y ), $a IN $Tup/a/node(), $y IN $Tup/y/node() LET $b = document("http://www.bn.com")/bib/book[author=$a,@year=$y] RETURN $a, $y, count($b)  with GROUPBY Without GROUPBY 

44. 44 Group-By in XQuery ?? FOR $b IN document("http://www.bn.com")/bib/book, $a IN $b/author, $y IN $b/@year, $t IN $b/title, $p IN $b/publisher RETURN GROUPBY $p, $y IN $p, $y, GROUPBY $a IN $a, GROUPBY $t IN $t  Nested GROUPBY’s

45. 45 XQuery Summary:[Demo]Demo FOR-LET-WHERE-RETURN = FLWR FOR/LET Clauses WHERE Clause RETURN Clause List of tuples of bounded variables List of pruned tuples of bounded variables Instance of XQuery data model