1. SDPL 2001Notes 8.2: XQuery1 8.2 W3C XML Query Language –Thanks for Helena Ahonen-Myka (University of Helsinki) for borrowing her slide originals for this section

2. SDPL 2001Notes 8.2: XQuery2 XML Query language n W3C 15.2.2001: (early) working drafts –XML Query Requirements –A Data model –XML Query Algebra –XQuery: A Query Language for XML n influenced by the work of many research groups and query languages n goal: a query language that is broadly applicable across all types of XML data sources

3. SDPL 2001Notes 8.2: XQuery3 Usage scenarios n Human-readable documents –perform queries on structured documents and collections of documents, such as technical manuals, »to retrieve individual documents, »to generate tables of contents, »to search for information in structures found within a document, or »to generate new documents as the result of a query

4. SDPL 2001Notes 8.2: XQuery4 Usage scenarios n Data-oriented documents –perform queries on the XML representation of database data, object data, or other traditional data sources »to extract data from these sources »to transform data into new XML representations »to integrate data from multiple heterogeneous data sources –the XML representation of data sources may be either physical or virtual: »data may be physically encoded in XML, or an XML representation of the data may be produced

5. SDPL 2001Notes 8.2: XQuery5 Usage scenarios n Mixed-model documents –perform both document-oriented and data-oriented queries on documents with embedded data, such as catalogs, patient health records, employment records n Administrative data –perform queries on configuration files, user profiles, or administrative logs represented in XML n Native XML repositories (databases)

6. SDPL 2001Notes 8.2: XQuery6 Usage scenarios n Filtering streams –perform queries on streams of XML data to process the data (logs of email messages, network packets, stock market data, newswire feeds, EDI) »to filter and route messages represented in XML »to extract data from XML streams »to transform data in XML streams n DOM –perform queries on DOM structures to return sets of nodes that meet the specified criteria

7. SDPL 2001Notes 8.2: XQuery7 Usage scenarios n Multiple syntactic environments –queries may be used in many environments –a query might be embedded in a URL, an XML page, or a JSP or ASP page –represented by a string in a program written in a general-purpose programming language –provided as an argument on the command-line or strandard input

8. SDPL 2001Notes 8.2: XQuery8 Requirements n Query language syntax –the XML Query Language may have more than one syntax binding »one must be convenient for humans to read and write »one must be expressed as XML reflecting the underlying structure of the query n Declarativity –the language must be declarative –it must not enforce a particular evaluation strategy

9. SDPL 2001Notes 8.2: XQuery9 Requirements n Reliance on XML Information Set –the XML Query Data Model relies on information provided by XML and Schema Processors –not allowed to depend on information that is not made available by such processors n Datatypes –the data model must represent both XML 1.0 character data and the simple and complex types of the XML Schema specification n Schema: –taken into acount, but not required for queries

10. SDPL 2001Notes 8.2: XQuery10 Requirements: functionality n Support operations (selection, projection, aggregation, sorting, etc.) on all data types: –Choose a part of the data based on content or structure –Also operations on hierarchy and order of document structures n Structural preservation and transformation: –Preserve the relative hierarchy and sequence of input document structures in the query results –Transform XML structures and create new XML structures n Combination and joining: –Combine related information from different parts of a given document or from multiple documents

11. SDPL 2001Notes 8.2: XQuery11 Requirements: functionality n References: –Queries must be able to traverse intra- and inter-document references n Closure property: –The result of an XML query is also XML (usually not a valid document, but a well-formed document fragment) –The results of a query can be used as input to another query n Extensibility: –The query language should support the use of externally defined functions on all datatypes of the data model

12. SDPL 2001Notes 8.2: XQuery12 XQuery n Design goals: –a small, easily implementable language –queries are concise and easily understood –flexible enough to query a broad spectrum of XML information sources (incl. both databases and documents) –a human-readable query syntax (XML-based query syntax coming later…) n features borrowed from many languages –Quilt, XPath, XQL, XML-QL, SQL, OQL, Lorel,...

13. SDPL 2001Notes 8.2: XQuery13 XQuery vs. another XML activities n Semantics of XQuery are defined in terms of the operators of the XML Query Algebra n type system is the type system of XML Schema n path expressions (for navigating in hierarchic documents) ~ path expressions of XPath

14. SDPL 2001Notes 8.2: XQuery14 XQuery n A query is represented as an expression n expressions can be nested with full generality n the input and output of a query are instances of a data model called the XML Query Data Model ~ the data model of XPath (tree of nodes) –a fragment of a document or a collection of documents may lack a common root and may be modeled as an ordered forest of nodes

15. SDPL 2001Notes 8.2: XQuery15 An instance of the XML Query Model - an ordered forest

16. SDPL 2001Notes 8.2: XQuery16 XQuery expressions n path expressions n element constructors n FLWR (”flower”; for-let-where-return) expressions n expressions involving operators and functions n conditional expressions n quantified expressions n list constructors n expressions that test or modify datatypes

17. SDPL 2001Notes 8.2: XQuery17 Path expressions n Use abbraviated syntax of XPath n the result of a path expression is an ordered list of nodes –each node includes its descendant nodes -> the result is an ordered forest n the top-level nodes in the result are ordered according to their position in the original hierarchy (in top-down, left-right order) n no duplicate nodes

18. SDPL 2001Notes 8.2: XQuery18 Element constructors n Similar to XSLT: n consists of a start tag and an end tag, enclosing an optional list of expressions that provide the content of the element –the start tag may also specify the values of one of more attributes –the name of the tag may be specified either by a constant or a variable n typical use: –nested inside another expression that binds variables that are used in the element constructor

19. SDPL 2001Notes 8.2: XQuery19 Example Generate an emp element containing an empid attribute and nested name and job elements. The values of the attribute and nested elements are specified elsewhere. Generate an emp element containing an empid attribute and nested name and job elements. The values of the attribute and nested elements are specified elsewhere. $n, $n, $j $j </emp>

20. SDPL 2001Notes 8.2: XQuery20 Example Generate an element with a computed name, containing nested elements named description and price Generate an element with a computed name, containing nested elements named description and price <$tagname> $d, $d, $p $p </$tagname>

21. SDPL 2001Notes 8.2: XQuery21 FLWR expressions n Constructed from FOR, LET, WHERE, and RETURN clauses n ~SQL select-from-where n clauses must appear in a specific order –1. FOR/LET, 2. WHERE, 3. RETURN n a FLWR expression binds values to one or more variables and then uses these variables to construct a result (in general, an ordered forest of nodes)

22. SDPL 2001Notes 8.2: XQuery22 Flow of data in a FLWR expression

23. SDPL 2001Notes 8.2: XQuery23 FOR clauses n A FOR clause introduces one or more variables, associating each variable with an expression that returns a list of nodes (e.g. a path expression) n the result of a FOR clause is a list of tuples, each of which contains a binding for each of the variables n each variable in a FOR clause can be though of as iterating over the nodes returned by its respective expression

24. SDPL 2001Notes 8.2: XQuery24 LET clauses n A LET clause is also used to bind one or more variables to one or more expressions n a LET clause binds each variable to the value of its respective expression without iteration n results in binding a single list for each variable n Compare: –FOR $x IN /library/book -> many bindings (books) –LET $x := /library/book -> a single list (of books)

25. SDPL 2001Notes 8.2: XQuery25 FOR/LET clauses n A FLWR expression may contain several FOR and LET clauses –each of these clauses may contain references to variables bound in previous clauses n the result of the FOR/LET sequence: –an ordered list of tuples of bound variables n the number of tuples generated by the FOR/LET sequence: –the product of the cardinalities of the node-lists returned by the expressions in the FOR clauses

26. SDPL 2001Notes 8.2: XQuery26 WHERE clause n Each of the binding tuples generated by the FOR and LET clauses can be filtered by an optional WHERE clause n only those tuples for which the condition in the WHERE clause is true are used to invoke the RETURN clause n the WHERE clause may contain several predicates connected by AND, OR, and NOT –predicates usually contain references to the bound variables

27. SDPL 2001Notes 8.2: XQuery27 WHERE clause n Variables bound by a FOR clause represent a single node  scalar predicates, e.g. $p/color = ”Red”  scalar predicates, e.g. $p/color = ”Red” n Variables bound by a LET clause may represent lists of nodes  list-oriented predicates, e.g. avg($p/price) > 100

28. SDPL 2001Notes 8.2: XQuery28 RETURN clause n The RETURN clause generates the output of the FLWR expression –a node, an ordered forest of nodes, primitive value n is executed on each binding tuple n contains an expression that often contains element constuctors, references to bound variables, and nested subexpressions

29. SDPL 2001Notes 8.2: XQuery29 Examples Assume: a document named ” bib.xml ” Assume: a document named ” bib.xml ” The document consists of a list of book s: + + The document consists of a list of book s: + +

30. SDPL 2001Notes 8.2: XQuery30 List the titles of books published by Morgan Kaufmann in 1998 FOR $b IN document(”bib.xml”)//book WHERE $b/publisher = ”Morgan Kaufmann” AND $b/year = ”1998” RETURN $b/title

31. SDPL 2001Notes 8.2: XQuery31 List each publisher and the average price of its books FOR $p IN distinct(document(”bib.xml”)//publisher) distinct(document(”bib.xml”)//publisher) LET $a := avg(document(”bib.xml”)/book[ avg(document(”bib.xml”)/book[ publisher =$p]/price) RETURN<publisher> $p/text(), $p/text(), $a $a </publisher>

32. SDPL 2001Notes 8.2: XQuery32 List the publishers of more than 100 books: <big_publishers> FOR $p IN FOR $p IN distinct(document(”bib.xml”)//publisher) distinct(document(”bib.xml”)//publisher) LET $b := document(”bib.xml”)/book[ LET $b := document(”bib.xml”)/book[ publisher = $p] WHERE count($b) > 100 WHERE count($b) > 100 RETURN $p RETURN $p</big_publishers>

33. SDPL 2001Notes 8.2: XQuery33 Invert the structure of the input document by grouping book titles by author names: <author_list> FOR $a IN FOR $a IN distinct(document(”bib.xml”)//author) RETURN RETURN $a/text() $a/text() FOR $b IN FOR $b IN document(”bib.xml”)//book[ document(”bib.xml”)//book[ author = $a] RETURN $b/title RETURN $b/title </author_list>

34. SDPL 2001Notes 8.2: XQuery34 For each book whose price is greater than the average, return the title of the book and the amount by which the book’s price exceeds the average price <result> LET $a := LET $a := avg(document(”bib.xml”)//book/price) avg(document(”bib.xml”)//book/price) FOR $b IN document(”bib.xml”)//book FOR $b IN document(”bib.xml”)//book WHERE $b/price > $a WHERE $b/price > $a RETURN RETURN $b/title, $b/title, $b/price - $a $b/price - $a </result>

35. SDPL 2001Notes 8.2: XQuery35 Make an alphabetic list of publishers; within each publisher, list books (title & price) in descending price order FOR $p IN distinct(document(”bib.xml”)//publisher) RETURN $p/text() $p/text() FOR $b IN document(”bib.xml”)/book[ FOR $b IN document(”bib.xml”)/book[ publisher = $p] RETURN RETURN $b/title, $b/title, $b/price $b/price SORTBY(price DESCENDING) SORTBY(price DESCENDING) SORTBY(name) SORTBY(name)

36. SDPL 2001Notes 8.2: XQuery36 Operators in expressions n Expressions can be constructed using infix and prefix operators; nested expressions inside parenthesis can serve as operands n arithmetic and logical operators; collection operators (UNION, INTERSECT, EXCEPT) n BEFORE and AFTER –operate on two lists –return elements in the first list that occur before (after) at least one element of the second list in document order

37. SDPL 2001Notes 8.2: XQuery37 Example Assume: surgical reports with elements procedure, incision, and anesthesia Assume: surgical reports with elements procedure, incision, and anesthesia n Prepare a ”critical sequence” report consisting of all elements that occur between the 1st and 2nd incision in the 1st procedure <critical_sequence> LET $p := //procedure[1] LET $p := //procedure[1] FOR $e IN //* AFTER ($p//incision)[1] FOR $e IN //* AFTER ($p//incision)[1] BEFORE ($p//incision)[2] BEFORE ($p//incision)[2] RETURN $e RETURN $e</critical_sequence>

38. SDPL 2001Notes 8.2: XQuery38 Conditional expressions n IF-THEN-ELSE n conditional expressions can be nested and used wherever a value is expected assume: a library has many holdings (element holding with a type attribute that identifies its type, e.g. book or journal). All holdings have a title; other nested elements depend on the type of holding assume: a library has many holdings (element holding with a type attribute that identifies its type, e.g. book or journal). All holdings have a title; other nested elements depend on the type of holding

39. SDPL 2001Notes 8.2: XQuery39 Make an ordered list of holdings; For journals, include the editor, and for others, the author FOR $h IN //holding RETURN $h/title, $h/title, IF $h/@type = ”Journal” IF $h/@type = ”Journal” THEN $h/editor THEN $h/editor ELSE $h/author ELSE $h/author SORTBY (title) SORTBY (title)

40. SDPL 2001Notes 8.2: XQuery40 Quantifiers n Sometimes necessary to test for existence of some element that satisfies a condition, or to determine whether all elements in some collection satisfy a condition n -> existential and universal quantifiers

41. SDPL 2001Notes 8.2: XQuery41 Find titles of books in which both sailing and windsurfing are mentioned in the same paragraph FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, ”sailing”) AND contains($p, ”windsurfing”) RETURN $b/title

42. SDPL 2001Notes 8.2: XQuery42 Find titles of books in which sailing is mentioned in every paragraph FOR $b IN //book FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES WHERE EVERY $p IN $b//para SATISFIES contains($p, ”sailing”) RETURN $b/title RETURN $b/title

43. SDPL 2001Notes 8.2: XQuery43 Filtering n Function filter(Forest1, Forest2) –in XQuery core function library n returns copies of some of the nodes in Forest1 represented by the first operand –order and hierarchy are preserved n nodes that are copied: –nodes that are present at any level in Forest1 and are also top-level nodes in the Forest2

44. SDPL 2001Notes 8.2: XQuery44 Action of FILTER on a hierarchy n Filter prunes from Forest1 any nodes that are not top-level nodes in Forest2

45. SDPL 2001Notes 8.2: XQuery45 Table of contents for document ”cookbook.xml”, containing nested sections and their title LET $b := document(”cookbook.xml”) RETURN filter($b, $b//section | filter($b, $b//section | $b//section/title | $b//section/title | $b//section/title/text() ) $b//section/title/text() )

46. SDPL 2001Notes 8.2: XQuery46 Other built-in functions n A core library of built-in functions n document: returns the root node of a named document (like in XSLT) n all functions of the XPath core function library n all the aggregation functions of SQL –avg, sum, count, max, min… n distinct: eliminates nodes with duplicate value n empty: returns TRUE if and only if its argument is an empty list

47. SDPL 2001Notes 8.2: XQuery47 User-defined functions n Users are allowed to define own functions n each function definition must –declare the datatypes of its parameters and result –provide an expression that defines how the result of the function is computed from its parameters n when a function is invoked, its arguments must be valid instances of the declared parameter types n the result must also be a valid instance of its declared type

48. SDPL 2001Notes 8.2: XQuery48 Functions n A function may be defined recursively –the function is referenced in its own definition n Example: find the maximum depth of the document named ”partlist.xml” FUNCTION depth(ELEMENT $e) RETURNS xsd:integer { IF empty($e/*) THEN 1 ELSE max(depth($e/*)) + 1 ELSE max(depth($e/*)) + 1} call : depth(document(”partlist.xml”))

49. SDPL 2001Notes 8.2: XQuery49 Querying relational data n A lot of data is stored in relational databases n an XML query language should be able to access this data n Example: suppliers and parts –Table S: supplier numbers (sno) and names (sname) –Table P: part numbers (pno) and descriptions (descrip) –Table SP: relationships between suppliers and the parts they supply, including the price (price) of each part from each supplier

50. SDPL 2001Notes 8.2: XQuery50 One possible XML representation of relational data

51. SDPL 2001Notes 8.2: XQuery51 SQL vs. XQuery n SQL: n XQuery: SELECT pno FROM p WHERE descrip LIKE ’Gear’ ORDER BY pno; FOR $p IN document(”p.xml”)//p_tuple WHERE contains($p/descrip, ”Gear”) RETURN $p/pno SORTBY(.)

52. SDPL 2001Notes 8.2: XQuery52 Grouping n Many relational queries involve forming data into groups and applying some aggregation function such as count or avg to each group n in SQL: GROUP BY and HAVING clauses n Example: Find the part number and average price, for parts that have at least 3 suppliers

53. SDPL 2001Notes 8.2: XQuery53 Grouping: SQL SELECT pno, avg(price) AS avgprice FROM sp GROUP BY pno HAVING count(*) >= 3 ORDER BY pno;

54. SDPL 2001Notes 8.2: XQuery54 Grouping: XQuery FOR $pn IN distinct(document(”sp.xml”)//pno) LET $sp := document(”sp.xml”)//sp_tuple[ pno = $pn] WHERE count($sp) >= 3 RETURN $pn, $pn, avg($sp/price) avg($sp/price) </avgprice) SORTBY(pno)

55. SDPL 2001Notes 8.2: XQuery55 Joins n Example: Return a ”flat” list of supplier names and their part descriptions, in alphabetic order FOR $sp IN document(”sp.xml”)//sp_tuple, $p IN document(”p.xml”)//p_tuple[ $p IN document(”p.xml”)//p_tuple[ pno = $sp/pno], $s IN document(”s.xml”)//s_tuple[ sno = $sp/sno] $s IN document(”s.xml”)//s_tuple[ sno = $sp/sno] RETURN RETURN $s/sname, $s/sname, $p/descrip $p/descrip SORTBY (sname, descrip) SORTBY (sname, descrip)