XMLTA Transparency No. 1 Query XML Documents with XQuery Cheng-Chia Chen

XMLTA Transparency No. 2 Objectives How XML generalizes relational databases The XQuery language How XML may be supported in databases

XMLTA Transparency No. 3 XQuery 1.0 XML documents naturally generalize database relations XQuery is the corresponding generalization of SQL

XMLTA Transparency No. 4 Queries on XML documents XML documents generalize relational data: A database is composed of many tables A table (relation) is composed of tuples A tuple (record; rows) is composed of attributes An attribute (field; column) contains a primitve data (, which may serves as a key/key reference).

XMLTA Transparency No. 5 From Relations to Trees

XMLTA Transparency No. 6 Only Some Trees are Relations A relation is a tree of height two with an unbounded number of children (rows) all of which have the same fixed number of child nodes (columns) The database community has been looking for a richer data model than relations. Hierarchical, object-oriented, or multi-dimensional databases have emerged, but neither has reached consensus. A DTD for RDB :

XMLTA Transparency No. 7 Trees Are Not Relations Not all XML Document trees satisfy the previous characterization Trees are ordered, while both rows and columns of tables may be permuted without changing the meaning of the data Trees may have height > 2. Trees is in general not homogeneous.  Elements of the same type may have different number and types of children. This does not mean that we cannot store these documents in traditional data base but only means that a more complex encoding of xml documents into data base tables is required.

XMLTA Transparency No. 8 An examlpe student grade records Joe Average 21 Biology Jack Doe 18 Physics XML Science

XMLTA Transparency No. 9 A Corresponding Student Database key point of data base design: decompose general xml tree into flat data base tables

XMLTA Transparency No. 10 For Relational Data base, we have well-established theory and practice, but can we have the same achievement for XML ? database XML DDL(DataDefintionLang) XML Schema DQL(DataQueryLang;SQL) XQuery How should query languages like SQL be similarly generalized?

XMLTA Transparency No. 11 XQuery Design Requirements Must have at least one XML syntax and at least one human-readable syntax Must be declarative Must be namespace aware Must coordinate with XML Schema Must support simple and complex datatypes Must be able to combine information from multiple documents Must be able to transform and create XML trees

XMLTA Transparency No. 12 The XQuery language Developed by W3C RecommendationRecommendation. Derived from several previous proposals: XML-QL XML-QL,YATL,Lorel,QuiltYATLLorelQuilt which all agree on the fundamental principles. XQuery relies on XPath and XML Schema datatypes. There have been many commercial/non-commerical implementations released.commercial/non-commerical implementations ExistExist : AltovaXMLAltovaXML (free) … Two formats XQuery : plaintext syntax XQueryX : XML syntax.

XMLTA Transparency No. 13 Relationship to XPath XQuery 1.0 is a strict superset of XPath 2.0 Every XPath 2.0 expression is directly an XQuery 1.0 expression (a query) The extra expressive power is the ability to join information from different sources and generate new XML fragments

XMLTA Transparency No. 14 Relationship to XSLT XQuery and XSLT are both domain-specific languages (DSL) for combining and transforming XML data from multiple sources They are vastly different in design, partly for historical reasons XQuery is designed from scratch, XSLT is an intellectual descendant of CSS (cascaded StyleSheet) – about how to render a document. Technically, they may emulate each other. XQuery Grammar

XMLTA Transparency No. 15 XQuery concepts A query in XQuery is an expression that: reads a sequence of XML nodes or atomic values returns a sequence of XML nodes or atomic values The principal forms of XQuery expressions are: path expressions element constructors FLWOR ("flower") expressions list expressions (for,let) conditional expressions (where ) order by (o) return expr(r) Xpath constructor: quantified expressions (some/every) datatype expressions (xs:int(“123”)); {…}

XMLTA Transparency No. 16 XQuery Modules and Prologs An XQuery is a main module: Module ::= VersionDecl? ( MainModule | LibraryModule)VersionDecl VersionDecl:  xquery version "1.0" encoding "UTF-8" ;  (: this is versiondecl; encoding is optional :) a module may import other library modules. MainModule ::= Prolog QueryBobyMainModule LibraryModule ::= ModuleDecl PrologLibraryModuleModuleDecl  Ex: module namespace math = " Like XPath expressions, XQuery expressions are evaluated relatively to a context This is explicitly provided by a prolog Settings define various parameters for the XQuery processor language, such as: declare bounndary-space (preserve | strip );

XMLTA Transparency No. 17 More From the Prolog PrologProlog ::= (: Prolog consists of two parts. :) ( ( DefaultNamespaceDecl // declare default function or element/type NSsDefaultNamespaceDecl | Setter // settings for XQuery processingSetter | NamespaceDecl // declare prefix-URI bindingsNamespaceDecl | Import // import schema or other modulesImport ) Separator // each ends with ‘;’Separator )* ( ( (: Second part contains variable decl, function decl and options :) VarDecl VarDecl // global variable declarations | FunctionDecl // function declarationsFunctionDecl | OptionDecl // implementation-defined options declarationOptionDecl ) Separator // each ends with ';'Separator )* optionDecl ::= declare option QName “aLiteral”

XMLTA Transparency No. 18 Namespace Declarations Format: // declare prefix-namexpace binding declare namespace prefix = ”aURI” ; Ex: declare namespace bk = ” ; Predeclared namespaces prefixes: (: for xml:lang, xml:space, etc :) declare namespace xml = " (: for xml schema and schema instance :) xs = " xsi = " (: for xpath fucntion and data types ; this is the default function namespace:) fn = " (: for xquery fucntions :) local=" only xml prefix cannot be redefined.

XMLTA Transparency No. 19 Setter [7] Setter ::= BoundarySpaceDecl | // preserve space b/t elementsSetterBoundarySpaceDecl declare boundary-space (preserve | strip); DefaultCollationDeclDefaultCollationDecl | // default collation for ordering declare default collation "aURILiteral"URILiteral BaseURIDeclBaseURIDecl | declare base-uri " ConstructionDeclConstructionDecl | // preserve type of copied constructs? declare construction (strip | preserve); OrderingModeDeclOrderingModeDecl | // ordered or not for return values declare ordering (ordered | unordered) ; EmptyOrderDeclEmptyOrderDecl | //empty or NaN as least/greatest key declare default order empty (greatest | least); CopyNamespacesDeclCopyNamespacesDecl // copy prefix-NS bindings  see (here. and ( e.2.D) for details)here e.2.D  cf: constructed node ;copied nodes; original elements  ex: {.//book}

XMLTA Transparency No. 20 Default namespace declarations Default namespace Defautl element/type namespace Default function namespace Ex: (: can be disabled by ”” for element (and type as well) :) declare default element namespace ” ; declare default function namespace “aURI”; (: default is " if not specifiedhttp:// or disabled by "" So f1(.) has the same effect as fn:f1(.) :) Unprefixed attribute names and variable names are in no namespace.

XMLTA Transparency No. 21 Import declarations // import element/type/attribute declared in a schema // may also declare prefix/default namespace for the target namespace // default namespace uri can be "" for importing noNamespaceSchema import schema import (library) module Ex: import schema "targetNamespaceURI" ( at "schema_loc_uri1", "schema_loc_uri2" )? ; import schema namespace prefix = "targetNamespaceURI" (at "loc1" )?; import schema default element namespace "targetNamespaceURI" ( at "loc_uri" )? ; Ex: // import a schema and use it as the default element/type namespace import schema default element namespace " // import a schema which has no target namespace and use it as default namespace. import schema default element namespace "" at "

XMLTA Transparency No. 22 Module imports Import declarations // import function/global variables declared in an XQ library module import (library) moduleimport (library) module // may also declare its namespace prefix Notes: Only variables and functions are imported; Hence in scope schema definition and statically known namespaces are not imported. Module imports are not transitive: A imports B and B imports C do not imply A imports C. Ex: import module " "; // can give hints about module locations import module " (at " "book2.xq” )? ; // can also bind a prefix for imported module namespace uri. import module namespace book=" ( at "book2.xq” )? ; import module namespace math = "

XMLTA Transparency No. 23 Variable and function declarations VarDecl ::= VarDecl declare variable $QName TypeDeclaration? ((":=" ExprSingle) | "external")QNameTypeDeclaration ExprSingle declare variable $math:pi xs:decimal opt := ; declare variable $math:e xs:double := 2.71E0 declare variable $ext:var1 external ; FunctionDecl ::= FunctionDecl declare function QName ( ParamList? ) (as SequenceType)? (EnclosedExpr | external)QNameParamListSequenceTypeEnclosedExpr

XMLTA Transparency No. 24 Ex : declare function local:summary// name ( $emps as element(employee)* ) // parameters as element(dept) * // return type { //body for $d in fn:distinct-values($emps/deptno) //a FLWOR expression let $e := $emps[deptno = $d] return {$d} {fn:count($e)} {fn:sum($e/salary)} }; Direct element construction!

XMLTA Transparency No. 25 Example Suppose we have the file at loc "employee.xml" dept2 Wang dept1 Lee dept3 Chen dept1 Cheng The following query sho how to use the function summary xquery version "1.0" encoding "UTF-8" ; (: Definition of fun summary inserted here or import it here :) let $emps := doc("employees.xml")/employees/employee return {summary($emps)} Result: dept dept dept

XMLTA Transparency No. 26 Xquery Conecpts QueryBody = Expr [31] Expr ::= ExprSingle ("," ExprSingle)*ExprExprSingle [32] ExprSingle ::=ExprSingle FLWORExpr | QuantifiedExpr xpath20FLWORExprQuantifiedExpr | TypeswitchExpr | IfExpr xpath20 | OrExpr xpath20+constructionTypeswitchExprIfExprOrExpr Expressions are evaluated relative to a expression context:expression context namespaces variables functions date and time context item,context position, context size …

XMLTA Transparency No. 27 XPath Expressions XPath expressions are also legal XQuery expressions same data model : sequence of items items : nodes or atomic data The XQuery prolog gives the required static context namespace, function library, variable bindings The initial context node, position, and size are undefined Since XQuery is intended to operate on multple docuemnts. initial context info can be obtained by starting from fn:doc()

XMLTA Transparency No. 28 Datatype Expressions Same atomic values as XPath 2.0 Also lots of primitive simple values: xs:string("XML is fun") xs:boolean("true") xs:decimal("3.1415") xs:float(" E23") xs:dateTime(" T13:20:00-05:00") xs:time("13:20:00-05:00") xs:date(" ") xs:gYearMonth(" ") xs:gYear("1999") xs:hexBinary("48656c6c6f0a") xs:base64Binary("SGVsbG8K") xs:anyURI(" xs:QName("rcp:recipe") Functions for theses types can be found in fn:XXX().

XMLTA Transparency No. 29 XML Expressions XQuery expressions may produce new XML nodes, which do not exist in input documents. Expressions may denote element, character data, comment, and processing instruction nodes Each node is created with a unique node identity Constructors may be either direct (called literal result element in XSLT) or computed Both may be nested with each other.

XMLTA Transparency No. 30 Uses the standard XML syntax Ex: The expression baz evaluates to the given XML fragment Note that the expression (containing two direct elements) is evaluates to false why ? ( unique identity of each node ) Direct Constructors

XMLTA Transparency No. 31 Namespaces in Constructors (1/3) The following four constructions have the same effect. declare default element namespace " John Doe CEO, Widget Inc. (202)

XMLTA Transparency No. 32 Namespaces in Constructors (2/3) declare namespace b = " John Doe CEO, Widget Inc. (202)

XMLTA Transparency No. 33 Namespaces in Constructors (3/3) John Doe CEO, Widget Inc. (202) John Doe CEO, Widget Inc. (202)

XMLTA Transparency No. 34 Enclosed Expressions XQ also allow us to embed some XQ expressions into direct elements : Such expression is called attribuate value template in XSLT. but can be appled to atribute value as well as element contents {1, 2, 3, 4, 5} {1, "2", 3, 4, 5} {1 to 5} 1 {1+1} {" "} {"3"} {" "} {4 to 5} ‘{‘ and '}' now have special meaning and must be escaped using { ( or {{ ) and } (or }} ).

XMLTA Transparency No. 35 Direct Constructors v.s Computed constructors Direct construction in XML syntax : Direct construction John Doe CEO, Widget Inc. (202) Non-XML alternative syntax (Computed Constructors)Computed Constructors syntax declare default element namespace ; element card { John Doe, (: may use direct contructor :) element title { text { "CEO, Widget Inc." } }, element { text { } }, element phone { text { "(202) " } }, element logo { attribute uri { "widget.gif" } }}

XMLTA Transparency No. 36 Computed QNames Names of elements/attributes may need to be computed! Syntax: nodeType (QName| {NameExpr})? { ContentExpr } nodeType : element | attribute | processing-instruction – need a name | document | comment | text -- no name declare default element namespace " element { lower-case("CARD") } { element { "name" } { text { "John Doe" } }, element { "title" } { text { "CEO, Widget Inc." } }, element { " " } { text { } }, element { "phone" } { text { "(202) " } }, element { "logo" } { attribute { "uri" } { "widget.gif" }, "aLogo" } }

XMLTA Transparency No. 37 Biliingual Business Cards xquery version "1.0" encoding "big5" ; declare default element namespace "uri:businesscard.org"; declare variable $lang := "zh_TW" ; element { if ($lang="zh_TW") then " 名片 " else "card" } { element { if ($lang="zh_TW") then " 姓名 " else "name" }{ "John Doe" }, element { if ($lang="zh_TW") then " 頭銜 " else "title" }{ "CEO, Widget Inc." }, element { " " }{ }, element { if ($lang="zh_TW") then " 電話 " else "phone" } {"(202) " }, element logo { attribute { "uri" } { "widget.gif" }} }

XMLTA Transparency No. 38 FLWOR Expressions Used for general queries: Find the names of all students which have more than one major and order them by student id. { for $s in fn:doc("students.xml")//student let $m := $s/major where fn:count($m) ge 2 order by return { $s/name/text() } }

XMLTA Transparency No. 39 The Difference Between For and Let (1/4) for $x in (1, 2, 3, 4) let $y := ("a", "b", "c") return ($x, $y) 1, a, b, c, 2, a, b, c, 3, a, b, c, 4, a, b, c

XMLTA Transparency No. 40 The Difference Between For and Let (2/4) let $x in (1, 2, 3, 4) for $y := ("a", "b", "c") return ($x, $y) 1, 2, 3, 4, a, 1, 2, 3, 4, b, 1, 2, 3, 4, c

XMLTA Transparency No. 41 The Difference Between For and Let (3/4) for $x in (1, 2, 3, 4) for $y in ("a", "b", "c") return ($x, $y) 1, a, 1, b, 1, c, 2, a, 2, b, 2, c, 3, a, 3, b, 3, c, 4, a, 4, b, 4, c

XMLTA Transparency No. 42 The Difference Between For and Let (4/4) let $x := (1, 2, 3, 4) let $y := ("a", "b", "c") return ($x, $y) 1, 2, 3, 4, a, b, c

XMLTA Transparency No. 43 Computing Joins Find distinct names of recipses that make use of at least one stuff in a refrigerator? declare namespace rcp = " fn:distinct-values( for $r in fn:doc("recipes.xml")//rcp:recipe, $i in $s in fn:doc("frige.xml")//stuff[text()=$i] return $r/rcp:title/text() ) eggs olive oil ketchup unrecognizable moldy thing

XMLTA Transparency No. 44 Computing Joins Find distinct names of recipses that make use of at least one stuff in a refrigerator? more efficient version: declare namespace rcp = " for $r in fn:doc("recipes.xml")//rcp:recipe let $names := name let $stuffs := fn:doc("fridge.xml")//stuff/text() (: may be put at 2 nd line:) where $names = $s (: general comparison :) return $r/rcp:title/text()

XMLTA Transparency No. 45 Inverting a Relation From Recipe  ingredients to ingredients  recipes declare namespace rcp = " { for $i in distinct-values( (: for each ingredient i :) (: find all ingredients :) ) return { for $r in fn:doc("recipes.xml")//rcp:recipe (: for each receipe :) where (: which uses ingrediwent i :) return $r/rcp:title/text() } }

XMLTA Transparency No. 46 Sorting the Results declare namespace rcp = " { for $i in distinct-values( ) order by $i return { for $r in fn:doc("recipes.xml")//rcp:recipe where order by $r/rcp:title/text() return $r/rcp:title/text() } }

XMLTA Transparency No. 47 A More Complicated Sorting for $s in document("students.xml")//student order by descending, fn:count($s/major) descending, xs:integer($s/age/text()) ascending return $s/name/text()

XMLTA Transparency No. 48 Using Functions declare function local:grade($g as xs:string ) { if ($g="A") then 4.0 else if ($g="A-") then 3.7 else if ($g="B+") then 3.3 else if ($g="B") then 3.0 else if ($g="B-") then 2.7 else if ($g="C+") then 2.3 else if ($g="C") then 2.0 else if ($g="C-") then 1.7 else if ($g="D+") then 1.3 else if ($g="D") then 1.0 else if ($g="D-") then 0.7 else 0 }; declare function local:gpa($s as element(student) ) { fn:avg( for $g in return local:grade($g) ) }; { for $s in fn:doc("students.xml")//student return }

XMLTA Transparency No. 49 A Height Function Find the height of an node tree. can use recursion ! declare function local:height($x as node() ) { if (fn:empty($x/*)) then 0 (: an element without child element has height 0:) else fn:max(for $y in $x/* return local:height($y) )+1 } ;

XMLTA Transparency No. 50 A Textual Outline intended textual outline of a recipe unusual in that it generate plain text instead of xml output. Cailles en Sarcophages //recipe title pastry // ingredient chilled unsalted butter // sub ingredient flour salt ice water filling baked chicken marinated chicken small chickens, cut up Herbes de Provence dry white wine orange juice minced garlic truffle oil...

XMLTA Transparency No. 51 Computing Textual Outlines declare namespace rcp = " declare function local:ingredients($i,$p){ fn:string-join( (:$p is ident spaces :) for $j in $i/rcp:ingredient return fn:string-join( ($p, local:ingredients($j, fn:concat($p," "))), ""), "") }; declare function local:recipes($r) { fn:concat( $r/rcp:title/text(), " ", local:ingredients($r," ") ) }; (: Query body :) for $r in fn:doc(" return local:recipes($r)

XMLTA Transparency No. 52 Sequence Types 2 instance of xs:integer 2 instance of item() 2 instance of xs:integer? () instance of empty() () instance of xs:integer* (1,2,3,4) instance of xs:integer* (1,2,3,4) instance of xs:integer+ instance of item() instance of node() instance of element() instance of element(foo) instance of attribute() instance of attribute(bar) fn:doc("recipes.xml")//rcp:ingredient instance of element()+ fn:doc("recipes.xml")//rcp:ingredient instance of element(rcp:ingredient)+

XMLTA Transparency No. 53 An Untyped Function declare function local:grade($g) { if ($g="A") then 4.0 else if ($g="A-") then 3.7 else if ($g="B+") then 3.3 else if ($g="B") then 3.0 else if ($g="B-") then 2.7 else if ($g="C+") then 2.3 else if ($g="C") then 2.0 else if ($g="C-") then 1.7 else if ($g="D+") then 1.3 else if ($g="D") then 1.0 else if ($g="D-") then 0.7 else 0 };

XMLTA Transparency No. 54 A Default Typed Function declare function local:grade($g as item()*) as item()* { if ($g="A") then 4.0 else if ($g="A-") then 3.7 else if ($g="B+") then 3.3 else if ($g="B") then 3.0 else if ($g="B-") then 2.7 else if ($g="C+") then 2.3 else if ($g="C") then 2.0 else if ($g="C-") then 1.7 else if ($g="D+") then 1.3 else if ($g="D") then 1.0 else if ($g="D-") then 0.7 else 0 };

XMLTA Transparency No. 55 A Precisely Typed Function declare function local:grade($g as xs:string) as xs:decimal { if ($g="A") then 4.0 else if ($g="A-") then 3.7 else if ($g="B+") then 3.3 else if ($g="B") then 3.0 else if ($g="B-") then 2.7 else if ($g="C+") then 2.3 else if ($g="C") then 2.0 else if ($g="C-") then 1.7 else if ($g="D+") then 1.3 else if ($g="D") then 1.0 else if ($g="D-") then 0.7 else 0 };

XMLTA Transparency No. 56 Another Typed Function declare function local:grades($s as element(students)) as attribute(grade)* { };

XMLTA Transparency No. 57 Runtime Type Checks Type annotations are checked during runtime A runtime type error is provoked when an actual argument value does not match the declared type a function result value does not match the declared type a valued assigned to a variable does not match the declared type

XMLTA Transparency No. 58 Built-In Functions Have Signatures fn:contains($x as xs:string?, $y as xs:string?) as xs:boolean op:union($x as node()*, $y as node()*) as node()*

XMLTA Transparency No. 59 XQueryX for $t in fn:doc("recipes.xml")/rcp:collection/rcp:recipe/rcp:title return $t <xqx:module xmlns:xqx=" xmlns:xsi=" xsi:schemaLocation=" xqueryx.xsd"> t doc recipes.xml child rcp:collection child rcp:recipe child xqx:nodeName> rcp:title t

XMLTA Transparency No. 60 XML Databases How can XML and databases be merged? Several different approaches: extract XML views of relations use SQL to generate XML shred XML into relational databases

XMLTA Transparency No. 61 The Student Database Again

XMLTA Transparency No. 62 Automatic XML Views (1/2) Columns as attributes

XMLTA Transparency No. 63 Automatic XML Views (2/2) Column as elements Joe Average Jack Doe 18

XMLTA Transparency No. 64 Programmable Views wrap query results by xml elements SQL/XML ( SQL/XML An extension of SQL allowing constructin of XML data as a resultof SQL queries. xmlelement(name "Students", select xmlelement(name "record", xmlattributes(s.id, s.name, s.age) ) from Students s ) xmlelement(name "Students", select xmlelement(name "record", xmlforest(s.id, s.name, s.age) ) from Students s )

XMLTA Transparency No. 65 XML Shredding Shredding an XML documents into multiple data base Tables Each element type is represented by a relation Each element node is assigned a unique key in document order so siblings can be ordered. Each element node contains the key of its parent so parent –child relations can be maintained The possible attributes are represented as fields, where absent attributes have the null value Contents consisting of a single character data node (#PCDATA only) is inlined as a field.

XMLTA Transparency No. 66 From XQuery to SQL Any XML document can be faithfully represented This takes advantage of the existing database implementation Queries must now be phrased in ordinary SQL rather than XQuery But an automatic translation is possible select ingredient.amount from ingredient where ingredient.name="butter"

XMLTA Transparency No. 67 Summary XML trees generalize relational tables XQuery similarly generalizes SQL XQuery and XSLT have roughly the same expressive power But they are suited for different application domains: data-centric vs. document-centric

XMLTA Transparency No. 68 Essential Online Resources implementations: AltovaXML