1. XML Publishing Introduction General approach XPERRANTO SilkRoute Microsoft SQL 2000 Summary

2. Introduction What is XML Publishing? XML Publishing is the task of transforming the relational data into XML, for the purpose of exchange over the Internet. More specifically, publishing XML data involves joining tables, selecting and projecting the data that needs to be exported, creating XML hierarchies; and processing values in an application specific manner.

3. Introduction Why need XML Publishing? - most business data are stored in relational database systems. - XML is a standard for exchanging business data on the web. - it’s a simply, platform independent, unicode based syntax for which simple and efficient parsers are widely available. - it can not only represent structured data, but also provide an uniform syntax for semi-structured data and marked-up content.

4. Introduction Two data model: Relational data - fragmented into many flat relations - normalized - proprietary XML data - nested - un-normalized - public (450 schemas at www.biztalk.org)

5. General Approach Create XML views over Relational Data, each of these XML views can provide an alternative, application-specific view of the underlying relational data. Through these XML views, business partners can access existing relational data as though it was in some industry-standard XML format.

6. Virtual vs. Materialize Materialized XML Publishing Materialize the entire XML view on request and return the resulting XML document. Virtual XML Publishing Support queries over XML views, return what user applications actually want.

7. Virtual vs. Materialize Materialized XML Publishing - applications can access all the data without interfering with the relational engine - XML view need to be refreshed periodically - inefficient in some cases Virtual XML Publishing - guarantee data freshness - leverage the processing power of relational engines - translation of an XML query of an XML view into SQL may be complex

8. Middleware System Interface between Relational Database and User Application - defines and manages XML views - translates incoming XML queries into SQL and submits them to the database system - receives the queries’ results, then translates them back into XML terms.

9. Applications Web/Intranet XML Query Processor XML Views Manager XML Tagger RDBMS Middleware System Figure 1 A high-level architecture of middleware system View Definition View Description Result XML Documents User XML Queries SQL QueriesTuples Streams

10. XPERRANTO vs. SilkRoute IBM XPERRANTO - pure XML, single query language approach. XML views are defined by XML query language which is using the type system of XML schema. SilkRoute - XML views are defined using a declarative query language called RXL (Relational to XML Transformation Language).

11. XPERRANTO vs. SilkRoute XPERRANTO - user only need be familiar with XML - both relation data and meta-data can be represented and queried in the same framework - can publish object-relational structures - pushes all relational logic down to database engine

12. Query Translation XML View Services XML Tagger Figure 2 XPERRANTO Architecture View Definition View Description SQL Queries Data Tuples XML-QL Parser Query Rewrite SQL Translation XQGM XML Schema Generator XML Result O-R Database SQL Query Processor Stored Tables System Catalog Catalog Info. XML Schema

13. Example 1: Relational Schema vs. XML View Schema DDL (Data Definition Language) for O-R Schema in SQL99 Terms 1.Create Table Book AS (bookID CHAR(30), name VARCHAR(225), publisher VARCHAR(30)) 2.Create Table publisher AS (name VARCHAR(30), address VARCHAR(255)) 3.Create Type author_type AS (bookID CHAR(30), first VARCHAR(30), last VARCHAR(30)) 4.Create Table author OF author_type (REF IS ssn USER GENERATED)

14. XML View Schema over Example O-R database Create Type author_type AS... Create Table book AS... Create Table author OF...

15. Default XML View over Example O-R database … … … … … … … … … …similar to and

16. Example 2: From XQuery to SQL XPERRANTO Query Engine XQuery Parser Query Rewrite & View Composition Computational Pushdown XQuery RDBMS SQL Query Tagger Runtime Tuples Query Result XQGM Tagger Graph

17. A Purchase Order Database and its Default View id Smith Construction10 9 custnum custname Western Builders 7734 7725 order oidcostdesc 10 generator backhoe 8000 12000 item oidamtdue 10 1/10/01 6/10/01 20000 12000 payment 10 Smith Construction 7734 9 Western Builders 7725 10 generator 8000 10 backhoe 24000 …similar to and

18. XML Purchase Order Smith Construction 8000 24000 20000 12000 … 01. create view orders as ( 02. for $order in view (“default”)/order/row 03. return 04. 05. $order/custname 06. 07. for $item in view(“default”)/item/row 08. where $order/id=$item/oid 09. return 10. 11. $item/cost 12. 13. 14. 15. for $payment in view(“default”)/payment/row 16. where $order/id=$payment/oid 17. return 18. 19. $payment/amount 20. 21. sortby(@due) 22. 23. ) User-defined XML “ orders ” view

19. 1. for $order in view(“orders”) 2. where $order/customer/text() like “Smith%” 3. return $order XQuery over “ orders ” view XQuery Parser

20. Query Parsing XQGM (XML Query Graph Model) - extension of a SQL internal query representation called Query Graph Model (QGM). - consists of a set of operators and functions that are designed to capture the semantics of an XML query.

21. OPERATORDESCRIPTION TableRepresents a table in a relational database ProjectComputes results based on its input SelectRestricts its input JoinJoin two or more inputs GroupbyApplies aggregate functions and grouping OrderbySorts input based on column values UnionUnions two or more inputs UnnestApplies super-scalar functions to input ViewRepresents a view FunctionRepresents an Xquery function XML FUNCTIONDESCRIPTIONOPERATORS 1cr8Elem(Tag, Atts, Clist)Creates an element with tag name Tag, attribute list Atts, and contents ClistProject 2cr8AttList(A1,…,An)Creates a list of attributes from the attributes passed as parametersProject 3cr8XMLFragList(C1,…,Cn)Creates an XML fragment list from the content (element/text) parametersProject 4aggXMLFrags©Aggregate function that creates an XML fragment list from content inputsGroupby 5getTagName(Elem)Returns the element name of ElemProject, Select 6getAttributes(Elem)Returns the list of attributes of ElemProject, Select 7getAttName(Att)Returns the name of attribute AttProject, Select 8Is Element(E)Returns true if E is an element, returns false otherwiseSelect 9isText(T)Returns true if T is text, returns false otherwiseSelect 10Unnest(List)Superscalar function that unnest a listUnnest Part of the XML Functions and Operators in XQGM

22. table: itemtable: paymenttable: order project: $order= $custname $items $pmts $oid $desc$cost $id $custname $oid$due$amt select: $oid = $id $desc $cost$due$amt project: $item = … project: $pmt = … groupby: $items = aggXMLFrags($item) groupby: orderby (on $due): $pmts = aggXMLFrags($pmt) $due $pmt$item join (correlated): $id $custname $items$pmts $items $pmts $order correlation on order.id view result 12 3 4 5 6 7 8 9 10 11 XQGM for the XML Orders View

23. project: $elems = getContents($order) $elems View: orders $order Unnest: $elem = unnest($elems) $elem select: isElement($elem) and getTagName($elem) = “customer” $elem project: $vals = getContents($elem) $vals Unnest: $val = unnest($vals) $val select: isText($val) and $val like “Smith%” $val join (correlated): $order 1 2 3 4 5 6 7 8 correlation on $order XQGM for the Query over Orders View for $order in view(“order”) where $order/customer/text() like “Smith%” return $order

24. XQGM after the Query Parsing Stage is composed with the views it references (orders view here) and rewrite optimizations are performed to eliminate the construction of intermediate XML fragments and push down predicates. View Composition

25. FUNCTIONCOMPOSES WITHREDUCTION 1 getTagNamecr8Elem(Tag, Atts, Clist)Tag 2 getAttributescr8Elem(Tag, Atts, Clist)Atts 3 getContentscr8Elem(Tag, Atts, Clist)Clist 4 getAttNamecr8Att(Name, Val)Name 5 getAttValuecr8Att(Name, Val)Val 6 isElementcr8Elem(Tag, Atts, Clist)True 7 isElementOther than cr8ElemFalse 8 isTextPCDATATrue 9 isTextOther than PCDATAFalse 10 unnestaggXMLFrags( C )C 11 unnestcr8XMLFragList(C1,…,Cn) C1 ∪ … ∪ Cn 12 unnestcr8AttList(A1,…, An) A1 ∪ … ∪ An Composition Rules

26. table: itemtable: paymenttable: order project: $order= … $oid $desc$cost $id $custname $oid$due$amt select: $oid = $id $desc $cost$due$amt project: $item = … project: $pmt = … groupby: $items = aggXMLFrags($item) groupby: orderby (on $due): $pmts = aggXMLFrags($pmt) $due $pmt$item join (correlated): $id $custname $items$pmts $items $pmts $order correlation on order.id 12 3 4 5 6 7 8 9 10 11 Select: $custname like “Smith%” $custname join (correlated): $order Select: $custname like “Smith%” $custname $id $custnameQueryView 12 13 Predicate pushdown

27. The goal in this phase of query processing is to push all data and memory intensive operations down to the relational engine as an efficient SQL query. Two techniques are available: 1. Query Decorrelation 2. Tagger Pull-up Computation Pushdown

28. Complex expressions in Xquery can be represented using correlations. However, it has been shown in earlier work that executing correlated XML queries over a relational database leads to poor performance, so query de-correlation is a necessary step for efficient XML query execution. Query Decorrelation

29. table: itemtable: paymenttable: order left outer join: $id = $id $oid $desc$cost $id $custname $oid$due$amt join: $oid = $id $desc $cost$due$amt project: $item = … project: $pmt = … Groupby (on $id) : $items = aggXMLFrags($item) groupby: orderby (on $due): $pmts = aggXMLFrags($pmt) $due $pmt$item right outer join: $id = $id $id $custname $items $pmts $order 12 3 4 5 6 7 8 9 12 11 Select: $custname like “Smith%” $custname $id $custname XQGM after Decorrelation 10 project: $order= … $order 13 $id $items$pmts $id

30. This step comes right after the query decorrelation. It separates the tagger and SQL operations before SQL query are generated Relational operations are pushed to the bottom of the graph. SQL statements are generated and sent to the relational engine for execution. XML construction functions are pulled up to the top of the query graph and transformed into a “tagger run-time” graph, which produces the result XML documents. Tagger Pull-up

31. input: $id $custname input: $oid = $id $desc $cost$due$amt merge: $item = … merge: $pmt = … aggregate: $items = aggXMLFrags($item) aggregate:: $pmts = aggXMLFrags($pmt) $pmt$item Merge: $order= … $items$pmts $order correlation on id 1 2 3 4 5 6 7 8 XQGM after Tagger Pull-up select p.oid, i.desc, i.cost from item i, order o where o.custname like ‘Smith%’ and i.oid = o.id order by o.id select o.id, o.custname from order o where o.custname like ‘Smith%’ order by o.id select p.oid, p.due, p.amt from payment p, order o where o.custname like ‘Smith%’ and p.oid = o.id order by o.id, p.due

32. SilkRoute Approach

33. Applications Web/Intranet Plan Generator XML Tagger RDBMS SilkRoute’s Architecture Query RXL Result XML Documents User XML Queries SQL QueriesTuples Streams SilkRoute Query Composer XML Template Source Description XML Virtual View Or Materialized View RXL

34. SilkRoute Approach Database administrator starts by writing an RXL query that defines the XML view of the database. It is called the view query. A materialized view is fed directly into the Plan Generator, which generates a set of SQL queries and one XML template. A virtual view is first composed by the Query Composer with a user query resulting another RXL query which then is fed into Plan Generator. SQL queries are sent to the RDMS server, which returns one sorted tuple stream per SQL query XML Tagger merges the tuple streams and produces the XML document, which is returned to the application.

35. Query Composer This component takes a user XML-QL query and composes it with the RXL view query resulting a new RXL query. It combines fragments of the view query and user query. Works the similar way that the Query Parser and Query Rewrite components in XPERRANTO do.

36. Plan Generator This component in SilkRoute uses a greedy optimization algorithm to choose an optimal set of SQL queries for a given RXL view definition. The algorithm bases its decisions on query cost estimations provided by the relational engine and can return more than one plan, which will be integrated with additional optimization algorithms that optimize specific parameters, such as network traffic or server load. Details of the greedy algorithm can be found in: Efficient evaluation of XML middle-ware queries. M. Fernandez etc.

37. XML Publishing : SQL Server Two approaches SQL-centric approach extend the function of SQL queries to realize the transformation. The extended version of SQL query is called “FOR XML”. Virtual XML views approach use XDR (XML-based XML-Data Reduced) schema language to define virtual XML views over relation database, then do querying with XPath.

38. XML Publishing : SQL Server SQL-centric approach Three modes RAW mode Auto Mode Explicit Mode

39. XML Publishing : SQL Server, RAW Mode SELECT CustomerID, OrderID FROM Customer LEFT OUTER JOIN ORDERS ON Customers.CustomerID = Orders.CustomerID For XML Raw SELECT CustomerID, OrderID FROM Customer LEFT OUTER JOIN ORDERS ON Customers.CustomerID = Orders.CustomerID For XML Raw.... flat XML default tag and attribute names

40. XML Publishing : SQL Server Auto Mode SELECT Customers.CustomerID, OrderID FROM Customer LEFT OUTER JOIN ORDERS ON Customers.CustomerID = Orders.CustomerID ORDER BY Customers.OrderID For XML Auto SELECT Customers.CustomerID, OrderID FROM Customer LEFT OUTER JOIN ORDERS ON Customers.CustomerID = Orders.CustomerID ORDER BY Customers.OrderID For XML Auto.... default tag and attribute names no differently typed sibling elements

41. XML Publishing : SQL Server Explicit Mode Nested XML User defined tags and attributes Idea: write SQL queries with complex column names Ad-hoc, order dependent semantics

42. XML Publishing : SQL Server Virtual XML Views The core mechanism of providing XML views over relation data is the concept of an annotated schema, which consist of a schema description of the XML view and annotations that describe the mapping of the XML schema constructs to the relational schema constructs. Then the XPath query together with the annotated schema is translated into a FOR XML query that only returns the data that is required by the query.

43. Summary IBM XPERRANTO pure XML, single query language approach. XML views are defined by XML query language which is using the type system of XML schema. SilkRoute XML views are defined using a declarative query language called RXL (Relational to XML Transformation Language). Microsoft SQL 2000 Supports queries over XML views, but the support is very limited, because queries are specified using XPath, which is a subset of XQuery.

44. Future Work IBM XPERRANTO - provides support for insertable and updateable XML views - pushes tagging inside the database system SilkRoute - looks for better algorithms for translating of RXL into efficient SQL and minimization of composed RXL views Microsoft SQL 2000 - finds out whether query composition and decomposition is possible for the complete XQuery language or for only a subset of the language