INT-2: XQuery Levels the Data Integration Playing Field Carlo (Minollo) Innocenti DataDirect XML Technologies, Program Manager

© 2007 Progress Software Corporation 2 INT-2: XQuery Levels the Data Integration Playing Field Agenda  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary

© 2007 Progress Software Corporation 3 INT-2: XQuery Levels the Data Integration Playing Field The user submits a request for a report about their stock holdings The web server needs to fetch user’s personal data, stock holdings and live stock data to compile a report to send back to the user A public service offers live (delayed) stock prices A Typical Data Integration Problem Different repositories are used for different parts of the information necessary to create a stock holdings report

© 2007 Progress Software Corporation 4 INT-2: XQuery Levels the Data Integration Playing Field Some implementation constraints… HTML SOAP through AXIS dBASE IV APIs JDBC Java ™ /JSP code accessing the various Java APIs and generating the HTML report Java Open Client or JDBC

© 2007 Progress Software Corporation 5 INT-2: XQuery Levels the Data Integration Playing Field Data Access Layer A dangerous approach

© 2007 Progress Software Corporation 6 INT-2: XQuery Levels the Data Integration Playing Field XQuery The XQuery Vision

© 2007 Progress Software Corporation 7 INT-2: XQuery Levels the Data Integration Playing Field Agenda  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary

© 2007 Progress Software Corporation 8 INT-2: XQuery Levels the Data Integration Playing Field What is XQuery?  W3C Query Language for XML Native XML Programming Language “The SQL for XML” Designed to query, process, and create XML  High level functionality Find anything in an XML structure Querying and combining data Creating XML structures Functions User-defined function libraries

© 2007 Progress Software Corporation 9 INT-2: XQuery Levels the Data Integration Playing Field XQuery Basics  Path Expressions: Finding Data doc("holdings.xml")/holdings/entry  FLWOR Expressions: Querying and Combining Data for $h in doc("holdings.xml")/holdings/holding, $c in doc("companies.xml")/companies/company where $h/userid = "Minollo" and $c/ticker = $h/stockticker return $c/name

© 2007 Progress Software Corporation 10 INT-2: XQuery Levels the Data Integration Playing Field XQuery Basics  Path Expressions, FLWOR Expressions, and XML Constructors for $h in doc("holdings.xml")/holdings/entry, $c in doc("companies.xml")/companies/company where $h/userid = "Minollo" and $c/ticker = $h/stockticker return { $c/companyname } { $c/annualrevenues }

© 2007 Progress Software Corporation 11 INT-2: XQuery Levels the Data Integration Playing Field Functions and Modules  A Function in a Library Module module namespace stock=" declare function stock:companies($user as xs:string) { for $h in doc("holdings")/holdings/entry, $c in doc("companies")/companies/company where $h/userid = $user and $c/ticker = $h/stockticker return … };

© 2007 Progress Software Corporation 12 INT-2: XQuery Levels the Data Integration Playing Field Functions and Modules (2)  Importing and Using a Library Module import module namespace stock=" ; stock:companies("Minollo")

© 2007 Progress Software Corporation 13 INT-2: XQuery Levels the Data Integration Playing Field Why XQuery?  Native Support for XML Conventional programming and query languages are not designed for XML No more parse, navigate, cast, repeat – XML is the native datatype  Designed for Data Integration Native XML and non-XML data can be used the same way Vastly simplifies development when input includes XML, relational, EDI… –Requires support from implementation for the data sources you need

© 2007 Progress Software Corporation 14 INT-2: XQuery Levels the Data Integration Playing Field Why XQuery?  XML Output is Directly Useful XML is becoming the industry standard for data exchange Dynamic Web Sites Publishing Applications Web Messages We normally don’t exchange SQL tables or present them to users!  Programmer Productivity Readable, declarative code – transparent, easier to maintain 7 to 20 times less code than Java + SQL + JDBC + XML APIs

© 2007 Progress Software Corporation 15 INT-2: XQuery Levels the Data Integration Playing Field Why XQuery?  Performance Declarative code – can be optimized by the XQuery Engine Relational database vendors and experts very involved in the design Actually performance depends on the implementation …

© 2007 Progress Software Corporation 16 INT-2: XQuery Levels the Data Integration Playing Field Benefits of XQuery  Data Integration is harder without XQuery! Every data source is different Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…) Mediating among data sources accounts for a lot of code XQuery treats all data sources as XML

© 2007 Progress Software Corporation 17 INT-2: XQuery Levels the Data Integration Playing Field Benefits of XQuery  Processing XML is harder without XQuery! Most programming languages don’t know XML structures Parse, navigate, cast, repeat XML is the native data structure for XQuery  XML Reporting is harder without XQuery! XML input and output may have very complex structure Many different desired XML outputs Data Integration, Native XML Processing are needed XQuery gives full query processing for any XML input and output

© 2007 Progress Software Corporation 18 INT-2: XQuery Levels the Data Integration Playing Field Agenda  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary

© 2007 Progress Software Corporation 19 INT-2: XQuery Levels the Data Integration Playing Field What is XQJ?  XQuery API for Java (XQJ) – JSR 225  “The JDBC for XQuery”

© 2007 Progress Software Corporation 20 INT-2: XQuery Levels the Data Integration Playing Field Benefits of XQJ  Industry Standard, similar to JDBC No need to learn a new proprietary API for each product and each version Can build on existing JDBC knowledge  Lets XQuery fit into any Java architecture  Queries can be created or parameterized at run-time Example: A portfolio for a given user at a given date  Interfaces are designed for use in J2EE applications Example: Results can be retrieved as DOM, SAX, StAX, or text

© 2007 Progress Software Corporation 21 INT-2: XQuery Levels the Data Integration Playing Field Agenda  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary

© 2007 Progress Software Corporation 22 INT-2: XQuery Levels the Data Integration Playing Field An XQuery architecture

© 2007 Progress Software Corporation 23 INT-2: XQuery Levels the Data Integration Playing Field DataDirect XQuery **  High performance  Scalable  Embeddable  Plugs into any Java architecture  Accesses almost any data source  No dependency on servers  Standards-based

© 2007 Progress Software Corporation 24 INT-2: XQuery Levels the Data Integration Playing Field HOLDINGS USERIDTICKERSHARES JonathanPRGS23 MinolloPRGS JonathanAMZN3000 MinolloAMZN3000 Amazon.com, Inc eBay Inc Int'l Business Machines C Progress Software COMPANIES TICKERNAMEREVENUES AMZNAmazon.com, Inc.7780 EBAYeBay Inc PRGSProgress Software493.4 YHOOYahoo! Inc  Highly optimized for relational sources  Minimizes retrieval of data No more rows than needed No more columns than needed  Uses database functionality Joins Sorting Etc..  Optimizes for each SQL dialect  Efficient JDBC retrieval Embeds DataDirect JDBC technology Optimizations added to support XQuery  Supports incremental retrieval  Optimizes for XML hierarchies Sort-merge algorithm Minimal cost of XML construction  Leverages SQL library  Supports hints XQuery can be fast for relational data!

© 2007 Progress Software Corporation 25 INT-2: XQuery Levels the Data Integration Playing Field XQuery can be fast for XML files! APPLE COMPUTER AAPL /14/2006 4:01pm % N/A N/A N/A B Nov-95 N/A 0.00  General XQuery rewrites Constant-folding, elimination of common sub-expressions, loop rewrites, ordering rewrites, etc…  Document projection XML construction accounts for much of the cost Don’t build parts of the document that the query doesn’t need!  Document streaming Discard parts of the document when no longer needed Makes memory usage near constant with size of file  Multiple Gigabytes can be queried

© 2007 Progress Software Corporation 26 INT-2: XQuery Levels the Data Integration Playing Field XQuery can use XML Converters EDI File ISA+00+DATADIRECT+00+STYLUS DATA DIRECT +01+STYLUS STUDIO ~ I+:' GS+BF+DATADIRECT+STYLUS X ' ST ' BGN GM' NM1+2L+4+Progress Software Corporation' N3+14 Oak Park Drive' N4+Bedford+MA US+AA' REF+1Z+PRGS' NM1+2L+4+Apple Computer, Inc.' N3+1 Infinite Loop' N4+Cupertino+CA US+AA' REF+1Z+AAPL' SE ' GE+1+256' IEA ' doc("adapter://EDI?ticker-request.edi") 00 DATADIRECT 00 STYLUS DATA DIRECT 01 STYLUS STUDIO ~ I : BF DATADIRECT STYLUS X Convert non-XML format to XML on-the-fly! EDI message types Comma-delimited or tab-delimited files dBase RTF mbox Batch conversions are supported Custom conversions

© 2007 Progress Software Corporation 27 INT-2: XQuery Levels the Data Integration Playing Field declare function local:amazon-listing($isbn) { All Ship ASIN { $isbn } Medium }; let $loc := let $payload := local:amazon-listing(" ") return ws:call($loc, $payload) Leverage existing SOA architecture in queries! Integrate queries with web services Easily generate complex web service requests Vastly increases the reach of your queries XQuery can access Web Services

© 2007 Progress Software Corporation 28 INT-2: XQuery Levels the Data Integration Playing Field Questions so far?

© 2007 Progress Software Corporation 29 INT-2: XQuery Levels the Data Integration Playing Field  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary Agenda

© 2007 Progress Software Corporation 30 INT-2: XQuery Levels the Data Integration Playing Field A Data Integration problem HTML SOAP through AXIS dBASE IV APIs JDBC Java/JSP code accessing the various Java APIs and generating the HTML report Java Open Client or JDBC

© 2007 Progress Software Corporation 31 INT-2: XQuery Levels the Data Integration Playing Field Data Access Layer A dangerous approach

© 2007 Progress Software Corporation 32 INT-2: XQuery Levels the Data Integration Playing Field XQuery The XQuery Vision

© 2007 Progress Software Corporation 33 INT-2: XQuery Levels the Data Integration Playing Field The DataDirect XQuery Solution HTML

© 2007 Progress Software Corporation 34 INT-2: XQuery Levels the Data Integration Playing Field Step by step  XQuery to aggregate data from the multiple data sources  XQuery to publish an HTML or XSL-FO (PDF) report directly  Pipelining multiple XQueries with validation steps  Exposing an XQuery Web Service and consuming it from OpenEdge ®

© 2007 Progress Software Corporation 35 INT-2: XQuery Levels the Data Integration Playing Field Step by step  XQuery to aggregate data from the multiple data sources  XQuery to publish an HTML or XSL-FO (PDF) report directly  Pipelining multiple XQueries with validation steps  Exposing an XQuery Web Service and consuming it from OpenEdge

© 2007 Progress Software Corporation 36 INT-2: XQuery Levels the Data Integration Playing Field Step by step  XQuery to aggregate data from the multiple data sources  XQuery to publish an HTML or XSL-FO (PDF) report directly  Pipelining multiple XQueries with validation steps  Exposing an XQuery Web Service and consuming it from OpenEdge

© 2007 Progress Software Corporation 37 INT-2: XQuery Levels the Data Integration Playing Field Step by step  XQuery to aggregate data from the multiple data sources  XQuery to publish an HTML or XSL-FO (PDF) report directly  Pipelining multiple XQueries with validation steps  Exposing an XQuery Web Service and consuming it from OpenEdge

© 2007 Progress Software Corporation 38 INT-2: XQuery Levels the Data Integration Playing Field Agenda  The Problem  Why XQuery  XQuery for Java API (XQJ)  XQuery for Data Integration  XQuery Demos, Code Walk-throughs  Summary

© 2007 Progress Software Corporation 39 INT-2: XQuery Levels the Data Integration Playing Field Benefits of XQuery  Data Integration is harder without XQuery! Every data source is different Many applications use several languages and APIs to address data sources (e.g. Java+JDBC+DOM, SQL, Perl, XSLT…) Mediating among data sources accounts for a lot of code XQuery treats all data sources as XML

© 2007 Progress Software Corporation 40 INT-2: XQuery Levels the Data Integration Playing Field Benefits of XQuery  Processing XML is harder without XQuery! Most programming languages don’t know XML structures Parse, navigate, cast, repeat XML is the native data structure for XQuery  XML Reporting is harder without XQuery! XML input and output may have very complex structure Many different desired XML outputs Data Integration, Native XML Processing are needed XQuery gives full query processing for any XML input and output

© 2007 Progress Software Corporation 41 INT-2: XQuery Levels the Data Integration Playing Field DataDirect XQuery

© 2007 Progress Software Corporation 42 INT-2: XQuery Levels the Data Integration Playing Field Getting Started …  Examples & Tutorials XQuery Tutorial XQJ Tutorial DataDirect XQuery Tutorial

© 2007 Progress Software Corporation 43 INT-2: XQuery Levels the Data Integration Playing Field Questions?

© 2007 Progress Software Corporation 44 INT-2: XQuery Levels the Data Integration Playing Field Thank you for your time

© 2007 Progress Software Corporation 45 INT-2: XQuery Levels the Data Integration Playing Field