1. Overview of XML Data Management Research at Cornell Jayavel Shanmugasundaram Cornell University

2. Why XML? Internet data exchange –XML emerging as dominant standard for data interactions over the Internet (e.g., SOAP) –Consequently, web application developers deal with XML (e.g., WebSphere) Captures structured and unstructured data –Content management –Semi-structured data

3. Outline XML for data exchange XML for structured and unstructured data Conclusion

4. XML for Data Exchange Tires R Us Cars R Us Order Fulfillment Application Purchasing Application Internet eXtensible Markup Language (XML) Relational Database System

5. Key Challenges Publishing relational data as XML –For the foreseeable future, most business data will continue to be store in relational databases –Need to publish relational data as XML Storing XML using relational database systems –Need to manage XML documents being transferred across the wire (Purchase orders for auditing etc.) –Do we need to build a specialized XML database? Or can we leverage relational technology?

6. Outline XML for data exchange –Publishing relational data as XML –Querying XML using relational databases XML for structured and unstructured data Conclusion

7. Example Relational Data idcustnamecustnum 10Smith Construction7734 9Western Builders7725 order oiddesccost 10generator8000 10backhoe24000 oiddueamt 101/10/0120000 106/10/0112000 itempayment

8. XML View for Users Smith Construction 8000 24000 20000 12000 …

9. Allow Users to Query View Get all orders of customer ‘Smith…’ for $order in view(“orders”) where $order/customer/text() like ‘Smith%’ return $order

10. // First prepare all the SQL statements to be executed and create cursors for them Exec SQL Prepare CustStmt From “select cust.id, cust.name from Customer cust where cust.name = ‘Jack’“ Exec SQL Declare CustCursor Cursor For CustStmt Exec SQL Prepare AcctStmt From “select acct.id, acct.acctnum from Account acct where acct.custId = ?“ Exec SQL Declare AcctCursor Cursor For AcctStmt Exec SQL Prepare PorderStmt From “select porder.id, porder.acct, porder.date from PurchOrder porder where porder.custId = ?“ Exec SQL Declare PorderCursor Cursor For PorderStmt Exec SQL Prepare ItemStmt From “select item.id, item.desc from Item item where item.poId = ?“ Exec SQL Declare ItemCursor Cursor For ItemStmt Exec SQL Prepare PayStmt From “select pay.id, pay.desc from Payment pay where item.poId = ?“ Exec SQL Declare PayCursor Cursor For PayStmt // Now execute SQL statements in nested order of XML document result. Start with customer XMLresult = ““ Exec SQL Open CustCursor while (CustCursor has more rows) { Exec SQL Fetch CustCursor Into :custId, :custName XMLResult += “ “ + custName + “ “ // For each customer, issue sub-query to get account information and add to custAccts Exec SQL Open AcctCursor Using :custId while (AcctCursor has more rows) { Exec SQL Fetch AcctCursor Into :acctId, :acctNum XMLResult += “ “ + acctNum + “ “ } XMLResult += “ “ // For each customer, issue sub-query to get purchase order information and add to custPorders Exec SQL Open PorderCursor Using :custId while (PorderCursor has more rows) { Exec SQL Fetch PorderCursor Into :poId, :poAcct, :poDate XMLResult += “ “+poDate +“ “ // For each purchase order, issue a sub-query to get item information and add to porderItems Exec SQL Open ItemCursor Using :poId while (ItemCursor has more rows) { Exec SQL Fetch ItemCursor Into :itemId, :itemDesc XMLResult += “ “ + itemDesc + “ “ } XMLResult += “ “ // For each purchase order, issue a sub-query to get payment information and add to porderPays Exec SQL Open PayCursor Using :poId while (PayCursor has more rows) { Exec SQL Fetch PayCursor Into :payId, :payDesc XMLResult += “ “ + payDesc + “ “ } XMLResult += “ “ } // End of looping over all purchase orders associated with a customer XMLResult += “ “ Return XMLResult as one result row; reset XMLResult = ““ } // loop until all customers are tagged and output

11. Previous Work SQL extensions for publishing relational data as XML –[Shanmugasundaram et al., VLDB 2000] –Prototyped in DB2 –Input into SQL/X working group XML publishing using XQuery –[Shanmugasundaram et al., VLDB 2001] –Initially XPERANTO prototype –Now XTables product initiative

12. Updates Updating XML views of relational data –Extend XQuery with update semantics –Translate XQuery updates to SQL updates (when possible!); efficiently! for $order in view(“orders”) where $order/customer/text() = ‘Smith’ update $order/cost = $order/cost + 100.00

13. Recursion ‘//’ queries are very popular –Navigational recursion XQuery functions allow structural recursion –Part hierarchies –Nested catalogs How can we evaluate them using a relational database system? –View composition –Fix-point recursion in SQL

14. Outline XML for data exchange –Publishing relational data as XML –Querying XML using relational databases XML for structured and unstructured data Conclusion

15. Native XML Documents

16. Querying Native XML Documents Native XML database systems –Specialized for XML document processing Extend relational (or object-oriented) database systems –Leverage > 30 years of research and development –Harness sophisticated functionality, tools

17. Previous Work [Shanmugasundaram et. al., VLDB’99] [Shanmugasundaram et al., SIGMOD Record’01] Relational Database System XML Translation Layer XML Schema Relational Schema Translation Information XML Documents Tuples XML Query SQL Query Relational Result XML Result

18. Query Workload Different XML shredding techniques can have a dramatic influence on performance –How can we choose appropriate shredding based on XML query and update workload? –“SMART for XML”

19. Typing XML Schemas have many sophisticated constraints –Min occurs, max occurs –Structural constraints How can we preserve these in relational database systems in presence of updates? –Relational constraints? –Materialized views?

20. Outline XML for data exchange XML for structured and unstructured data Conclusion

21. 30000-foot view of Data Management Today Essentially two “camps” Structured camp: Relational database systems –Highly structured data –Precise and sophisticated queries over this data Unstructured camp: Information retrieval systems –Unstructured data –Keyword search queries returning ranked results

22. Traditional Data Management Landscape StructuredUnstructured Complex and Structured Ranked Keyword Search Data Queries (Relational) Database Systems Information Retrieval Systems 12 34

23. Primary Advantages of Ranked Keyword Search Simple –As witnessed by popularity of keyword search over the Internet Facilitates information discovery –Ranks results in order of importance –Users do not need to know the schema of the underlying data (if there is any) “John Ithaca”

24. Ranked Keyword Search over Structured and Unstructured Data Content management –Semi-structured data Scientific documents, Shakespeare’s plays, … –Mix of structured and unstructured data Database with date and time of accident (structured data) and accident description (unstructured data) Support flexible keyword search interface

25. Semi-structured Document XML and Information Retrieval: A SIGIR 2000 Workshop David Carmel, Yoelle Maarek, Aya Soffer XQL and Proximal Nodes Ricardo Baeza-Yates Gonzalo Navarro Searching on structured text is becoming more important with XML … The XQL language … … A Query Language … …

26. Key Challenges Generalize ranked keyword query semantics –Should work as usual for unstructured data –Should generalize to structured data too! –Allows users to query across both forms of data Generalized inverted lists –Indexing mix of structured and unstructured data More on this soon!

27. Traditional Data Management Landscape StructuredUnstructured Complex and Structured Ranked Keyword Search Data Queries (Relational) Database Systems Information Retrieval Systems 12 34

28. Complex Queries over Semi- structured Data: Motivation 1 Document repositories do not typically conform to a rigid schema –Scientific documents –Powerpoint presentation –Push to publish these in XML form Complex queries over such heterogeneous collections (in conjunction with structured data) –Find all documents on “XML” authored by Almaden employees

29. Complex Queries over Semi- structured Data: Motivation 2 Even “structured” data can have a widely varying structure –Example: electronic parts market place –2 million parts each having about 10-15 attributes –A total of 5000 distinct attributes Structure changes very often (“schema chaos”) –New parts are added every day May be better to treat this data as “semi-structured” But … still need to ask structured queries –Find capacitors with capacitance between 10 and 20

30. Indexing and Query Processing Index both schema and data –/book[author/name = ‘Jane’] –Treat schema as a data value Benefits –Can capture arbitrarily heterogeneous schema –Easy schema evolution –Can implement it using a relational database system! (using regular B+-trees) Supports wildcard (‘//’) queries

31. Order [Tatarinov et. al., SIGMOD’02] Shakespeare’s plays marked up as XML –Acts ordered one after the other –Cannot view this as an “unordered set” XQuery queries support ordered predicates –Find acts after Hamlet said “to be or not to be” Again, treat order as a data value –Order encoding methods –Can be implemented in a relational database system!

32. Traditional Data Management Landscape StructuredUnstructured Complex and Structured Ranked Keyword Search Data Queries (Relational) Database Systems Information Retrieval Systems 12 34

33. Unifying IR and Database Systems: Motivation The Internet is opening the door to ad-hoc queries by end users –E.g., Used car marketplace –Find all “bright red ford mustangs” that cost less than 20% of the average price of cars in its class Characteristics of queries –Ranked keyword search (for ease of use) –Complex query operations (information synthesis) –Want to see ranked results!

34. Main Challenge Integrate ranking with structured query operations Developing ~XQuery framework –Build ranking into core of language –Both keyword search and structured operators Open question –Will we be able to extend relational databases for this purpose? Find “bright red ford mustangs” that cost less than 20% of the average price of cars in its class

35. Outline XML for data exchange XML for structured and unstructured data –Overview –Ranked keyword search over XML documents Conclusion

36. XRANK: Ranked Keyword Search over XML Documents Lin Guo Feng Shao Chavdar Botev Jayavel Shanmugasundaram

37. Traditional Data Management Landscape StructuredUnstructured Complex and Structured Ranked Keyword Search Data Queries (Relational) Database Systems Information Retrieval Systems 12 34

38. Semi-structured Document XML and Information Retrieval: A SIGIR 2000 Workshop David Carmel, Yoelle Maarek, Aya Soffer XQL and Proximal Nodes Ricardo Baeza-Yates Gonzalo Navarro Searching on structured text is becoming more important with XML … The XQL language … … A Query Language … …

39. Design Principles 1)Return most specific element containing the query keywords

40. Semi-structured Document XML and Information Retrieval: A SIGIR 2000 Workshop David Carmel, Yoelle Maarek, Aya Soffer XQL and Proximal Nodes Ricardo Baeza-Yates Gonzalo Navarro Searching on structured text is becoming more important with XML … The XQL language … … A Query Language … …

41. Design Principles 1)Return most specific element containing the query keywords 2)Ranking has to be done at the granularity of elements

42. Semi-structured Document XML and Information Retrieval: A SIGIR 2000 Workshop David Carmel, Yoelle Maarek, Aya Soffer XQL and Proximal Nodes Ricardo Baeza-Yates Gonzalo Navarro Searching on structured text is becoming more important with XML … The XQL language … … A Query Language … …

43. Design Principles 1)Return most specific element containing the query keywords 2)Ranking has to be done at the granularity of elements 3)Two-dimensional keyword proximity Height of result XML tree Width of result XML tree

44. System Architecture Standing Computation Hybrid Dewey Inverted List Query Evaluator Input XML Documents XML Elements with Standings Keyword query Ranked Results Data access

45. Naïve Approach One main difference between document and XML keyword search is result granularity –Treat each element as a document –Build regular inverted list index structures over elements Drawbacks –Space overhead (depth of document) –Ranking (two-dimensional proximity) –Spurious query results

46. Semi-structured Document XML and Information Retrieval: A SIGIR 2000 Workshop David Carmel, Yoelle Maarek, Aya Soffer XQL and Proximal Nodes Ricardo Baeza-Yates Gonzalo Navarro Searching on structured text is becoming more important with XML … The XQL language … … A Query Language … …

47. Main Problem with Naïve Approach Decouples representation of ancestors and descendants –Space overhead –Spurious query results Dewey encoding for ids –General knowledge classification (1850s) –LDAP, Ordered XML, …

48. Dewey Encoding 0.0date0.1 0 0.2 0.3 28 July …XML and …David Carmel … 0.3.0 0.3.1 … 0.3.0.0 0.3.0.1 …… XQL and …Ricardo …

49. Dewey Inverted List (DIL) XQL 5.0.3.0.08532 Dewey Id Standing Position List 5.0.3.8.33889 Sorted by Dewey Id ……… Ricardo 5.0.3.0.18238 8.2.1.4.29952 Sorted by Dewey Id ……… … 91

50. Query Processing Can answer XML keyword search queries in single pass over DIL –Space savings –Time savings (smaller inverted lists) Ranking refinements –Ranked Dewey Inverted List (RDIL) –Hybrid Dewey Inverted List (HDIL)

51. Outline XML for data exchange XML for structured and unstructured data Conclusion

52. Two main uses of XML –Data exchange –Managing structured and unstructured data Each of these gives rise to exciting data management opportunities Pursuing these actively at Cornell http://www.cs.cornell.edu/people/jai