2. From Semistructured Data to XML Dan Suciu AT&T Labs http://www.research.att.com/~suciu/vldb99-tutorial.pdf

3. How the Web is Today HTML documents all intended for human consumption many generated automatically by applications Easy to fetch any Web page, from any server, any platform

4. Limits of the Web Today application cannot consume HTML HTML wrapper technology is brittle –screen scraping OO technology (Corba) requires controlled environment companies merge, form partnerships; need interoperability fast people are inventive: send data by fax !

5. Paradigm Shift on the Web new Web standard XML: –XML generated by applications –XML consumed by applications data exchange –across platforms: enterprise interoperability –across enterprises Web: from collection of documents to data and documents

6. Database Community Can Help query optimization, processing views, transformations data warehouses, data integration mediators, query rewriting secondary storage, indexes

7. But Needs a Paradigm Shift Too Web data differs from database data: –self-describing, schema-less –structure changes without notice –heterogeneous, deeply nested, irregular –documents and data mixed together designed by document, not db experts need Web data management

8. What This Tutorial is About what the database community has done –semistructured data model –query languages, schemas what the Web community has done: –data formats/models: XML, RDF –transformation language (XSL), schemas where they meet and where they differ

9. Outline Semistructured data and XML Query languages Schemas Systems issues Conclusions

10. Part 1 Semistructured Data and XML

11. Semistructured Data Origins: integration of heterogeneous sources data sources with non-rigid structure biological data Web data

12. The Semistructured Data Model &o1 &o12&o24&o29 &o43 &96 &243 &206 &25 “Serge” “Abiteboul” 1997 “Victor” “Vianu” 122133 paper book paper references author title year http author title publisher author title page firstname lastname firstnamelastnamefirst last Bib Object Exchange Model (OEM) complex object atomic object

13. Syntax for Semistructured Data Bib: &o1 { paper: &o12 { … }, book: &o24 { … }, paper: &o29 { author: &o52 “Abiteboul”, author: &o96 { firstname: &243 “Victor”, lastname: &o206 “Vianu”}, title: &o93 “Regular path queries with constraints”, references: &o12, references: &o24, pages: &o25 { first: &o64 122, last: &o92 133} }

14. Syntax for Semistructured Data May omit oid’s: { paper: { author: “Abiteboul”, author: { firstname: “Victor”, lastname: “Vianu”}, title: “Regular path queries …”, page: { first: 122, last: 133 } }

15. Characteristics of Semistructured Data missing or additional attributes multiple attributes different types in different objects heterogeneous collections self-describing, irregular data, no a priori structure

16. Comparison with Relational Data { row: { name: “John”, phone: 3634 }, row: { name: “Sue”, phone: 6343 }, row: { name: “Dick”, phone: 6363 } } row name phone “John”3634“Sue”“Dick”63436363

17. XML a W3C standard to complement HTML origins: structured text SGML motivation: –HTML describes presentation –XML describes content http://www.w3.org/TR/REC-xml (2/98)

18. From HTML to XML HTML describes the presentation

19. HTML Bibliography Foundations of Databases Abiteboul, Hull, Vianu Addison Wesley, 1995 Data on the Web Abiteoul, Buneman, Suciu Morgan Kaufmann, 1999

20. XML Foundations… Abiteboul Hull Vianu Addison Wesley 1995 … XML describes the content

21. XML Terminology tags: book, title, author, … start tag:, end tag: elements: …, … elements are nested empty element: abbrv. an XML document: single root element well formed XML document: if it has matching tags

22. More XML: Attributes Foundations of Databases Abiteboul … 1995 attributes are alternative ways to represent data

23. More XML: Oids and References Jane Mary John oids and references in XML are just syntax

24. XML Data Model does not exists Document Object Model (DOM): –http://www.w3.org/TR/REC-DOM-Level-1 (10/98) –class hierarchy (node, element, attribute,…) –objects have behavior –defines API to inspect/modify the document

25. XML Parsers traditional: return data structure (DOM?) event based: SAX (Simple API for XML) –http://www.megginson.com/SAX –write handler for start tag and for end tag

26. XML Namespaces http://www.w3.org/TR/REC-xml-names (1/99) name ::= [prefix:]localpart … 15 ….

27. XML Namespaces syntactic:, semantic: provide URL for schema … defined here

28. XML v.s. Semistructured Data both described best by a graph both are schema-less, self-describing

29. Similarities and Differences Alan 42 ab@com { person: &o123 { name: “Alan”, age: 42, email: “ab@com” } } person nameageemail Alan42ab@com person name age email Alan42ab@com father … { person: { father: &o123 …} } similar on trees, different on graphs

30. More Differences XML is ordered, ssd is not XML can mix text and elements: Making Java easier to type and easier to type Phil Wadler XML has lots of other stuff: entities, processing instructions, comments

31. RDF http://www.w3.org/TR/REC-rdf-syntax (2/99) purpose: metadata for Web –help search engines syntax in XML semantics: edge-labeled graphs

32. RDF Syntax birds, butterflies, snakes John Smith

33. RDF Data Model www.mypage.com birds, butterflies, snakes JohnSmith aboutauthor firstnamelastname the RDF Data Model is very close to semistructured data

34. More RDF Examples www.mypage.com birds, butterflies, snakes JohnSmith aboutauthor firstnamelastname www.anotherpage.com author related Joe Doe author

35. birds, butterflies, snakes John Smith Joe Doe

36. RDF Terminology subject object predicate statement

37. More RDF: Containers bag, sequence, alternative s1 s2

38. RDF Containers (cont’d) Bag s1 s2 a rdf:type rdf_1 rdf_2

39. More RDF: Higher Order Statements “the author of www.thispage.com says: ‘the topic of www.thatpage.com is environment’ “ www.thatpage.com environment topic www.thispage.com says author RDF uses reification

40. Summary of Data Models semistructured data, XML, RDF data is self-describing, irregular schema embedded in the data

41. Part 2 Query Languages Semistructured data and XML Query languages Schemas Systems issues Conclusions

42. Query Languages: Motivation granularity of the HTML Web: one file granularity of Web data varies: –single data item: “get John’s salary” –entire database: “get all salaries” –aggregates: “get average salary” need query language to define granularity

43. Query Languages: Outline for semistructured data: –Lorel –UnQL –StruQL for XML: XML-QL a different paradigm –structural recursion –XSL

44. Lorel part of the Lore system (Stanford) adapts OQL to semistructured data select X.title from Bib.paper X where X.year > 1995 select X.title from Bib.paper X where X.year > 1995 select Bib.paper.title from Bib.paper where Bib.paper.year > 1995 select Bib.paper.title from Bib.paper where Bib.paper.year > 1995 example: abbreviated to:

45. Lorel v.s. OQL implicit coercions: 1995 to “1995” missing attributes –empty answer v.s. type error set-valued attributes –in X.year>1995, X may have several years regular path expressions (next)

46. Regular Path Expressions Useful for: syntactic substitute for inheritance: paper|book navigating partially known structures: lastname? transitive closure: reference+ select X.title from Bib.paper X, Bib.(paper|book) Y where Y.author.lastname? = “Ullman” and Y.reference+ X select X.title from Bib.paper X, Bib.(paper|book) Y where Y.author.lastname? = “Ullman” and Y.reference+ X

47. select T where Bib.paper: { title: T, year: Y, journal: “TODS”} and Y > 1995 select T where Bib.paper: { title: T, year: Y, journal: “TODS”} and Y > 1995 UnQL Unstructured Query Language patterns, templates, structural recursion patterns:

48. UnQL: Templates select result: { fn: F, ln: L, pub: { title: T, year: Y }} where Bib.paper: { title: T, year: Y, journal: “TODS”} and Y > 1995 select result: { fn: F, ln: L, pub: { title: T, year: Y }} where Bib.paper: { title: T, year: Y, journal: “TODS”} and Y > 1995 Result looks like: { result: { fn: “John”, ln: “Smith”, pub: { title: “P equals NP”, year: 2005}}, result: { fn: “Joe”, ln: “Doe”, pub: { title: “Errata to P=NP”, year: 2006}} … }

49. Skolem Functions Maier, 1986 –in OO systems Kifer et al, 1989 –F-logic Hull and Yoshikawa, 1990 –deductive db (ILOG) Papakonstantinou et al., 1996 –semistructured db (MSL) illustrate with Strudel (next)

50. Skolem Functions in StruQL Strudel: a Web Site Management System StruQL: its query language

51. Example: Bibliography Data {Bib: { paper: { author: “Jones”, author: “Smith”, title: “The Comma”, year: 1994 } }, { paper: ….. } }

52. Example: A Complex Web Site Root() YearPage(“Smith”, 1994) YearPage(“Smith”, 1996) YearPage(“Jones”, 1994) YearPage(“Jones”, 1998) YearPage(“Mark”, 1996) yearentry publication PubPage(“The Comma”)PubPage(“The Dot”) publication title author HomePage(“Smith”)HomePage(“Jones”)HomePage(“Mark”) person

53. Example: Skolem Functions in StruQL where Root -> “Bib” -> X, X -> “paper” -> P, P -> “author” -> A, P -> “title” -> T, P -> “year” -> Y create Root(), HomePage(A), YearPage(A,Y), PubPage(P) link Root() -> “person” -> HomePage(A), HomePage(A) -> “yearentry” -> YearPage(A,Y), YearPage(A,Y) -> “publication” -> PubPage(P), PubPage(P) -> “author” -> HomePage(A), PubPage(P) -> “title” -> T where Root -> “Bib” -> X, X -> “paper” -> P, P -> “author” -> A, P -> “title” -> T, P -> “year” -> Y create Root(), HomePage(A), YearPage(A,Y), PubPage(P) link Root() -> “person” -> HomePage(A), HomePage(A) -> “yearentry” -> YearPage(A,Y), YearPage(A,Y) -> “publication” -> PubPage(P), PubPage(P) -> “author” -> HomePage(A), PubPage(P) -> “title” -> T

54. XML-QL: A Query Language for XML http://www.w3.org/TR/NOTE-xml-ql (8/98) features: –regular path expressions –patterns, templates –Skolem Functions based on OEM data model

55. Pattern Matching in XML-QL where Morgan Kaufmann $a in “www.a.b.c/bib.xml” construct $a where Morgan Kaufmann $a in “www.a.b.c/bib.xml” construct $a

56. Simple Constructors in XML-QL Note: abbreviates or or... where $a in “www.a.b.c/bib.xml” construct $a $l where $a in “www.a.b.c/bib.xml” construct $a $l Smith English Smith Mandarin Doe English

57. Skolem Functions in XML-QL where $a in “www.a.b.c/bib.xml” construct $a $l where $a in “www.a.b.c/bib.xml” construct $a $l Smith English Mandarin Doe English

58. A Different Paradigm: Structural Recursion Data as sets with a union operator: {a:3, a:{b:”one”, c:5}, b:4} = {a:3} U {a:{b:”one”,c:5}} U {b:4}

59. Structural Recursion f(T1 U T2) = f(T1) U f(T2) f({L: T}) = f(T) f({}) = {} f(V) = if isInt(V) then {result: V} else {} f(T1 U T2) = f(T1) U f(T2) f({L: T}) = f(T) f({}) = {} f(V) = if isInt(V) then {result: V} else {} Example: retrieve all integers in the data a a b b c3 “one”5 4 result 354 standard textbook programming on trees

60. Structural Recursion Example: increase all engine prices by 10% f(T1 U T2) = f(T1) U f(T2) f({L: T}) = if L= engine then {L: g(T)} else {L: f(T)} f({}) = {} f(V) = V f(T1 U T2) = f(T1) U f(T2) f({L: T}) = if L= engine then {L: g(T)} else {L: f(T)} f({}) = {} f(V) = V g(T1 U T2) = g(T1) U g(T2) g({L: T}) = if L= price then {L:1.1*T} else {L: g(T)} g({}) = {} g(V) = V g(T1 U T2) = g(T1) U g(T2) g({L: T}) = if L= price then {L:1.1*T} else {L: g(T)} g({}) = {} g(V) = V enginebody partprice part price 100 1000 100 1000 enginebody partprice part price 110 1100 100 1000

61. XSL two W3C drafts: XSLT and XPATH –http://www.w3.org/TR/xpath, 7/99 –http://www.w3.org/TR/WD-xslt, 7/99 in commercial products (e.g. IE5.0) purpose: stylesheet specification language: –stylesheet: XML -> HTML –in general: XML -> XML

62. XSL Templates and Rules query = collection of template rules template rule = match pattern + template Retrieve all book titles:

63. XPath Expressions in Match Patterns bibmatches a bib element *matches any element /matches the root element /bibmatches a bib element under root bib/papermatches a paper in bib bib//papermatches a paper in bib, at any depth //papermatches a paper at any depth paper|bookmatches a paper or a book @pricematches a price attribute bib/book/@pricematches price attribute in book, in bib

64. Flow Control in XSL

65. 1 2 3 4 1 2 3 4

66. XSL is Structural Recursion Equivalent to: f(T1 U T2) = f(T1) U f(T2) f({L: T}) = if L= c then {C: t} else L= b then {B: f(t)} else L= a then {A: f(t)} else f(t) f({}) = {} f(V) = V f(T1 U T2) = f(T1) U f(T2) f({L: T}) = if L= c then {C: t} else L= b then {B: f(t)} else L= a then {A: f(t)} else f(t) f({}) = {} f(V) = V XSL query = single function XSL query with modes = multiple function

67. XSL and Structural Recursion XSL: trees only may loop Structural Recursion: arbitrary graphs always terminates stack overflow on IE 5.0 add the following rule:

68. Summary of Query Languages studied extensively in semistructured data some quite powerful features no standard for XML QL yet (WG soon) XSL available today (for stylesheets) XSL = structural recursion

69. Part 3 Schemas Semistructured data and XML Query languages Schemas Systems issues Conclusions

70. Schemas why ? –XML: to describe semantics –semistructured data: to improve processing what ? –semistructured data: foundational –XML: several concrete proposals here lies our interest

71. Schemas when ? –semistructured data, XML: a posteriori –RDBMS: a priori, to interpret binary data how ? –semistructured data: schema is independent –XML: schema is hardwired with the data

72. Outline schemas for semistructured data: –foundations –schema extraction schemas for XML: –DTD –XML-Schema –RDF-Schema

73. Schemas: An Example &r1 &c1&c2 &s2&s3&s6&s7 &s10 company name address name url address “Widget” “Trenton” “Gadget” “www.gp.fr” “Paris” &p2&p1&p3 &s0&s1&s4&s5&s8&s9 person “Smith” name position name phone name position “Manager” “Jones” “5552121” “Dupont” “Sales” employee manages c.e.o. works-for c.e.o. &a1 &a2 &a3 &a4 &a5 &a6 &a7 description procurement salesrep contact task eval 1997 1998 “on target” “below target” Some database:

74. Lower-Bound Schemas Root Company Employee string company person works-for c.e.o. address name managed-by name

75. Upper Bound Schemas Root Company Employee string company person works-for c.e.o. | employee name | address | url managed-by name | phone | position Any description -

76. The Two Questions to Ask Conformance: does that data conform to this schema ? Classification: if so, then which objects belong to what classes ?

77. Graph Simulation Definition Two edge-labeled graphs G1, G2 A simulation is a relation R between nodes: if (x1, x2) in R, and (x1,a,y1) in G1, then exists (x2,a,y2) in G2 (same label) s.t. (y1,y2) in R x1x2 a R G1G2 y1 a R y2 Note: a simulation can be efficiently computed [Henzinger, et a. 1995]

78. Using Simulation Data graph D, schema S upper bound schema: –conformance: find simulation R from D to S –classification: check if (x,c) in R lower bound schema –conformance: find simulation R from S to D –classification: check if (c,x) in R [Buneman et al 1997]

79. Example &r1 &c1&c2 &s2&s3&s6&s7 &s10 company name address name url address “Widget” “Trenton” “Gadget” “www.gp.fr” “Paris” &p2&p1&p3 &s0&s1&s4&s5&s8&s9 person “Smith” name positionname phone name position “Manager” “Jones” “5552121” “Dupont” “Sales” employee manages c.e.o. works-for c.e.o. &a1 &a2 &a3 &a4 &a5 &a6 &a7 description procurement salesrep contact task eval 1997 1998 “on target” “below target” Root Company Employee string company person works-for c.e.o. address name managed-by name Root Company Employee string company person works-for c.e.o. | employee name | address | url managed-by name | phone | position Any description - DatabaseLower BoundUpper Bound simulation: efficient technique for checking conformance to schema

80. Application 1: Improve Secondary Storage Root Company Employee string company person works-for c.e.o. address name managed-by name Company Employee Store rest in overflow graph Lower-bound schema

81. Application 2: Query Optimization Bib paperbook year journal title intstring address author title zip city street last name first name string select X.title from Bib._ X where X.*.zip = “12345” select X.title from Bib._ X where X.*.zip = “12345” select X.title from Bib.book X where X.address.zip = “12345” select X.title from Bib.book X where X.address.zip = “12345” Upper-bound schema [Fernandez, Suciu 1998]

82. Schema Extraction (From Data) Problem statement given data instance D find the “most specific” schema S for D In practice: S too large, need to relax [Nestorov et al. 1998]

83. Schema Extraction: Sample Data &r &p8&p1&p2&p3&p4&p5&p6&p7 &c company employee worksfor manages managedby manages managedby

84. Lower Bound Schema Extraction Root &r Bosses &p1,&p4,&p6 Regulars &p2,&p3,&p5,&p7,&p8 Company &c company employee manages managedby worksfor employee

85. Upper Bound Schema Extraction: Data Guides Root &r Employees &p1,&p1,&p3,P4 &p5,&p6,&p7,&p8 Bosses &p1,&p4,&p6 Regulars &p2,&p3,&p5,&p7,&p8 Company &c company employee manages managedby manages managedby worksfor

86. Schemas in XML Document Type Definition (DTD) XML Schema RDF Schema

87. Document Type Definition: DTD part of the original XML specification an XML document may have a DTD terminology for XML: –well-formed: if tags are correctly closed –valid: if it has a DTD and conforms to it validation is useful in data exchange

88. DTDs as Grammars <!DOCTYPE paper [ ]> <!DOCTYPE paper [ ]> …

89. DTDs as Schemas Not so well suited: impose unwanted constraints on order references cannot be constraint can be to vague:

90. XML Schemas very recent proposal unifies previous schema proposals generalizes DTDs uses XML syntax two documents: structure and datatypes –http://www.w3.org/TR/xmlschema-1 –http://www.w3.org/TR/xmlschema-2

91. XML Schemas DTD:

92. RDF Schemas http://www.w3.org/TR/PR-rdf-schema (3/99) object-oriented flavor

93. RDF Schemas recall RDF data: –resources –properties RDF schema: –classes –properties subject object predicate statement

94. RDF Schemas Data: My Honda 50000

95. RDF Schemas Schema:

96. RDF Schemas Truck MotorVehicle car001 type Class type subClassOf name miles My Honda 50000

97. RDF Schemas different from object-oriented systems: –OO: define a class by set of properties –RDF: define a property in terms of its classes metadata in RDF: –an RDF schema described as an RDF data

98. Summary of Schemas in SS data: –graph theoretic –data and schema are decoupled –used in data processing in XML –from grammar to object-oriented –schema wired with the data –emphasis on semantics for exchange

99. Part 4 Systems Issues Semistructured data and XML Query languages Schemas Systems issues Conclusions

100. Systems Issues servers mediators

101. Servers for Semistructured Data / XML storage index query evaluation [McHugh, Widom 1999]

102. XML Storage text file (XML) store in ternary relation use DTD to derive schema mine data to derive schema build special purpose repository (Lore)

103. XML Storage: Text File advantages –simple –less space than one thinks –reasonable clustering disadvantage –no updates –require special purpose query processor

104. &o1 &o3 &o2 &o4&o5 paper title author year &o6 “The Calculus”“…” “1986” Store XML in Ternary Relation [Florescu, Kossman 1999] Ref Val

105. Use DTD to derive Schema DTD: ODMG classes: [Christophides et al. 1994, Shanmugasundaram et al. 1999] class Employee public type tuple (name:string, address:Address, project:List(Project)) class Address public type tuple (street:string, …)

106. Mine Data to Derive Schema paper author title year fn ln Paper1 Paper2 [Deutsch et al. 1999]

107. Indexing Semistructured Data coercions: 1995 v.s. “1995” regular path expressions –data guides [Goldman, Widom, 1997] –T-indexes [Milo, Suciu, 1999]

108. Indexing All Paths in the Data 1 23456 78910111213 ttttt abacadaab Semistructured Data 1 2 3 4 5 6 7 8 10 12 137 13 911 t a b c d Data Guide 1 2 3 4 5 6 7 138 10 12 911 t a bcd b T-Index

109. Mediators for Semistructured Data / XML XML = virtual view of Relational/OO/OR sources mediator = translation, integration issues: –query composition and rewriting [Papakonstatinou, et al. 1996 ] –limited source capabilities [Yerneni, et al. 1999]

110. Example: An XML Mediator relational database: virtual XML view: n1... n2... … StoreSBBook

111. Example: An XML Mediator specify mediator declaratively (a view): from Store, SB, Book where Store.sid=SB.sid and SB.bid=Book.bid construct Store.name Book.title from Store, SB, Book where Store.sid=SB.sid and SB.bid=Book.bid construct Store.name Book.title

112. Example: An XML Mediator users ask XML-QL queries: –find stores who sell “The Calculus” where $n The Calculus construct $n where $n The Calculus construct $n

113. Example: An XML Mediator system composes query with view: from Store, SB, Book where Store.sid=SB.sid and SB.bid=Book.bid and Book.title=“The Calculus” construct Store.name from Store, SB, Book where Store.sid=SB.sid and SB.bid=Book.bid and Book.title=“The Calculus” construct Store.name

114. Summary of Systems unclear today how XML will be used –materialized ? Need servers –virtual ? Need mediators most work is still ahead

115. Part 5 Conclusions Semistructured data and XML Query languages Schemas Systems issues Conclusions

116. Summary XML = what is out there semistructured data = what we can process paradigm shift, for both Web and db covered in tutorial: – data models, queries, schemas

117. Current and Future Technologies Web applications possible today: –export relational data to XML (e.g. Oracle) –import XML directly into applications Web applications in the future: –mediator technology (XML view) –store/process native XML data –compress XML –mine/analyze XML

118. Why This Is Cool for Database Researchers put to work what you teach in CS101 ! –tree traversals (structural recursion, XSL) –automata theory (DTD’s, path expressions) –graph theory (simulation) adapt old DB tricks to new kind of data save the trees: from fax to XML The End

119. Further Readings www. w3.org/XML www-db.stanford.edu/~widom www-rocq.inria.fr/~abiteboul db.cis.upenn.edu www.research.att.com/~suciu Abiteboul, Buneman, Suciu Data on the Web: From Relational to Semistructured to XML Morgan Kaufmann, 1999 (appears in October)