2. 1 Lecture 5: Relational Algebra and XML Monday, April 26th, 2004

3. 2 Course Agenda Today, XML and relational algebra Next two weeks: the internals of DBMS. –Covered in gory detail in the book, but stay tuned for reading assignments. May 20th (not 17th!): Phil Bernstein on meta-data management. May 24th: data integration. May 27th: final exam.

4. 3 Agenda Relational algebra XML: –What is it and why do we care? –Data model –Query language: XPath –Real query language: XQuery. –General ruminations about XML.

5. 4 Relational Algebra Formalism for creating new relations from existing ones Its place in the big picture: Declartive query language Algebra Implementation SQL, relational calculus Relational algebra Relational bag algebra

6. 5 Relational Algebra Five operators: –Union:  –Difference: - –Selection:  –Projection:  –Cartesian Product:  Derived or auxiliary operators: –Intersection, complement –Joins (natural,equi-join, theta join, semi-join) –Renaming: 

7. 6 1. Union and 2. Difference R1  R2 Example: –ActiveEmployees  RetiredEmployees R1 – R2 Example: –AllEmployees -- RetiredEmployees

8. 7 What about Intersection ? It is a derived operator R1  R2 = R1 – (R1 – R2) Also expressed as a join (will see later) Example –UnionizedEmployees  RetiredEmployees

9. 8 3. Selection Returns all tuples which satisfy a condition Notation:  c (R) Examples –  Salary > 40000 (Employee) –  name = “Smith” (Employee) The condition c can be =,, , <>

10. 9 Find all employees with salary more than $40,000.  Salary > 40000 (Employee)

11. 10 4. Projection Eliminates columns, then removes duplicates Notation:  A1,…,An (R) Example: project social-security number and names: –  SSN, Name (Employee) –Output schema: Answer(SSN, Name)

12. 11  SSN, Name (Employee)

13. 12 5. Cartesian Product Each tuple in R1 with each tuple in R2 Notation: R1  R2 Example: –Employee  Dependents Very rare in practice; mainly used to express joins

14. 13

15. 14 Renaming Changes the schema, not the instance Notation:  B1,…,Bn (R) Example: –  LastName, SocSocNo (Employee) –Output schema: Answer(LastName, SocSocNo)

16. 15 Renaming Example Employee NameSSN John999999999 Tony777777777 LastNameSocSocNo John999999999 Tony777777777  LastName, SocSocNo (Employee)

17. 16 Natural Join Notation: R1 ⋈ R2 Meaning: R1 ⋈ R2 =  A (  C (R1  R2)) Where: –The selection  C checks equality of all common attributes –The projection eliminates the duplicate common attributes

18. 17 Natural Join Example Employee NameSSN John999999999 Tony777777777 Dependents SSNDname 999999999Emily 777777777Joe NameSSNDname John999999999Emily Tony777777777Joe Employee Dependents =  Name, SSN, Dname (  SSN=SSN2 (Employee x  SSN2, Dname (Dependents))

19. 18 Natural Join R= S= R ⋈ S= AB XY XZ YZ ZV BC ZU VW ZV ABC XZU XZV YZU YZV ZVW

20. 19 Natural Join Given the schemas R(A, B, C, D), S(A, C, E), what is the schema of R ⋈ S ? Given R(A, B, C), S(D, E), what is R ⋈ S ? Given R(A, B), S(A, B), what is R ⋈ S ?

21. 20 Theta Join A join that involves a predicate R1 ⋈  R2 =   (R1  R2) Here  can be any condition

22. 21 Eq-join A theta join where  is an equality R1 ⋈ A=B R2 =  A=B (R1  R2) Example: –Employee ⋈ SSN=SSN Dependents Most useful join in practice

23. 22 Semijoin R ⋉ S =  A1,…,An (R ⋈ S) Where A 1, …, A n are the attributes in R Example: –Employee ⋉ Dependents

24. 23 Semijoins in Distributed Databases Semijoins are used in distributed databases SSNName... SSNDnameAge... Employee Dependents network Employee ⋈ ssn=ssn (  age>71 (Dependents)) T =  SSN  age>71 (Dependents) R = Employee ⋉ T Answer = R ⋈ Dependents

25. 24 Complex RA Expressions Person Purchase Person Product  name=fred  name=gizmo  pid  ssn seller-ssn=ssnpid=pidbuyer-ssn=ssn  name

26. 25 Operations on Bags A bag = a set with repeated elements All operations need to be defined carefully on bags {a,b,b,c}  {a,b,b,b,e,f,f}={a,a,b,b,b,b,b,c,e,f,f} {a,b,b,b,c,c} – {b,c,c,c,d} = {a,b,b,d}  C (R): preserve the number of occurrences  A (R): no duplicate elimination Cartesian product, join: no duplicate elimination Important ! Relational Engines work on bags, not sets ! Reading assignment: 5.3 – 5.4

27. 26 Finally: RA has Limitations ! How do we compute “transitive closure”? Find all direct and indirect relatives of Fred Name1Name2Relationship FredMaryFather MaryJoeCousin MaryBillSpouse NancyLouSister

28. 27 XMLXML

29. 28 XML eXtensible Markup Language XML 1.0 – a recommendation from W3C, 1998 Roots: SGML (a very nasty language). After the roots: a format for sharing data

30. 29 Why XML is of Interest to Us XML is just syntax for data –Note: we have no syntax for relational data –But XML is not relational: semistructured This is exciting because: –Can translate any data to XML –Can ship XML over the Web (HTTP) –Can input XML into any application –Thus: data sharing and exchange on the Web

31. 30 XML Data Sharing and Exchange application relational data Transform Integrate Warehouse XML DataWEB (HTTP) application legacy data object-relational Specific data management tasks

32. 31 From HTML to XML HTML describes the presentation

33. 32 HTML Bibliography Foundations of Databases Abiteboul, Hull, Vianu Addison Wesley, 1995 Data on the Web Abiteoul, Buneman, Suciu Morgan Kaufmann, 1999 Bibliography Foundations of Databases Abiteboul, Hull, Vianu Addison Wesley, 1995 Data on the Web Abiteoul, Buneman, Suciu Morgan Kaufmann, 1999

34. 33 XML Foundations… Abiteboul Hull Vianu Addison Wesley 1995 … Foundations… Abiteboul Hull Vianu Addison Wesley 1995 … XML describes the content

35. 34 Web Services A new paradigm for creating distributed applications? Systems communicate via messages, contracts. Example: order processing system. MS.NET, J2EE – some of the platforms XML – a part of the story; the data format.

36. 35 XML Terminology tags: book, title, author, … start tag:, end tag: elements: …, … elements are nested empty element: abbrv. an XML document: single root element well formed XML document: if it has matching tags

37. 36 More XML: Attributes Foundations of Databases Abiteboul … 1995 Foundations of Databases Abiteboul … 1995 attributes are alternative ways to represent data

38. 37 More XML: Oids and References Jane Mary John Jane Mary John oids and references in XML are just syntax

39. 38 XML Semantics: a Tree ! Mary Maple 345 Seattle John Thailand 23456 Mary Maple 345 Seattle John Thailand 23456 data Mary person name address name address streetnocity Maple345 Seattle John Thai phone 23456 id o555 Element node Text node Attribute node Order matters !!!

40. 39 XML Data XML is self-describing Schema elements become part of the data –Reational schema: persons(name,phone) –In XML,, are part of the data, and are repeated many times Consequence: XML is much more flexible XML = semistructured data

41. 40 Relational Data as XML John 3634 Sue 6343 Dick 6363 John 3634 Sue 6343 Dick 6363 row name phone “John”3634“Sue”“Dick”63436363 person XML: person

42. 41 XML is Semi-structured Data Missing attributes: Could represent in a table with nulls John 1234 Joe John 1234 Joe  no phone ! namephone John1234 Joe-

43. 42 XML is Semi-structured Data Repeated attributes Impossible in tables: Mary 2345 3456 Mary 2345 3456  two phones ! namephone Mary23453456 ???

44. 43 XML is Semi-structured Data Attributes with different types in different objects Nested collections (no 1NF) Heterogeneous collections: – contains both s and s John Smith 1234 John Smith 1234  structured name !

45. 44 Document Type Definitions DTD part of the original XML specification an XML document may have a DTD XML document: well-formed = if tags are correctly closed Valid = if it has a DTD and conforms to it validation is useful in data exchange

46. 45 Very Simple DTD <!DOCTYPE company [ ]> <!DOCTYPE company [ ]>

47. 46 Very Simple DTD 123456789 John B432 1234 987654321 Jim B123... 123456789 John B432 1234 987654321 Jim B123... Example of valid XML document:

48. 47 DTD: The Content Model Content model: –Complex = a regular expression over other elements –Text-only = #PCDATA –Empty = EMPTY –Any = ANY –Mixed content = (#PCDATA | A | B | C)* content model

49. 48 DTD: Regular Expressions <!ELEMENT name (firstName, lastName)).......... <!ELEMENT name (firstName?, lastName)) DTDXML <!ELEMENT person (name, phone*)) sequence optional <!ELEMENT person (name, (phone|email))) Kleene star alternation......................

50. 49 Querying XML Data XPath = simple navigation through the tree XQuery = the SQL of XML XSLT = recursive traversal –will not discuss in class

51. 50 Sample Data for Queries Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998 Addison-Wesley Serge Abiteboul Rick Hull Victor Vianu Foundations of Databases 1995 Freeman Jeffrey D. Ullman Principles of Database and Knowledge Base Systems 1998

52. 51 Data Model for XPath bib book publisherauthor.. Addison-WesleySerge Abiteboul The root The root element

53. 52 XPath: Simple Expressions Result: 1995 1998 Result: empty (there were no papers) /bib/book/year /bib/paper/year

54. 53 XPath: Restricted Kleene Closure Result: Serge Abiteboul Rick Hull Victor Vianu Jeffrey D. Ullman Result: Rick //author /bib//first-name

55. 54 Xpath: Text Nodes Result: Serge Abiteboul Jeffrey D. Ullman Rick Hull doesn’t appear because he has firstname, lastname Functions in XPath: –text() = matches the text value –node() = matches any node (= * or @* or text()) –name() = returns the name of the current tag /bib/book/author/text()

56. 55 Xpath: Wildcard Result: Rick Hull * Matches any element //author/*

57. 56 Xpath: Attribute Nodes Result: “55” @price means that price is has to be an attribute /bib/book/@price

58. 57 Xpath: Predicates Result: Rick Hull /bib/book/author[firstname]

59. 58 Xpath: More Predicates Result: … … /bib/book/author[firstname][address[//zip][city]]/lastname

60. 59 Xpath: More Predicates /bib/book[@price < “60”] /bib/book[author/@age < “25”] /bib/book[author/text()]

61. 60 Xpath: Summary bibmatches a bib element *matches any element /matches the root element /bibmatches a bib element under root bib/papermatches a paper in bib bib//papermatches a paper in bib, at any depth //papermatches a paper at any depth paper|bookmatches a paper or a book @pricematches a price attribute bib/book/@pricematches price attribute in book, in bib bib/book/[@price<“55”]/author/lastname matches…

62. 61 Comments on XPath? What’s good about it? What can’t it do that you want it to do? How does it compare, say, to SQL?

63. 62 XQuery Based on Quilt, which is based on XML-QL Uses XPath to express more complex queries

64. 63 FLWR (“Flower”) Expressions FOR... LET... WHERE... RETURN... FOR... LET... WHERE... RETURN...

65. 64 XQuery Find all book titles published after 1995: FOR $x IN document("bib.xml")/bib/book WHERE $x/year > 1995 RETURN { $x/title } FOR $x IN document("bib.xml")/bib/book WHERE $x/year > 1995 RETURN { $x/title } Result: abc def ghi

66. 65 XQuery Find book titles by the coauthors of “Database Theory”: FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN bib/book[author/text() = $x/text()]/title RETURN { $y/text() } FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN bib/book[author/text() = $x/text()]/title RETURN { $y/text() } Result: abc def ghi The answer will contain duplicates !

67. 66 XQuery Same as before, but eliminate duplicates: FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN distinct(bib/book[author/text() = $x/text()]/title) RETURN { $y/text() } FOR $x IN bib/book[title/text() = “Database Theory”]/author $y IN distinct(bib/book[author/text() = $x/text()]/title) RETURN { $y/text() } Result: abc def ghi distinct = a function that eliminates duplicates

68. 67 XQuery: Nesting For each author of a book by Morgan Kaufmann, list all books she published: FOR $a IN distinct(document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN { $a, FOR $t IN /bib/book[author=$a]/title RETURN $t } FOR $a IN distinct(document("bib.xml") /bib/book[publisher=“Morgan Kaufmann”]/author) RETURN { $a, FOR $t IN /bib/book[author=$a]/title RETURN $t }

69. 68 XQuery Jones abc def Smith ghi Jones abc def Smith ghi Result:

70. 69 XQuery FOR $x in expr -- binds $x to each value in the list expr LET $x = expr -- binds $x to the entire list expr –Useful for common subexpressions and for aggregations

71. 70 XQuery count = a (aggregate) function that returns the number of elms FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN { $p } FOR $p IN distinct(document("bib.xml")//publisher) LET $b := document("bib.xml")/book[publisher = $p] WHERE count($b) > 100 RETURN { $p }

72. 71 XQuery Find books whose price is larger than average: LET $a=avg( document("bib.xml") /bib/book/price) FOR $b in document("bib.xml") /bib/book WHERE $b/price > $a RETURN { $b } LET $a=avg( document("bib.xml") /bib/book/price) FOR $b in document("bib.xml") /bib/book WHERE $b/price > $a RETURN { $b } Let’s try to write this in SQL…

73. 72 XQuery Summary: FOR-LET-WHERE-RETURN = FLWR FOR/LET Clauses WHERE Clause RETURN Clause List of tuples Instance of Xquery data model

74. 73 FOR v.s. LET FOR Binds node variables  iteration LET Binds collection variables  one value

75. 74 FOR v.s. LET FOR $x IN document("bib.xml") /bib/book RETURN { $x } FOR $x IN document("bib.xml") /bib/book RETURN { $x } Returns:... LET $x IN document("bib.xml") /bib/book RETURN { $x } LET $x IN document("bib.xml") /bib/book RETURN { $x } Returns:...

76. 75 Collections in XQuery Ordered and unordered collections –/bib/book/author = an ordered collection –Distinct(/bib/book/author) = an unordered collection LET $a = /bib/book  $a is a collection $b/author  a collection (several authors...) RETURN { $b/author } Returns:...

77. 76 Collections in XQuery What about collections in expressions ? $b/price  list of n prices $b/price * 0.7  list of n numbers $b/price * $b/quantity  list of n x m numbers ?? $b/price * ($b/quant1 + $b/quant2)  $b/price * $b/quant1 + $b/price * $b/quant2 !!

78. 77 Sorting in XQuery FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text() }, FOR $b IN document("bib.xml")//book[publisher = $p] RETURN { $b/title, $b/price } SORTBY(price DESCENDING) SORTBY(name) FOR $p IN distinct(document("bib.xml")//publisher) RETURN { $p/text() }, FOR $b IN document("bib.xml")//book[publisher = $p] RETURN { $b/title, $b/price } SORTBY(price DESCENDING) SORTBY(name)

79. 78 If-Then-Else FOR $h IN //holding RETURN { $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author } SORTBY (title) FOR $h IN //holding RETURN { $h/title, IF $h/@type = "Journal" THEN $h/editor ELSE $h/author } SORTBY (title)

80. 79 Existential Quantifiers FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN { $b/title } FOR $b IN //book WHERE SOME $p IN $b//para SATISFIES contains($p, "sailing") AND contains($p, "windsurfing") RETURN { $b/title }

81. 80 Universal Quantifiers FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN { $b/title } FOR $b IN //book WHERE EVERY $p IN $b//para SATISFIES contains($p, "sailing") RETURN { $b/title }

82. 81 Other Stuff in XQuery BEFORE and AFTER –for dealing with order in the input FILTER –deletes some edges in the result tree Recursive functions –Currently: arbitrary recursion –Perhaps more restrictions in the future ?

83. 82 Final Comments on XML How are we going to process XML efficiently? –Special purpose XML engines, or –Add functionality to relational engines? Need to manage XML streams. Here, data management is much closer to other programming tasks.