Introduction Zachary G. Ives University of Pennsylvania CIS 650 – Implementing Data Management Systems January 10, 2005

2 Welcome to CIS 650, Database and Information Systems! Instructor: Zachary Ives,  576 Levine Hall North  Office hours: Tuesday, 2:30-3:30PM (before colloquium) Home page: Texts and readings:  Hellerstein and Stonebraker: Readings in Database Systems, 4 th ed.  (Should be available soon)  Supplementary papers (will be linked via schedule)

3 Course Format and Grading Very discussion-oriented; about one topic area per week or two  Readings in the text & other research papers – summaries/commentary on papers (20%) “Midterm report” (25%)  You’ll take one of the topics we’ve discussed and write a summary and synthesis paper  Graded for organization, clarity, grammar, etc. as well as content Project (50%) -- may choose to work in teams:  Implementation  Experimentation / validation  Project report (should be in the style of a research paper)  Brief (~15-minute) presentation Participation, discussion, intangibles (5%) At the end, you should be equipped to do research in this field, or to take ideas from databases and apply them to your field

4 So What Is This Course About? Not how to build an Oracle-driven Web site… … nor even how to build Oracle…

5 What Is Unique about Data Management?  It’s been said that databases and data management focus on scalability to huge volumes of data  What is it that makes this possible – and what makes the work interesting if NOT at huge scale?  Why are data management techniques useful in situations where scale isn’t the bottleneck?

6 The Key Principle: Data Independence  Most methods of programming don’t separate the logical and physical representations of data  The data structures, access methods, etc. are all given via interfaces!  The relational data model was the first model for data that is independent of its data structures and implementation

7 What Is Data Independence?  Codd points out that previous methods had:  Order dependence  Index dependence  Access path dependence  Still true in today’s Java/C#: what is the drawback?  What might you be able to do in removing those?

8 The Relational Data Model More than just tables!  True relations: sets of tuples  The only data representation a user/programmer “sees”  Explicit encoding of everything in values Additional integrity constraints  Key constraints, functional dependencies, … General and universal means of encoding everything!  (Semantics are pushed to queries) A secondary concept: views  Define virtual, derived relations that are always “live”  A way of encapsulating, abstracting data

9 Constraints and Normalization  Fundamental idea: we don’t want to build semantics into the data model, but we want to be able to encode certain constraints  Functional dependencies, key constraints, foreign-key constraints, multivalued dependencies, join dependencies, etc.  Allows limited data validation, plus opportunities for optimization  The theory of normalization (see CSE 330, CIS 550) makes use of known constraints  Idea: eliminate redundancy, in order to maintain consistency in the presence of updates  (Note that there’s no reason for normalization of data in views!)  Ergo, XML???

10 Relational Completeness (Plus Extensions): Declarativity What is special about relational query languages that makes them amenable to scalability?  Limited expressiveness – particularly when we consider conjunctive queries (even with recursion)  Guaranteed polytime execution in size of data  Can reason about containment, invert them, etc.  “Magic sets”  (What about XQuery’s Turing-completeness???)  Equivalence between relational calculus and algebra  Calculus  fully declarative, basis of query languages  Algebra  imperative but polytime, basis of runtime systems  Predictability of operations  cost models  Ability to supplement data with auxiliary structures for performance

11 Concurrency and Reliability (Generally requires full control)  Another key element of databases – ACID properties  Atomicity, Consistency, Isolation, Durability  Transaction : an atomic sequence of database actions (read/write) on data items (e.g. calendar entry)  Recoverability via a log: keeping track of all actions carried out by the database  How do distributed systems, Web services, service- oriented architectures, and the like affect these properties?

12 Other Data Models  Concepts from the relational data model have been adapted to form object-oriented data models (with classes and subclasses), XML models, etc.  But doesn’t this result in some loss of logical-physical independence?  GMAP and answering queries using views?

13 What Is a Data Management System?  Of course, there are traditional databases  The focus of most work in the past 25 years  “Tight loops” due to locally controlled data  Indexing, transactions, concurrency, recovery, optimization  But…

14 80% of the World’s Data is Not in Databases! Examples:  Scientific data (large images, complex programs that analyze the data)  Personal data  WWW and (some of it is stored in something resembling a DBMS)  Network traffic logs  Sensor data  Are there benefits to declarative techniques and data independence in tackling these issues?  XML is a great way to make this data available  Also need to deal with data we don’t control and can’t guarantee consistency over

15 An Example of Data Management with Heterogeneity: Data Integration A layer above heterogeneous sources, to combine them under a unified logical abstraction  Some of these are databases over which we have no control  Some must be accessed in special ways  Data integration system translates queries over mediated schema to the languages of the sources; converts answers to mediated schema XML “Mediated Schema”

16 Other Interesting Points Data streams and sensor data How do we process infinite amounts of data? Peer-to-peer architectures What’s the best way of finding data here? Personal information management Can we use integration-style concepts and a bit of AI to manage associations between our data? Web search What’s the back-end behind Google? Semantic Web How do we semantically interrelate data to build a better Web?

17 Layers of a Typical Data Management System API/GUI Optimizer Physical retrieval Exec. Engine Source Catalog Query Physical plan Pages RequestsData Pages Stats Schemas (Simplification!) Buffer Mgr Access Methods Data/etcRequests Data/etc Logging, recovery Red = logical Blue = physical

18 Query Answering in a Data Management System  Based on declarative query languages  Based on restricted first-order logic expressions over relations  Not procedural – defines constraints on the output  Converted into a query plan that exploits properties; run over the data by the query optimizer and query execution engine  Data may be local or remote  Data may be heterogeneous or homogeneous  Data sources may have different interfaces, access methods, etc.  Most common query languages:  SQL (based on tuple relational calculus)  Datalog (based on domain relational calculus, plus fixpoint)  XQuery (functional language; has an XML calculus core)

19 Processing the Query SELECT * FROM STUDENT, Takes, COURSE WHERE STUDENT.sid = Takes.sID AND Takes.cID = cid STUDENT Takes COURSE Merge Hash by cid Optimizer Execution Engine Storage Subsystem Web Server / UI / etc

20 DBMSs in the Real World  Big, mature relational databases  IBM, Oracle, Microsoft  “Middleware” above these  SAP, PeopleSoft, dozens of special-purpose apps  “Application servers”  Integration and warehousing systems  Current trends:  Web services; XML everywhere  Smarter, self-tuning systems  Stream systems

21 Our Agenda this Semester  Reading the canonical papers in the data management literature  Some are very systems-y  Some are very experimental  Some are highly algorithmic, complexity-oriented  Gaining an understanding of the principles of building systems to handle declarative queries over large volumes of data

22 For Next Time  Skim Codd if you haven’t already  Read the overview papers of the two first database systems:  Astrahan et al., pp  Wong et al. (skip Section 2; focus on pp. 200-)  Write a summary of your assigned paper and it to me at  Key question: how well did this system mesh with Codd’s relational model? (You may need to skim through other aspects of your assigned paper to help answer that question)