SLIDE 1IS 202 – FALL 2006 Prof. Ray Larson UC Berkeley SIMS SIMS 202: Information Organization and Retrieval Introduction to Database Design

SLIDE 2IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 3IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 4IS 202 – FALL 2006 What is a Database?

SLIDE 5IS 202 – FALL 2006 Database A Database is a collection of stored operational data used by the application systems of some particular enterprise (C.J. Date) –Paper “Databases” Still contain a large portion of the world’s knowledge –File-Based Data Processing Systems Early batch processing of (primarily) business data –Database Management Systems (DBMS)

SLIDE 6IS 202 – FALL 2006 Why DBMS? History –50’s and 60’s all applications were custom built for particular needs –File based –Many similar/duplicative applications dealing with collections of business data –Early DBMS were extensions of programming languages – E.F. Codd and the Relational Model – Ashton-Tate and first Microcomputer DBMS

SLIDE 7IS 202 – FALL 2006 From File Systems to DBMS Problems with file processing systems –Inconsistent data –Inflexibility –Limited data sharing –Poor enforcement of standards –Excessive program maintenance

SLIDE 8IS 202 – FALL 2006 DBMS Benefits Minimal data redundancy Consistency of data Integration of data Sharing of data Ease of application development Uniform security, privacy, and integrity controls Data accessibility and responsiveness Data independence Reduced program maintenance

SLIDE 9IS 202 – FALL 2006 Terms and Concepts A “database application” is an application program (or set of related programs) that is used to perform a series of database activities: –Create Add new data to the database –Read Read current data from the database –Update Update or modify current database data –Delete Remove current On behalf of database users

SLIDE 10IS 202 – FALL 2006 Terms and Concepts Enterprise –Organization Entity –Person, Place, Thing, Event, Concept... Attributes –Data elements (facts) about some entity –Also sometimes called fields or items or domains Data values –Instances of a particular attribute for a particular entity

SLIDE 11IS 202 – FALL 2006 Terms and Concepts Key –An attribute or set of attributes used to identify or locate records in a file Primary Key –An attribute or set of attributes that uniquely identifies each record in a file

SLIDE 12IS 202 – FALL 2006 Terms and Concepts Models –(1) Levels or views of the Database Conceptual, logical, physical –(2) DBMS types Relational, Hierarchic, Network, Object-Oriented, Object-Relational

SLIDE 13IS 202 – FALL 2006 Models (1) Conceptual Model Logical Model External Model Conceptual requirements Conceptual requirements Conceptual requirements Conceptual requirements Application 1 Application 2Application 3Application 4 Application 2 Application 3 Application 4 External Model External Model External Model Internal Model More later on this…

SLIDE 14IS 202 – FALL 2006 Data Models(2): History Hierarchical Model (1960’s and 1970’s) –Similar to data structures in programming languages Books (id, title) PublisherSubjects Authors (first, last)

SLIDE 15IS 202 – FALL 2006 Data Models(2): History Network Model (1970’s) –Provides for single entries of data and navigational “links” through chains of data. SubjectsBooks Authors Publishers

SLIDE 16IS 202 – FALL 2006 Data Models(2): History Relational Model (1980’s) –Provides a conceptually simple model for data as relations (typically considered “tables”) with all data visible

SLIDE 17IS 202 – FALL 2006 Data Models(2): History Object Oriented Data Model (1990’s) –Encapsulates data and operations as “Objects” Books (id, title) PublisherSubjects Authors (first, last)

SLIDE 18IS 202 – FALL 2006 Data Models(2): History Object-Relational Model (1990’s) –Combines the well-known properties of the Relational Model with such OO features as: User-defined datatypes User-defined functions Inheritance and sub-classing All of the major enterprise DBMS systems are now Object-Relational or incorporate Object-Relational features

SLIDE 19IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 20IS 202 – FALL 2006 Database System Life Cycle Growth, Change, & Maintenance 6 Operations 5 Integration 4 Design 1 Conversion 3 Physical Creation 2

SLIDE 21IS 202 – FALL 2006 Design (more later) Determination of the needs of the organization –That is: “what are the business rules of this organization”, what information does it use and need? Development of the Conceptual Model of the database –In Database design this is typically done with Entity- Relationship diagramming techniques Construction of a Data Dictionary Development of the Logical Model

SLIDE 22IS 202 – FALL 2006 Physical Creation Development of the Physical Model of the Database –Data formats and types –Determination of indexes, etc. Load a prototype database and test Determine and implement security, privacy and access controls Determine and implement integrity constraints

SLIDE 23IS 202 – FALL 2006 Conversion Convert existing data sets and applications to use the new database –May need programs, conversion utilities to convert old data to new formats

SLIDE 24IS 202 – FALL 2006 Integration Overlaps with Phase 3 Integration of converted applications and new applications into the new database

SLIDE 25IS 202 – FALL 2006 Operations All applications run full-scale Privacy, security, access control must be in place Recovery and Backup procedures must be established and used

SLIDE 26IS 202 – FALL 2006 Growth, Change, and Maintenance Change is a way of life –Applications, data requirements, reports, etc. will all change as new needs and requirements are found –The Database and applications and will need to be modified to meet the needs of changes

SLIDE 27IS 202 – FALL 2006 Another View of the Life Cycle Operations 5 Conversion 3 Physical Creation 2 Growth, Change 6 Integration 4 Design 1

SLIDE 28IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 29IS 202 – FALL 2006 Database design process Conceptual Model Logical Model External Model Conceptual requirements Conceptual requirements Conceptual requirements Conceptual requirements Application 1 Application 2Application 3Application 4 Application 2 Application 3 Application 4 External Model External Model External Model Internal Model

SLIDE 30IS 202 – FALL 2006 Conceptual Design Conceptual Model –Merge the collective needs of all applications –Determine what Entities are being used Some object about which information is to maintained –What are the Attributes of those entities? Properties or characteristics of the entity What attributes uniquely identify the entity –What are the Relationships between entities How the entities interact with each other?

SLIDE 31IS 202 – FALL 2006 Developing a Conceptual Model Overall view of the database that integrates all the needed information discovered during the requirements analysis Elements of the Conceptual Model are represented by diagrams, Entity-Relationship or ER Diagrams, that show the meanings and relationships of those elements independent of any particular database systems or implementation details Can also be represented using other modeling tools (such as UML)

SLIDE 32IS 202 – FALL 2006 Database Design Process Conceptual Model Logical Model External Model Conceptual requirements Conceptual requirements Conceptual requirements Conceptual requirements Application 1 Application 2Application 3Application 4 Application 2 Application 3 Application 4 External Model External Model External Model Internal Model

SLIDE 33IS 202 – FALL 2006 Logical Design Logical Model –How is each entity and relationship represented in the Data Model of the DBMS Hierarchic? Network? Relational? Object-Oriented?

SLIDE 34IS 202 – FALL 2006 Database Design Process Conceptual Model Logical Model External Model Conceptual requirements Conceptual requirements Conceptual requirements Conceptual requirements Application 1 Application 2Application 3Application 4 Application 2 Application 3 Application 4 External Model External Model External Model Internal Model

SLIDE 35IS 202 – FALL 2006 Physical Design Internal Model –Choices of index file structure –Choices of data storage formats –Choices of disk layout

SLIDE 36IS 202 – FALL 2006 Database Design Process Conceptual Model Logical Model External Model Conceptual requirements Conceptual requirements Conceptual requirements Conceptual requirements Application 1 Application 2Application 3Application 4 Application 2 Application 3 Application 4 External Model External Model External Model Internal Model

SLIDE 37IS 202 – FALL 2006 Database Application Design External Model –User views of the integrated database –Making the old (or updated) applications work with the new database design

SLIDE 38IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 39IS 202 – FALL 2006 ER Diagrams You may have already seen a UML form of one…

SLIDE 40IS 202 – FALL 2006 ER Diagrams Entity-Relationship (ER) diagrams are concerned with the entities involved in the organization/database and the relationships between those entities (as well as the attributes of each entity

SLIDE 41IS 202 – FALL 2006 Entity An Entity is an object in the real world (or even imaginary worlds) about which we want or need to maintain information –Persons (e.g.: customers in a business, employees, authors) –Things (e.g.: purchase orders, meetings, parts, companies) Employee

SLIDE 42IS 202 – FALL 2006 Attributes Attributes are the significant properties or characteristics of an entity that help identify it and provide the information needed to interact with it or use it (this is the Metadata for the entities) Employee Last Middle First Name SSN Age Birthdate Projects

SLIDE 43IS 202 – FALL 2006 Relationships Relationships are the associations between entities They can involve one or more entities and belong to particular relationship types

SLIDE 44IS 202 – FALL 2006 Relationships Class Attends Student Part Supplies project parts Supplier Project

SLIDE 45IS 202 – FALL 2006 Types of Relationships Concerned only with cardinality of relationship Truck Assigned EmployeeProject Assigned EmployeeProject Assigned Employee 11 n n 1 m Chen ER notation

SLIDE 46IS 202 – FALL 2006 Other Notations Truck Assigned EmployeeProject Assigned EmployeeProject Assigned Employee “Crow’s Foot”

SLIDE 47IS 202 – FALL 2006 More Complex Relationships Project Evaluation Employee Manager 1/n/n 1/1/1 n/n/1 Project Assigned Employee 4(2-10) 1 SSNProjectDate Manages Employee Manages Is Managed By 1 n

SLIDE 48IS 202 – FALL 2006 Supertype and Subtype Entities Clerk Is one of Sales-rep Invoice Other Employee Sold Manages

SLIDE 49IS 202 – FALL 2006 Many to Many Relationships Employee Project Is Assigned Project Assignment Assigned SSN Proj# SSN Proj# Hours

SLIDE 50IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 51IS 202 – FALL 2006 Relational Algebra Operations Restrict Project Product Union Intersect Difference Join Divide

SLIDE 52IS 202 – FALL 2006 Restrict Extracts specified tuples (rows) from a specified relation (table) –Restrict is AKA “Select”

SLIDE 53IS 202 – FALL 2006 Project Extracts specified attributes(columns) from a specified relation.

SLIDE 54IS 202 – FALL 2006 Join Builds a relation from two specified relations consisting of all possible concatenated pairs, one from each of the two relations, such that in each pair the two tuples satisfy some condition. (E.g., equal values in a given col.) A1 B1 A2 B1 A3 B2 B1 C1 B2 C2 B3 C3 A1 B1 C1 A2 B1 C1 A3 B2 C2 (Natural or Inner) Join

SLIDE 55IS 202 – FALL 2006 ER Diagram: Acme Widget Co. Contains Part Part#Count Price Customer Quantity Orders Cust# Invoice Writes Sales-Rep Invoice# Sales Rep# Line-Item Contains Part# Invoice# Cust# Hourly Employee ISA Emp# Wage

SLIDE 56IS 202 – FALL 2006 Join Items for Relational DB Line_itemParts Customer Invoice

SLIDE 57IS 202 – FALL 2006 Relational Operations What is the name of the customer who ordered Large Red Widgets? –Restrict “large red widget” row from Part as temp1 –Join temp1 with Line-item on Part # as temp2 –Join temp2 with Invoice on Invoice # as temp3 –Join temp3 with Customer on cust # as temp4 –Project Company from temp4 as answer

SLIDE 58IS 202 – FALL 2006 SQL Database Definition and Querying –Can be used as an interactive query language –Can be imbedded in programs Relational Calculus combines Restrict, Project and Join operations in a single command: SELECT

SLIDE 59IS 202 – FALL 2006 SELECT Syntax: SELECT [DISTINCT] attr1, attr2,…, attr3 FROM rel1 r1, rel2 r2,… rel3 r3 WHERE condition1 {AND | OR} condition2 ORDER BY attr1 [DESC], attr3 [DESC]

SLIDE 60IS 202 – FALL 2006 SQL SELECT SELECT c.COMPANY FROM Customer c, Parts p, Invoice i, Line_Items z WHERE c.Cust# = i.Cust# AND i.Invoice# = z.Invoice# AND z.Part# = p.Part# AND p.Name = ‘large red widget’;

SLIDE 61IS 202 – FALL 2006 Lecture Overview Databases and Database Design Database Life Cycle Database Design ER Diagrams Relational Model Operations Normalization

SLIDE 62IS 202 – FALL 2006 Normalization Normalization theory is based on the observation that relations with certain properties are more effective in inserting, updating and deleting data than other sets of relations containing the same data Normalization is a multi-step process beginning with an “unnormalized” relation

SLIDE 63IS 202 – FALL 2006 Normal Forms First Normal Form (1NF) Second Normal Form (2NF) Third Normal Form (3NF) Boyce-Codd Normal Form (BCNF) Fourth Normal Form (4NF) Fifth Normal Form (5NF)

SLIDE 64IS 202 – FALL 2006 Normalization Boyce- Codd and Higher Functional dependency of nonkey attributes on the primary key - Atomic values only Full Functional dependency of nonkey attributes on the primary key No transitive dependency between nonkey attributes All determinants are candidate keys - Single multivalued dependency

SLIDE 65IS 202 – FALL 2006 Unnormalized Relations First step in normalization is to convert the data into a two-dimensional table In unnormalized relations data can repeat within a column (The following is a highly contrived example that has only a very vague resemblance to the implementation of the Phone/Photo project database from IS202 in 2004 …)

SLIDE 66IS 202 – FALL 2006 Unnormalized Relations

SLIDE 67IS 202 – FALL 2006 First Normal Form To move to First Normal Form a relation must contain only atomic values at each row and column –No repeating groups –A column or set of columns is called a Candidate Key when its values can uniquely identify the row in the relation

SLIDE 68IS 202 – FALL 2006 First Normal Form

SLIDE 69IS 202 – FALL NF Storage Anomalies Insertion: A new person has not yet taken a picture -- hence no Picture # -- Since Picture # is part of the key we can’t insert Insertion: If a Person is are known and likely to be photographed, but haven’t been yet -- there is be no way to include that person in the database Update: If a Person changes status (e.g. Mary Jones becomes a Student) we have to change multiple rows in the database Deletion (type 1): Deleting a Person record may also delete all info about People in the pictures Deletion (type 2): When there are functional dependencies (like Object and Object_features) changing one item eliminates other information

SLIDE 70IS 202 – FALL 2006 Second Normal Form A relation is said to be in Second Normal Form when every nonkey attribute is fully functionally dependent on the primary key –That is, every nonkey attribute needs the full primary key for unique identification

SLIDE 71IS 202 – FALL 2006 Second Normal Form Person Table

SLIDE 72IS 202 – FALL 2006 Second Normal Form People Table

SLIDE 73IS 202 – FALL 2006 Second Normal Form Picture Table

SLIDE 74IS 202 – FALL NF Storage Anomalies Removed Insertion: Can now enter new Persons who haven’t yet taken pictures Insertion: Can now enter People who haven’t been photographed Deletion (type 1): If Charles Brown withdraws his photos the corresponding tuples from Person and Picture tables can be deleted without losing information on David Rosen Update: If John White takes a third picture, and has changed status (e.g., graduate), we only need to change the Person table in one place

SLIDE 75IS 202 – FALL NF Storage Anomalies Insertion: Cannot enter the fact that a particular object has a particular feature unless it is associated with a particular picture Deletion: If John White describes some other object that Beth Little has while shopping, we lose the fact that the bookbag is blue Update: If the features of an object change change we have to update multiple occurrences of object features

SLIDE 76IS 202 – FALL 2006 Third Normal Form A relation is said to be in Third Normal Form if there are no transitive functional dependencies between nonkey attributes –When one nonkey attribute can be determined with one or more nonkey attributes there is said to be a transitive functional dependency The Obect_Feature column in the Picture table is determined by the Object –Object_Feature is transitively functionally dependent on Object so Picture is not 3NF

SLIDE 77IS 202 – FALL 2006 Third Normal Form Picture Table

SLIDE 78IS 202 – FALL 2006 Third Normal Form Object Table

SLIDE 79IS 202 – FALL NF Storage Anomalies Removed Insertion: We can now enter the fact that an object has a particular feature Deletion: If John White describes some other object that Beth Little has while shopping, we don’t lose the fact that the bookbag is blue Update: The features for each object appear only once

SLIDE 80IS 202 – FALL 2006 Boyce-Codd Normal Form Most 3NF relations are also BCNF relations A 3NF relation is NOT in BCNF if: –Candidate keys in the relation are composite keys (they are not single attributes) –There is more than one candidate key in the relation, and –The keys are not disjoint, that is, some attributes in the keys are common

SLIDE 81IS 202 – FALL 2006 Most 3NF Relations Are Also BCNF – Is This One?

SLIDE 82IS 202 – FALL 2006 BCNF Relations

SLIDE 83IS 202 – FALL 2006 Additional Issues Why separate Person and People? –They are really all People/Persons in different roles Shouldn’t a picture have a unique ID regardless of Who is in it? Can’t we have multiple people in the same picture, multiple objects, etc.? Can’t objects have multiple characteristics?

SLIDE 84IS 202 – FALL 2006 BCNF Relations

SLIDE 85IS 202 – FALL 2006 BCNF Added Capabilities Can now have a picture with no (identified) people in it Can have multiple objects, activities, and people associated with each picture

SLIDE 86IS 202 – FALL 2006 Fourth Normal Form Any relation is in Fourth Normal Form if it is BCNF and any multivalued dependencies are trivial Eliminate non-trivial multivalued dependencies by projecting into simpler tables

SLIDE 87IS 202 – FALL 2006 Fifth Normal Form A relation is in 5NF if every join dependency in the relation is implied by the keys of the relation Implies that relations that have been decomposed in previous NF can be recombined via natural joins to recreate the original relation

SLIDE 88IS 202 – FALL 2006 Fifth Normal Form Relations People Table

SLIDE 89IS 202 – FALL 2006 Normalizing to Death Normalization splits database information across multiple tables To retrieve complete information from a normalized database, the JOIN operation must be used JOIN tends to be expensive in terms of processing time, and very large joins are very expensive