What Should a DBMS Do? Store large amounts of data Process queries efficiently Allow multiple users to access the database concurrently and safely. Provide durability of the data. How will we do all this??

Generic Architecture Query compiler Execution engine Index/record mgr. Buffer manager Storage manager storage User/ Application Query update Query execution plan Record, index requests Page commands Read/write pages

Query Optimization Purchase Person Buyer=name City=‘seattle’ phone>’ ’ buyer (Simple Nested Loops) Imperative query execution plan: SELECT S.sname FROM Purchase P, Person Q WHERE P.buyer=Q.name AND Q.city=‘seattle’ AND Q.phone > ‘ ’ Declarative SQL query Plan: Tree of R.A. ops, with choice of alg for each op. Ideally: Want to find best plan. Practically: Avoid worst plans! Goal: 

Alternate Plans Find names of people who bought telephony products Purchase Person Buyer=name Category=“telephony” buyer (hash join) Product prod=pname (hash join) Purchase Product Buyer=name Category=“telephony” buyer (hash join) Person prod=pname (hash join)   But what if we’re only looking for Bob’s purchases?

ACID ACID Properties A Atomicity: all actions of a transaction happen, or none happen. C Consistency: if a transaction is consistent, and the database starts from a consistent state, then it will end in a consistent state. I Isolation: the execution of one transaction is isolated from other transactions. D Durability: if a transaction commits, its effects persist in the database.

Problems with Transaction Processing Airline reservation system: Step 1: check if a seat is empty. Step 2: reserve the seat. Bad scenario: (but very common) Customer 1 - finds a seat empty Customer 2 - finds the same seat empty Customer 1 - reserves the seat. Customer 2 - reserves the seat. Customer 1 will not be happy; spends night in airport hotel.

The Memory Hierarchy Main Memory Disk Tape Volatile limited address spaces expensive average access time: nanoseconds 5-10 MB/S transmission rates 2-10 GB storage average time to access a block: msecs. Need to consider seek, rotation, transfer times. Keep records “close” to each other. 1.5 MB/S transfer rate 280 GB typical capacity Only sequential access Not for operational data Cache: access time 10 nano’s

Disk Space Manager Task: manage the location of pages on disk (page = block) Provides commands for: allocating and deallocating a page on disk reading and writing pages. Why not use the operating system for this task? Portability Limited size of address space May need to span several disk devices. Platters Spindle Disk head Arm movement Arm assembly Tracks Sector

Buffer Management in a DBMS Data must be in RAM for DBMS to operate on it! Table of pairs is maintained. LRU is not always good. DB MAIN MEMORY DISK disk page free frame Page Requests from Higher Levels BUFFER POOL choice of frame dictated by replacement policy

Buffer Manager Manages buffer pool: the pool provides space for a limited number of pages from disk. Needs to decide on page replacement policy. Enables the higher levels of the DBMS to assume that the needed data is in main memory. Why not use the Operating System for the task?? - DBMS may be able to anticipate access patterns - Hence, may also be able to perform prefetching - DBMS needs the ability to force pages to disk.

Record Formats: Fixed Length Information about field types same for all records in a file; stored in system catalogs. Finding i’th field requires scan of record. Note the importance of schema information! Base address (B) L1L2 L3L4 F1F2 F3F4 Address = B+L1+L2

Files of Records Page or block is OK when doing I/O, but higher levels of DBMS operate on records, and files of records. FILE : A collection of pages, each containing a collection of records. Must support: –insert/delete/modify record –read a particular record (specified using record id) –scan all records (possibly with some conditions on the records to be retrieved)

Cost Model for Our Analysis As a good approximation, we ignore CPU costs: –B: The number of data pages –R: Number of records per page –D: (Average) time to read or write disk page –Measuring number of page I/O’s ignores gains of pre-fetching blocks of pages; thus, even I/O cost is only approximated. *

Sorting Illustrates the difference in algorithm design when your data is not in main memory: –Problem: sort 1Gb of data with 1Mb of RAM. Arises in many places in database systems: –Data requested in sorted order (ORDER BY) –Needed for grouping operations –First step in sort-merge join algorithm –Duplicate removal –Bulk loading of B+-tree indexes.

2-Way Sort: Requires 3 Buffers Pass 1: Read a page, sort it, write it. –only one buffer page is used Pass 2, 3, …, etc.: – three buffer pages used. Main memory buffers INPUT 1 INPUT 2 OUTPUT Disk

Two-Way External Merge Sort Each pass we read + write each page in file. N pages in the file => the number of passes So total cost is: Idea: Divide and conquer: sort subfiles and merge Input file 1-page runs 2-page runs 4-page runs 8-page runs PASS 0 PASS 1 PASS 2 PASS 3 9 3,4 6,2 9,48,75,63,1 2 3,4 5,62,64,97,8 1,32 2,3 4,6 4,7 8,9 1,3 5,62 2,3 4,4 6,7 8,9 1,2 3,5 6 1,2 2,3 3,4 4,5 6,6 7,8

General External Merge Sort To sort a file with N pages using B buffer pages: –Pass 0: use B buffer pages. Produce sorted runs of B pages each. –Pass 2, …, etc.: merge B-1 runs. B Main memory buffers INPUT 1 INPUT B-1 OUTPUT Disk INPUT 2... * More than 3 buffer pages. How can we utilize them?

Cost of External Merge Sort Number of passes: Cost = 2N * (# of passes) E.g., with 5 buffer pages, to sort 108 page file: –Pass 0: = 22 sorted runs of 5 pages each (last run is only 3 pages) –Pass 1: = 6 sorted runs of 20 pages each (last run is only 8 pages) –Pass 2: 2 sorted runs, 80 pages and 28 pages –Pass 3: Sorted file of 108 pages

B: number of frames in the buffer pool; N: number of pages in relation. Number of Passes of External Sort