1. CS 440 Database Management Systems Lecture 6: Data storage & access methods 1

2. Database System Implementation 2 Conceptual Design Physical Storage Schema Entity Relationship(ER) Model Relational Model Files and Indexes User Requirements

3. The advantage of RDBMS It separates logical level (schema) from physical level (implementation). Physical data independence – Users do not worry about how their data is stored and processes on the physical devices. – It is all SQL! – Their queries work over (almost) all RDBMS deployments. 3

4. 4 DBMS Architecture Query Executor Buffer Manager Storage Manager Storage Transaction Manager Logging & Recovery Lock Manager Buffers Lock Tables Main Memory User/Web Forms/Applications/DBA query transaction Query Optimizer Query Rewriter Query Parser Files & Access Methods Process manager

5. Challenges in physical level Processor: 10000 – 100000 MIPS Main memory: around 10 Gb/ sec. Secondary storage: higher capacity and durability Disk random access – Seek time + rotational latency + transfer time – Seek time: 4 ms - 15 ms! – Rotational latency: 2 ms – 7 ms! – Transfer time: at most 1000 Mb/ sec – Read, write in blocks. 5

6. Gloomy future: Moor’s law Speed of processors and cost and maximum capacity of storage increase exponentially over time. But storage (main and secondary) access time grows much more slowly. 6

7. Random access versus sequential access Disk random access : Seek time + rotational latency + transfer time. Disk sequential access: reading blocks next to each other – No seek time or rotational latency – Much faster than random access 7

8. Units of data on physical device Fields: data items Records Blocks Files 8

9. Fields Fixed size – Integer, Boolean, … Variable length – Varchar, … – Null terminated – Size at the beginning of the string 9

10. Records: sets of fields Schema – Number of fields, types of fields, order, … Fixed format and length – Record holds only the data items Variable format and length – Record holds fields and their size, type, … information Range of formats in between 10

11. Record header Pointer to the record schema ( record type) Record size Timestamp Other info … 11

12. Blocks Collection of records Reduces number of I/O access Different from OS blocks – Why should RDBMS manage its own blocks? It knows the access pattern better than OS. Separating records in a block – Fixed size records: no worry! – Markers between records – Keep record size information in records or block header. 12

13. Spanned versus un-spanned Un-spanned – Each records belongs to only one block Spanned – Records may be stored across multiple blocks – Saves space – The only way to deal with large records and fields: blob, image, … 13

14. Block header Data about block File, relation, DB IDs Block ID and type Record directory Pointer to free space Timestamp Other info … 14

15. Heap versus sorted files Heap files – There is not any order in the file – New blocks are inserted at the end of the file. Sorted files – Order blocks (and records) based on some key. – Physically contiguous or using links to the next blocks. 15

16. Average cost of data operations Insertion – Heap files are more efficient. – Overflow areas for sorted files. Search for a record or a range of records – Sorted files are more efficient. Deletion – Heap files are more efficient – Although we find the record faster in the sorted file. 16

17. Row versus column stores We have talked about row store – All fields of a record are stored together. 17

18. Row versus column stores We can store the fields in columns. – We can store SSNs implicitly. 18

19. Row versus column store Column store – Compact storage – Faster reads on data analysis and mining operations Row store – Faster writes – Faster reads for record access (OLTP) Further reading – Mike Stonebreaker, et al, “C-Store, a column oriented DBMS”, VLDB’05. 19

20. Access paths The methods that RDBMS uses to retrieve the data. Attribute value(s)  Tuple(s) 20

21. Types of search queries Point query over Beers(name, manf) Select * From Beers Where name = ‘Bud’; Range query over Sells(bar, beer, price) Select * From Sells Where price > 2 AND price < 10; 21

22. Types of access paths Full table scan – Heap files – Inefficient for both point and range queries. Sequential access – Sorted files – Efficient for both point and range queries. – Inefficient to maintain Middle ground? 22

23. Indexing An old idea 23

24. Index A data structure that speeds up selecting tuples in a relation based on some search keys. Search key – A subset of the attributes in a relation – May not be the same as the (primary) key Entries in an index – (k, r) – k is the search key. – r is the pointer to a record (record id). 24

25. Index Data file stores the table data. Index file stores the index data structure. Index file is smaller than the data file. Ideally, the index should fit in the main memory. 25 10 20 30 40 50 60 70 80 10 20 30 40 50 60 Data FileIndex File

26. Index categorizations Clustered vs. unclustered – Records are stored according to the index order. – Records are stored in another order, or not any order. Dense vs. sparse – Each record is pointed by an entry in the index. – Each block has an entry in the index. – Size versus time tradeoff. Primary vs. secondary – Primary key is the search key – Other attributes. 26

27. Index categorizations Clustered and dense 27 10 20 30 40 50 60 70 80 10 20 30 40 50 60 DATAINDEX

28. Index categorizations Clustered and sparse 28 10 30 50 70 90 110 10 20 30 40 50 60 DATAINDEX 70 80

29. Duplicate search keys Clustered and dense 29 10 20 30 40 50 60 10 20 30 DATAINDEX 40 50

30. Duplicate search keys Clustered and sparse: – Any problem? 30 10 20 40 50 60 10 20 30 DATAINDEX 40 50

31. Duplicate search keys Clustered and sparse: – Point to the lowest new search key in every block 31 10 20 30 40 50 10 20 30 DATAINDEX 40 50

32. Unclustered Index Dense / sparse? 32 10 20 30 40 30 10 20 30 10 20 DATAINDEX 10 40

33. Well known index structures B+ trees: – very popular Hash tables: – Not frequently used 33

34. B+ trees The index of a very large data file gets too large. How about building an index for the index file? A multi-level index, or a tree 34

35. B+ trees Degree (order) of the tree: d Each node (except root) stores [d, 2d] keys: 35 103294 [A, 10)[10, 32)[32, 94)[94, B) Non-leaf nodes 122832 122832 394165 Leaf nodes Records

36. 36 Example 60 19508090110 121317192130405052606572 12 1317 1921 304050 52 6065 72 d = 2

37. B+ tree tuning How to choose the value of d? – Each node should fit in a block. Example – Key value: 8 byte – Record pointer: 16 bytes – Block size: 4096 bytes – 2d * 8 + (2d + 1) * 16 <= 4096 – d <= 85 37

38. Retrieving tuples using B+ tree Point queries – Start from the root and follow the links to the leaf. Range queries – Find the lowest point in the range. – Then, follow the links between the nodes. The top levels are kept in the buffer pool. 38

39. B+ tree and index categories B+ tree index could be – Dense / sparse? – Clustered/ unclustered? 39

40. Inserting a new key Pick the proper leaf node and insert the key. If the node contains more than 2d keys, split the node and insert the extra node in the parent. – If leaf level, add K3 to the right node 40 K1K2K3K4K5 R0R1R2R3R4R5 K1K2 R0R1R2 K4K5 R3R4R5 (K3, ) parent

41. 41 Insertion 60 19508090110 121317192130405052606572 12 1317 1921 304050 52 6065 72 Insert K = 18

42. 42 Insertion 60 19508090110 12131718192130405052606572 12 13 17 1921 304050 52 6065 72 Insert K = 18 18

43. 43 Insertion 60 19508090110 121317185052606572 12 1317 1921 30 40 50 52 6065 72 Insert K= 20 1920213040 20 18

44. 44 Insertion 60 19508090110 121317185052606572 12 13 17 1921 30 40 50 52 6065 72 Need to split the node 1920213040 20 18

45. 45 Insertion 60 1921508090110 121317185052606572 12 13 17 19 21 30 40 50 52 6065 72 Split and update the parent node. What if we need to split the root? 20 1920 213040 18

46. 46 Deletion 60 1921508090110 121317185052606572 12 13 17 19 21 30 40 50 52 6065 72 Delete K = 21 20 1920 213040 18

47. 47 Deletion 60 1921508090110 121317185052606572 12 13 17 19 304050 52 6065 72 Note: K = 21 may still remain in the internal levels 20 1920 3040 18

48. 48 Deletion 60 1921508090110 121317185052606572 12 13 17 19 304050 52 6065 72 Delete K = 20 20 1920 3040 18

49. 49 Deletion 60 1921508090110 121317185052606572 12 13 17 19 304050 52 6065 72 We need to update the number of keys on the node: Borrow from siblings: rotate 19 3040 18

50. 50 Deletion 60 1921508090110 1213175052606572 12 13 17 19 304050 52 6065 72 We need to update the number of keys on the node: Borrow from siblings: rotate 1819 3040 18

51. 51 Deletion 60 1821508090110 1213175052606572 12 13 17 19 304050 52 6065 72 We need to update the number of keys on the node: Borrow from siblings: rotate 1819 3040 18

52. 52 Deletion 60 1821508090110 1213175052606572 12 13 17 19 304050 52 6065 72 What if we cannot borrow from siblings? Example: delete K = 30 1819 3040 18

53. 53 Deletion 60 1821508090110 1213175052606572 12 13 17 19 4050 52 6065 72 What if we cannot borrow from siblings? Merge with a sibling. 1819 40 18

54. 54 Deletion 60 1821508090110 1213175052606572 12 13 17 19 4050 52 6065 72 What if we cannot borrow from siblings? Merge siblings! 181940 18

55. 55 Deletion 60 1821508090110 1213175052606572 12 13 17 19 4050 52 6065 72 What to do with the dangling key and pointer? simply remove them 181940 18

56. 56 Deletion 60 18508090110 1213175052606572 12 13 17 19 4050 52 6065 72 Final tree 181940 18

57. Index creation CREATE TABLE Person(Name varchar(50), Pos int, Age int); CREATE INDEX Person_ID ON Person(ID); CLUSTER Person USING ON Person_ID; CREATE INDEX Pos_Age ON Person(Pos, Age); 57 Default is normally B-tree. Cluster Person_ID index Multi-attribute index

58. Index selection Let’s index every attribute on every table to speed up all queries! Indexes generally slow down data manipulation – INSERT, DELETE, UPDATE. 58

59. Index selection Given a query workload and a schema, find the set of indexes that optimize the execution. The query workload: – Queries and their frequencies. – Queries are both data retrieval (SELECT) and data manipulation (INSERT, UPDATE, DELETE). 59

60. Index selection Part of physical database design – File structure, indexing, tuning queries,… Physical database design may affect logical design! – Change the schema to run the queries faster 60

61. Index selection Generally a hard problem. RDBMS vendors provide wizards: – Started with AutoAdmin project for SQL Server – SQL Server/ Oracle Index Tuning Wizard – DB2 Index Advisor They try many configurations and pick the one that minimizes the time and overheads. 61