1. Indexes

2. Primary Indexes Dense Indexes Pointer to every record of a sequential file, (ordered by search key). Can make sense because records may be much bigger than key-­pointer pairs. - Fit index in memory, even if data file does not? - Faster search through index than data file? Sparse Indexes Key-­pointer pairs for only a subset of records, typically first in each block. Saves index space.

3. Dense Index

4. Sparse Index

5. Num. Example of Dense Index Data file = 1,000,000 tuples that fit 10 at a time into a block of 4096 bytes (4KB) 100,000 blocks  data file = 400 MB Index file: For typical values of key 30 Bytes, and pointer 8 Bytes, we can fit: 4096/(30+8)  100 (key,pointer) pairs in a block. So, we need 10,000 blocks = 40 MB for the index file. This might fit into available main memory.

6. Num. Example of Sparse Index Data file and block sizes as before One (key,pointer) record for the first record of every block  index file = 100,000 (key, pointer) pairs = 100,000 * 38Bytes = 1,000 blocks = 4MB If the index file could fit in main memory  1 disk I/O to find record given the key

7. Multiple levels of index An index maybe large; using an index on the index may improve the search time;  1. Build a (dense or sparse) index on the data file  2. Build a sparse index on the index file Ex: 2nd level index has 1,000 key-pointer pairs = 10 blocks = 40KB.  40 KB for the 2nd level index file fits in m.m.  2 disk I/O’s to find the record, given a key

8. Lookup for key K Issues: sparse vs. dense? 1.Find key K in dense index. 2.Find largest key  K in sparse index. Follow pointer. a) Dense: just follow. b) Sparse: follow to block, examine block. Dense vs. Sparse: Dense index can answer: ”Is there a record with key K?” Sparse index can not!

9. Cost of Lookup We can do binary search. log 2 (number of index blocks) I/O’s to find the desired record. All binary searches to the index will start at the block in the middle, then at 1/4 and 3/4 points, 1/8, 3/8, 5/8, 7/8. - So, if we store some of these blocks in main memory, I/O’s will be significantly lower. For our example: Binary search in the index may use at most log 2 10,000 = 14 blocks (or I/O’s) to find the record, given the key, … or much less if we store some of the index blocks as above.

10. Secondary Indexes A primary index is an index on a sorted file. - Such an index “controls” the placement of records to be “primary,” A secondary index is an index that does not control placement, surely not on a file sorted by its search key. - Sparse, secondary index makes no sense. - Usually, search key is not a “key.”

11. Indirect Buckets To avoid repeating keys in index, use a level of indirection, called buckets. Additional advantage: allows intersection of sets of records without looking at records themselves. Example Movies(title, year, length, studioName); secondary indexes on studioName and year. SELECT title FROM Movies WHERE studioName = 'Disney' AND year = 1995;

13. Inverted Indexes Similar (to secondary indexes) idea from information­retrieval community, but: - Record  document. - Search­key value of record  presence of a word in a document. Usually used with “buckets.”

14. Additional Information in Buckets We can extend bucket to include role, position of word, e.g. Type Position

15. Example of index selection Movie(title,year,length,studioName) Studio(name,address,president) Frequent query: Find movies given a studioName (Q1)  use primary/secondary index on Movie.studioName Also, if FAQs include: Find all movies produced by a studio, given the president of the studio (Q2) and Find movies made by a studio, given its address (Q3)  use a clustered file organization, with secondary indexes: one on president and another on address If we still want to answer Q1 efficiently, we now need a primary index on studioName

16. Operations with Indexes Deletions and insertions are problematic for flat indexes. Eventually, we need to reorganize entries and records. - E.g. insert 15 …that’s a messy approach.

17. B­Trees Generalizes multilevel index. Number of levels varies with size of data file, but is often 3. B+ tree = form we'll discuss. - All nodes have same format. - a B-tree of order n has n keys and n + 1 pointers. Useful for primary, secondary indexes, primary keys, nonkeys. Leaf has at least key-pointer pairs Interior nodes use at least pointers. Root has at least one key and two pointers

18. 13 723 3143 2 35 7 11 13 1719 23 29 31 3741 43 47 B-Tree of order 3 Recursive procedure: If we are at a leaf, look among the keys there. If the i-th key is K, the the i-th pointer will take us to the desired record. If we are at an internal node with keys K 1,K 2,…,K n, then if K<K 1 we follow the first pointer, if K 1  K<K 2 we follow the second pointer, and so on. Try to find a record with search key 40.

19. B­-Trees: A typical leaf and interior node 9581 57 To record with key 57 To record with key 81 To record with key 95 To next leaf in sequence Leaf 9581 57 To subtree with keys K<57 To subtree with keys 57  K<81 To subtree with keys 81  K<95 Interior Node To subtree with keys K  95 57, 81, and 95 are the least keys we can reach by via the corresponding pointers.

20. Operations in B-Tree Will illustrate with a dense index, but straightforward to generalize for sparse indices. Operations 1.Lookup 2.Insertion 3.Deletion

21. 13 723 3143 2 35 7 11 13 1719 23 29 31 3741 43 47 Lookup Recursive procedure: If we are at a leaf, look among the keys there. If the i-th key is K, the the i-th pointer will take us to the desired record. If we are at an internal node with keys K 1,K 2,…,K n, then if K<K 1 we follow the first pointer, if K 1  K<K 2 we follow the second pointer, and so on. Try to find a record with search key 40.

22. 13 723 3143 2 35 7 11 13 1719 23 29 31 3741 43 47 Range Queries Lookup key 10 Follow the “next-leaf” pointer collect all data-pointers (or retrieve the records) Until you reach key 25 Number of I/O’s depends on the range SELECT * FROM R WHERE R.key >= 10 AND R.key <= 25;

23. Insertion into B-Trees in words… We try to find a place for the new key in the appropriate leaf, and we put it there if there is room. If there is no room in the proper leaf, we “split” the leaf into two and divide the keys between the two new nodes, so each is half full or just over half full. - Split means “add a new block” The splitting of nodes at one level appears to the level above as if a new key-pointer pair needs to be inserted at that higher level. - We may thus apply this strategy to insert at the next level: if there is room, insert it; if not, split the parent node and continue up the tree. As an exception, if we try to insert into the root, and there is no room, then we split the root into two nodes and create a new root at the next higher level; - The new root has the two nodes resulting from the split as its children.

24. 13 723 3143 2 35 7 11 13 1719 23 29 31 3741 43 47 Insertion Try to insert a search key = 40. First, lookup for it, in order to find where to insert. It has to go here, but the node is full!

25. 13 723 3143 2 35 7 11 13 1719 23 29 31 37 43 47 40 41 Beginning of the insertion of key 40 Observe the new node and the redistribution of keys and pointers What’s the problem? No parent yet for the new node!

26. 13 723 3143 2 35 7 11 13 1719 23 29 31 37 43 47 40 41 Continuing of the Insertion of key 40 We must now insert a pointer to the new leaf into this node. We must also associate with this pointer the key 40, which is the least key reachable through the new leaf. But the node is full. Thus it too must split!

27. 13 723 31 2 35 7 11 13 1719 23 29 31 37 43 47 40 41 Completing of the Insertion of key 40 43 This is a new node. We have to redistribute 3 keys and 4 pointers. We leave three pointers in the existing node and give two pointers to the new node. 43 goes in the new node. But where the key 40 goes? 40 is the least key reachable via the new node.

28. 13 40 723 31 2 35 7 11 13 1719 23 29 31 37 43 47 40 41 Completing of the Insertion of key 40 43 It goes here! 40 is the least key reachable via the new node.

29. Structure of B-trees Degree n means that all nodes have space for n search keys and n+1 pointers Node = block Let - block size be 4096 Bytes, - key 4 Bytes, - pointer 8 Bytes. Let’s solve for n: 4n + 8(n+1)  4096  n  340 n = degree = order = fanout

30. Example n = 340, however a typical node has 255 keys At level 3 we have: 255 2 nodes, which means 255 3  16  2 20 records can be indexed. Suppose record = 1024 Bytes  we can index a file of size 16  2 20  2 10  16 GB If the root is kept in main memory accessing a record requires 3 disk I/O

31. Deletion from B-trees in words… If the node from which we delete still has the minimum no. of keys we’re done(possibly raise new key to parent) If the node from which we delete now has too few keys, then - If an adjacent sibling has more than the min. no. of key, borrow a key-pointer from that sibling. Adjust the keys in the parent - Else merge the underfull node and the node with the min. number of keys. Delete a key-pointer from the parent, and adjust the parent recursively

32. 13 723 3143 2 35 7 11 13 1719 23 29 31 3741 43 47 Deletion Suppose we delete key=7

33. 13 5 23 3143 2 3 5 11 13 1719 23 29 31 3741 43 47 Deletion (Raising a key to parent) This node is less than half full. So, it borrows key 5 from sibling.

34. 13 5 23 3143 2 3 5 11 13 1719 23 29 31 3741 43 47 Deletion Suppose we delete now key=11. No siblings with enough keys to borrow.

35. 13 23 3143 2 35 13 1719 23 29 31 3741 43 47 Deletion We merge, i.e. delete a block from the index. However, the parent ends up not having any key.

36. 23 13 3143 2 35 13 1719 23 29 31 3741 43 47 Deletion Parent: Borrow from sibling!

37. The slides from here to the end of the file are by Hector Garcia-Molina

38. Root B+Tree Examplen=3 100 120 150 180 30 3 5 11 30 35 100 101 110 120 130 150 156 179 180 200

39. CS 245Notes 439 Size of nodes:n+1 pointers n keys (fixed)

40. CS 245Notes 440 Don’t want nodes to be too empty Use at least Non-leaf:  (n+1)/2  pointers Leaf:  (n+1)/2  pointers to data

41. CS 245Notes 441 Full nodemin. node Non-leaf Leaf n=3 120 150 180 30 3 5 11 30 35 counts even if null

42. CS 245Notes 442 B+tree rulestree of order n (1) All leaves at same lowest level (balanced tree) (2) Pointers in leaves point to recordsexcept for “sequence pointer”

43. CS 245Notes 443 (3) Number of pointers/keys for B+tree Non-leaf (non-root) n+1n  (n+1)/ 2   (n+1)/ 2  - 1 Leaf (non-root) n+1n Rootn+1n11 Max Max Min Min ptrs keys ptrs  data keys  (n+ 1) / 2 

44. CS 245Notes 444 Insert into B+tree (a) simple case - space available in leaf (b) leaf overflow (c) non-leaf overflow (d) new root

45. CS 245Notes 445 (a) Insert key = 32 n=3 3 5 11 30 31 30 100

46. CS 245Notes 446 (a) Insert key = 32 n=3 3 5 11 30 31 30 100 32

47. CS 245Notes 447 (a) Insert key = 7 n=3 3 5 11 30 31 30 100

48. CS 245Notes 448 (a) Insert key = 7 n=3 3 5 11 30 31 30 100 3535 7

49. CS 245Notes 449 (a) Insert key = 7 n=3 3 5 11 30 31 30 100 3535 7 7

50. CS 245Notes 450 (c) Insert key = 160 n=3 100 120 150 180 150 156 179 180 200

51. CS 245Notes 451 (c) Insert key = 160 n=3 100 120 150 180 150 156 179 180 200 160 179

52. CS 245Notes 452 (c) Insert key = 160 n=3 100 120 150 180 150 156 179 180 200 180 160 179

53. CS 245Notes 453 (c) Insert key = 160 n=3 100 120 150 180 150 156 179 180 200 160 180 160 179

54. CS 245Notes 454 (d) New root, insert 45 n=3 10 20 30 123123 10 12 20 25 30 32 40

55. CS 245Notes 455 (d) New root, insert 45 n=3 10 20 30 123123 10 12 20 25 30 32 40 45

56. CS 245Notes 456 (d) New root, insert 45 n=3 10 20 30 123123 10 12 20 25 30 32 40 45 40

57. CS 245Notes 457 (d) New root, insert 45 n=3 10 20 30 123123 10 12 20 25 30 32 40 45 4030 new root

58. CS 245Notes 458 (a) Simple case - no example (b) Coalesce with neighbor (sibling) (c) Re-distribute keys (d) Cases (b) or (c) at non-leaf Deletion from B+tree

59. CS 245Notes 459 (b) Coalesce with sibling - Delete 50 10 40 100 10 20 30 40 50 n=4

60. CS 245Notes 460 (b) Coalesce with sibling - Delete 50 10 40 100 10 20 30 40 50 n=4 40

61. CS 245Notes 461 (c) Redistribute keys - Delete 50 10 40 100 10 20 30 35 40 50 n=4

62. CS 245Notes 462 (c) Redistribute keys - Delete 50 10 40 100 10 20 30 35 40 50 n=4 35

63. CS 245Notes 463 40 45 30 37 25 26 20 22 10 14 1313 10 2030 40 (d) Non-leaf coalese –Delete 37 n=4 25

64. CS 245Notes 464 40 45 30 37 25 26 20 22 10 14 1313 10 2030 40 (d) Non-leaf coalese –Delete 37 n=4 30 25

65. CS 245Notes 465 40 45 30 37 25 26 20 22 10 14 1313 10 2030 40 (d) Non-leaf coalese –Delete 37 n=4 40 30 25

66. CS 245Notes 466 40 45 30 37 25 26 20 22 10 14 1313 10 2030 40 (d) Non-leaf coalese –Delete 37 n=4 40 30 25 new root