1. CS 277 – Spring 2002Notes 41 CS 277: Database System Implementation Notes 4: Indexing Arthur Keller

2. CS 277 – Spring 2002Notes 42 Indexing & Hashing value Chapter 4  value record

3. CS 277 – Spring 2002Notes 43 Topics Conventional indexes B-trees Hashing schemes

4. CS 277 – Spring 2002Notes 44 Sequential File 20 10 40 30 60 50 80 70 100 90

5. CS 277 – Spring 2002Notes 45 Sequential File 20 10 40 30 60 50 80 70 100 90 Dense Index 10 20 30 40 50 60 70 80 90 100 110 120

6. CS 277 – Spring 2002Notes 46 Sequential File 20 10 40 30 60 50 80 70 100 90 Sparse Index 10 30 50 70 90 110 130 150 170 190 210 230

7. CS 277 – Spring 2002Notes 47 Sequential File 20 10 40 30 60 50 80 70 100 90 Sparse 2nd level 10 30 50 70 90 110 130 150 170 190 210 230 10 90 170 250 330 410 490 570

8. CS 277 – Spring 2002Notes 48 Comment: {FILE,INDEX} may be contiguous or not (blocks chained)

9. CS 277 – Spring 2002Notes 49 Question: Can we build a dense, 2nd level index for a dense index?

10. CS 277 – Spring 2002Notes 410 Notes on pointers: (1) Block pointer (sparse index) can be smaller than record pointer BP RP

11. CS 277 – Spring 2002Notes 411 (2) If file is contiguous, then we can omit pointers (i.e., compute them) Notes on pointers:

12. CS 277 – Spring 2002Notes 412 K1 K3 K4 K2 R1R2R3R4 say: 1024 B per block if we want K3 block: get it at offset (3-1)1024 = 2048 bytes

13. CS 277 – Spring 2002Notes 413 Sparse vs. Dense Tradeoff Sparse: Less index space per record can keep more of index in memory Dense: Can tell if any record exists without accessing file (Later: –sparse better for insertions –dense needed for secondary indexes)

14. CS 277 – Spring 2002Notes 414 Terms Index sequential file Search key (  primary key) Primary index (on Sequencing field) Secondary index Dense index (all Search Key values in) Sparse index Multi-level index

15. CS 277 – Spring 2002Notes 415 Next: Duplicate keys Deletion/Insertion Secondary indexes

16. CS 277 – Spring 2002Notes 416 Duplicate keys 10 20 10 30 20 30 45 40

17. CS 277 – Spring 2002Notes 417 10 20 10 30 20 30 45 40 10 20 30 10 20 10 30 20 30 45 40 10 20 30 Dense index, one way to implement? Duplicate keys

18. CS 277 – Spring 2002Notes 418 10 20 10 30 20 30 45 40 10 20 30 40 Dense index, better way? Duplicate keys

19. CS 277 – Spring 2002Notes 419 10 20 10 30 20 30 45 40 10 20 30 Sparse index, one way? Duplicate keys careful if looking for 20 or 30!

20. CS 277 – Spring 2002Notes 420 10 20 10 30 20 30 45 40 10 20 30 Sparse index, another way? Duplicate keys – place first new key from block should this be 40?

21. CS 277 – Spring 2002Notes 421 Duplicate values, primary index Index may point to first instance of each value only File Index Summary a a a b 

22. CS 277 – Spring 2002Notes 422 Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150

23. CS 277 – Spring 2002Notes 423 Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 – delete record 40

24. CS 277 – Spring 2002Notes 424 Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 – delete record 30 40

25. CS 277 – Spring 2002Notes 425 Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 – delete records 30 & 40 50 70

26. CS 277 – Spring 2002Notes 426 Deletion from dense index 20 10 40 30 60 50 80 70 10 20 30 40 50 60 70 80

27. CS 277 – Spring 2002Notes 427 Deletion from dense index 20 10 40 30 60 50 80 70 10 20 30 40 50 60 70 80 – delete record 30 40

28. CS 277 – Spring 2002Notes 428 Insertion, sparse index case 20 1030 50 4060 10 30 40 60

29. CS 277 – Spring 2002Notes 429 Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 34 34 our lucky day! we have free space where we need it!

30. CS 277 – Spring 2002Notes 430 Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 15 15 20 30 20 Illustrated: Immediate reorganization Variation: – insert new block (chained file) – update index

31. CS 277 – Spring 2002Notes 431 Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 25 25 overflow blocks (reorganize later...)

32. CS 277 – Spring 2002Notes 432 Insertion, dense index case Similar Often more expensive...

33. CS 277 – Spring 2002Notes 433 Secondary indexes Sequence field 50 30 70 20 40 80 10 100 60 90

34. CS 277 – Spring 2002Notes 434 Secondary indexes Sequence field 50 30 70 20 40 80 10 100 60 90 Sparse index 30 20 80 100 90... does not make sense!

35. CS 277 – Spring 2002Notes 435 Secondary indexes Sequence field 50 30 70 20 40 80 10 100 60 90 Dense index 10 20 30 40 50 60 70... 10 50 90... sparse high level

36. CS 277 – Spring 2002Notes 436 With secondary indexes: Lowest level is dense Other levels are sparse Also: Pointers are record pointers (not block pointers; not computed)

37. CS 277 – Spring 2002Notes 437 Duplicate values & secondary indexes 10 20 40 20 40 10 40 10 40 30

38. CS 277 – Spring 2002Notes 438 Duplicate values & secondary indexes 10 20 40 20 40 10 40 10 40 30 10 20 30 40... one option... Problem: excess overhead! disk space search time

39. CS 277 – Spring 2002Notes 439 Duplicate values & secondary indexes 10 20 40 20 40 10 40 10 40 30 10 another option... 403020 Problem: variable size records in index!

40. CS 277 – Spring 2002Notes 440 Duplicate values & secondary indexes 10 20 40 20 40 10 40 10 40 30 10 20 30 40 50 60... Another idea (suggested in class): Chain records with same key? Problems: Need to add fields to records Need to follow chain to know records

41. CS 277 – Spring 2002Notes 441 Duplicate values & secondary indexes 10 20 40 20 40 10 40 10 40 30 10 20 30 40 50 60... buckets

42. CS 277 – Spring 2002Notes 442 Why “bucket” idea is useful IndexesRecords Name: primary EMP (name,dept,floor,...) Dept: secondary Floor: secondary

43. CS 277 – Spring 2002Notes 443 Query: Get employees in (Toy Dept) ^ (2nd floor) Dept. indexEMP Floor index Toy 2nd  Intersect toy bucket and 2nd Floor bucket to get set of matching EMP’s

44. CS 277 – Spring 2002Notes 444 This idea used in text information retrieval Documents...the cat is fat......was raining cats and dogs......Fido the dog... Inverted lists cat dog

45. CS 277 – Spring 2002Notes 445 IR QUERIES Find articles with “cat” and “dog” Find articles with “cat” or “dog” Find articles with “cat” and not “dog” Find articles with “cat” in title Find articles with “cat” and “dog” within 5 words

46. CS 277 – Spring 2002Notes 446 Common technique: more info in inverted list cat Title5 100 Author10 Abstract57 Title12 d3d3 d2d2 d1d1 dog type position location

47. CS 277 – Spring 2002Notes 447 Posting: an entry in inverted list. Represents occurrence of term in article Size of a list:1Rare words or (in postings) miss-spellings 10 6 Common words Size of a posting: 10-15 bits (compressed)

48. CS 277 – Spring 2002Notes 448 IR DISCUSSION Stop words Truncation Thesaurus Full text vs. Abstracts Vector model

49. CS 277 – Spring 2002Notes 449 Vector space model w1 w2 w3 w4 w5 w6 w7 … DOC = Query= PRODUCT = 1 + ……. = score

50. CS 277 – Spring 2002Notes 450 Tricks to weigh scores + normalize e.g.: Match on common word not as useful as match on rare words...

51. CS 277 – Spring 2002Notes 451 How to process VS. Queries? w1 w2 w3 w4 w5 w6 … Q =

52. CS 277 – Spring 2002Notes 452 Try Altavista, Excite, Infoseek, Lycos...

53. CS 277 – Spring 2002Notes 453 Summary so far Conventional index –Basic Ideas: sparse, dense, multi-level… –Duplicate Keys –Deletion/Insertion –Secondary indexes –Buckets of Postings List

54. CS 277 – Spring 2002Notes 454 Conventional indexes Advantage: - Simple - Index is sequential file good for scans Disadvantage: - Inserts expensive, and/or - Lose sequentiality & balance

55. CS 277 – Spring 2002Notes 455 ExampleIndex (sequential) continuous free space 10 20 30 40 50 60 70 80 90 39 31 35 36 32 38 34 33 overflow area (not sequential)

56. CS 277 – Spring 2002Notes 456 Outline: Conventional indexes B-Trees  NEXT Hashing schemes

57. CS 277 – Spring 2002Notes 457 NEXT: Another type of index –Give up on sequentiality of index –Try to get “balance”

58. CS 277 – Spring 2002Notes 458 Root B+Tree Examplen=3 100 120 150 180 30 3 5 11 30 35 100 101 110 120 130 150 156 179 180 200

59. CS 277 – Spring 2002Notes 459 Sample non-leaf to keysto keysto keys to keys < 5757  k<8181  k<95  95 57 81 95

60. CS 277 – Spring 2002Notes 460 Sample leaf node: From non-leaf node to next leaf in sequence 57 81 95 To record with key 57 To record with key 81 To record with key 85

61. CS 277 – Spring 2002Notes 461 In textbook’s notationn=3 Leaf: Non-leaf: 30 35 30 35 30

62. CS 277 – Spring 2002Notes 462 Size of nodes:n+1 pointers n keys (fixed)

63. CS 277 – Spring 2002Notes 463 Don’t want nodes to be too empty Use at least Non-leaf:  (n+1)/2  pointers Leaf:  (n+1)/2  pointers to data

64. CS 277 – Spring 2002Notes 464 Full nodemin. node Non-leaf Leaf n=3 120 150 180 30 3 5 11 30 35 counts even if null

65. CS 277 – Spring 2002Notes 465 B+tree rulestree of order n (1) All leaves at same lowest level (balanced tree) (2) Pointers in leaves point to records except for “sequence pointer”

66. CS 277 – Spring 2002Notes 466 (3) Number of pointers/keys for B+tree Non-leaf (non-root) n+1n  (n+1)/ 2   (n+1)/ 2  - 1 Leaf (non-root) n+1n Rootn+1n11 Max Max Min Min ptrs keys ptrs  data keys  (n+ 1) / 2 

67. CS 277 – Spring 2002Notes 467 Insert into B+tree (a) simple case –space available in leaf (b) leaf overflow (c) non-leaf overflow (d) new root

68. CS 277 – Spring 2002Notes 468 (a) Insert key = 32 n=3 3 5 11 30 31 30 100 32

69. CS 277 – Spring 2002Notes 469 (a) Insert key = 7 n=3 3 5 11 30 31 30 100 3535 7 7

70. CS 277 – Spring 2002Notes 470 (c) Insert key = 160 n=3 100 120 150 180 150 156 179 180 200 160 180 160 179

71. CS 277 – Spring 2002Notes 471 (d) New root, insert 45 n=3 10 20 30 123123 10 12 20 25 30 32 40 45 4030 new root

72. CS 277 – Spring 2002Notes 472 (a) Simple case - no example (b) Coalesce with neighbor (sibling) (c) Re-distribute keys (d) Cases (b) or (c) at non-leaf Deletion from B+tree

73. CS 277 – Spring 2002Notes 473 (b) Coalesce with sibling –Delete 50 10 40 100 10 20 30 40 50 n=4 40

74. CS 277 – Spring 2002Notes 474 (c) Redistribute keys –Delete 50 10 40 100 10 20 30 35 40 50 n=4 35

75. CS 277 – Spring 2002Notes 475 40 45 30 37 25 26 20 22 10 14 1313 10 2030 40 (d) Non-leaf coalese –Delete 37 n=4 40 30 25 new root

76. CS 277 – Spring 2002Notes 476 B+tree deletions in practice –Often, coalescing is not implemented –Too hard and not worth it!

77. CS 277 – Spring 2002Notes 477 Comparison: B-trees vs. static indexed sequential file Ref #1: Held & Stonebraker “B-Trees Re-examined” CACM, Feb. 1978

78. CS 277 – Spring 2002Notes 478 Ref # 1 claims: - Concurrency control harder in B-Trees - B-tree consumes more space For their comparison: block = 512 bytes key = pointer = 4 bytes 4 data records per block

79. CS 277 – Spring 2002Notes 479 Example: 1 block static index 127 keys (127+1)4 = 512 Bytes -> pointers in index implicit!up to 127 blocks k1 k2 k3 k1k2k3 1 data block

80. CS 277 – Spring 2002Notes 480 Example: 1 block B-tree 63 keys 63x(4+4)+8 = 512 Bytes -> pointers needed in B-treeup to 63 blocks because index isblocks not contiguous k1 k2... k63 k1k2k3 1 data block next -

81. CS 277 – Spring 2002Notes 481 Size comparison Ref. #1 Static Index B-tree # data blocks height 2 -> 1272 2 -> 63 2 128 -> 16,129 3 64 -> 3968 3 16,130 -> 2,048,3834 3969 -> 250,047 4 250,048 -> 15,752,961 5

82. CS 277 – Spring 2002Notes 482 Ref. #1 analysis claims For an 8,000 block file, after 32,000 inserts after 16,000 lookups  Static index saves enough accesses to allow for reorganization Ref. #1 conclusion Static index better!!

83. CS 277 – Spring 2002Notes 483 Ref #2: M. Stonebraker, “Retrospective on a database system,” TODS, June 1980 Ref. #2 conclusion B-trees better!!

84. CS 277 – Spring 2002Notes 484 DBA does not know when to reorganize DBA does not know how full to load pages of new index Ref. #2 conclusion B-trees better!!

85. CS 277 – Spring 2002Notes 485 Buffering –B-tree: has fixed buffer requirements –Static index: must read several overflow blocks to be efficient (large & variable size buffers needed for this) Ref. #2 conclusion B-trees better!!

86. CS 277 – Spring 2002Notes 486 Speaking of buffering… Is LRU a good policyfor B+tree buffers?  Of course not!  Should try to keep root in memory at all times (and perhaps some nodes from second level)

87. CS 277 – Spring 2002Notes 487 Interesting problem: For B+tree, how large should n be? … n is number of keys / node

88. CS 277 – Spring 2002Notes 488 Sample assumptions: (1) Time to read node from disk is (70+0.05n) msec. (2) Once block in memory, use binary search to locate key: (a + b LOG 2 n) msec. For some constants a,b; Assume a << 70 (3) Assume B+tree is full, i.e., # nodes to examine is LOG n N where N = # records

89. CS 277 – Spring 2002Notes 489  Can get: f(n) = time to find a record f(n) n opt n

90. CS 277 – Spring 2002Notes 490  FIND n opt by f’(n) = 0 Answer is n opt = “few hundred” (see homework for details)  What happens to n opt as Disk gets faster? CPU get faster?

91. CS 277 – Spring 2002Notes 491 Variation on B+tree: B-tree (no +) Idea: –Avoid duplicate keys –Have record pointers in non-leaf nodes

92. CS 277 – Spring 2002Notes 492 to record to record to record with K1 with K2 with K3 to keys to keys to keys to keys k3 K1 P1K2 P2K3 P3

93. CS 277 – Spring 2002Notes 493 B-tree examplen=2 65 125 145 165 85 105 25 45 10 20 30 40 110 120 90 100 70 80 170 180 50 60 130 140 150 160 sequence pointers not useful now! (but keep space for simplicity)

94. CS 277 – Spring 2002Notes 494 Note on inserts Say we insert record with key = 25 10 20 30 n=3 leaf 10 – 20 – 25 30 Afterwards:

95. CS 277 – Spring 2002Notes 495 So, for B-trees: MAXMIN Tree Rec Keys PtrsPtrs Ptrs Ptrs Non-leaf non-root n+1nn  (n+1)/2   (n+1)/2  -1  (n+1)/2  -1 Leaf non-root 1nn 1  (n+1)/2   (n+1)/2  Root non-leaf n+1nn 2 1 1 Root Leaf 1nn 1 1 1

96. CS 277 – Spring 2002Notes 496 Tradeoffs: B-trees have faster lookup than B+trees  in B-tree, non-leaf & leaf different sizes  in B-tree, deletion more complicated  B+trees preferred!

97. CS 277 – Spring 2002Notes 497 But note: If blocks are fixed size (due to disk and buffering restrictions) Then lookup for B+tree is actually better!!

98. CS 277 – Spring 2002Notes 498 Example: - Pointers 4 bytes - Keys4 bytes - Blocks100 bytes (just example) - Look at full 2 level tree

99. CS 277 – Spring 2002Notes 499 Root has 8 keys + 8 record pointers + 9 son pointers = 8x4 + 8x4 + 9x4 = 100 bytes B-tree: Each of 9 sons: 12 rec. pointers (+12 keys) = 12x(4+4) + 4 = 100 bytes 2-level B-tree, Max # records = 12x9 + 8 = 116

100. CS 277 – Spring 2002Notes 4100 Root has 12 keys + 13 son pointers = 12x4 + 13x4 = 100 bytes B+tree: Each of 13 sons: 12 rec. ptrs (+12 keys) = 12x(4 +4) + 4 = 100 bytes 2-level B+tree, Max # records = 13x12 = 156

101. CS 277 – Spring 2002Notes 4101 So... ooooooooooooo ooooooooo 156 records 108 records Total = 116 B+ B 8 records Conclusion: –For fixed block size, –B+ tree is better because it is bushier

102. CS 277 – Spring 2002Notes 4102 Outline/summary Conventional Indexes Sparse vs. dense Primary vs. secondary B trees B+trees vs. B-trees B+trees vs. indexed sequential Hashing schemes-->Next