1. Files & Indexing

2. Files of Records uPage or block is OK when doing I/O, but higher levels of DBMS operate on records, and files of records. uFILE : A collection of pages, each containing a collection of records. Must support: w insert/delete/modify records w read a particular record (specified using record id) w scan records (possibly with some conditions on the records to be retrieved)

3. Alternative File Organizations Many alternatives exist, tradeoffs for each: wHeap files: Suitable when typical access is file scan of all records. wSorted Files: Best for retrieval in search key order Also good for search based on search key w Indexes: Organize records via trees or hashing. Like sorted files, speed up searches for search key fields Updates are much faster than in sorted files.

4. Unordered (Heap) Files uSimplest file structure contains records in no particular order. uAs file grows and shrinks, disk pages are allocated and de-allocated. uTo support record level operations, we must: w keep track of the pages in a file w keep track of free space on pages w keep track of the records on a page uThere are many alternatives for keeping track of this.

5. Heap File Implemented as a List uThe header page id and Heap file name must be stored someplace. uProblem: Most pages might be on free space list (holes) Header Page Data Page Data Page Data Page Data Page Data Page Data Page Pages with Free Space Full Pages

6. Heap File Using a Page Directory uThe entry for a page can include the number of free bytes on the page. uThe directory is a collection of pages; linked list implementation is just one alternative. w Much smaller than linked list of all HF pages! Data Page 1 Data Page 2 Data Page N Header Page DIRECTORY

7. Indexes uSometimes need to retrieve records by the values in one or more fields, e.g., wFind all students in the “CS” department wFind all students with a gpa > 3 uAn index on a file is a: wDisk-based data structure wSpeeds up selections on the search key fields for the index. wAny subset of the fields of a relation can be index search key wSearch key is not the same as (candidate) key (e.g. doesn’t have to be unique). uAn index wContains a collection of index and data entries wSupports efficient retrieval of all records with a given search key value k.

8. Given condition(s) on attribute(s) find qualified records Attr = value Condition may also be uAttr>value uAttr>=value  value Qualified records value Goal of Indexing

9. 9 First Question About Indexes uWhat kinds of selections do they support? wSelections of form field constant wEquality selections (op is =) wRange selections (op is one of, =, BETWEEN) wMore exotic selections: 2-dimensional ranges (“east of Troy and west of Schenectady and North of Albany and South of Watervliet”) –Or n-dimensional 2-dimensional distances (“within 2 miles of Sage Hall”) –Or n-dimensional Ranking queries (“10 italian restaurants closest to Troy”) Regular expression matches, genome string matches, etc.

10. Alternatives for Data Entry k* in Index uThree alternatives:  Actual data record (with key value k)  uChoice is orthogonal to the indexing technique. wtechniques: B+ trees, hash-tables, R trees, … wTypically, index contains auxiliary information that directs searches to the desired data entries uCan have multiple (different) indexes per file. wE.g. file sorted by age, with a hash index on salary and a B+tree index on name.

11. Basic Indexing Methods uIndexed Sequential File uB-Tree uHash Index

12. Indexed Sequential File uSearch key (  primary key) uPrimary index (on Sequencing field) wThe index on the attribute (a.k.a. search key) that determines the sequencing of the table uSecondary index wIndex on any other attribute uDense index (all Search Key values in) uSparse index uMulti-level index

13. Sequential File 20 10 40 30 60 50 80 70 100 90 Tuples are sorted by their primary key Block

14. Sequential File 20 10 40 30 60 50 80 70 100 90 Dense Index 10 20 30 40 50 60 70 80 90 100 110 120 Index file needs much fewer blocks than the data file, hence easier to fit in memory For a given key K, only log 2 n, out of n, index blocks need to be accessed

15. Sequential File 20 10 40 30 60 50 80 70 100 90 Sparse Index 10 30 50 70 90 110 130 150 170 190 210 230 Typically, only one key per data block Find the index record with largest value that is less or equal to the value we are looking

16. Sequential File 20 10 40 30 60 50 80 70 100 90 Sparse 2nd level 10 30 50 70 90 110 130 150 170 190 210 230 10 90 170 250 330 410 490 570 Treat the index as a file and build an index on it Two levels are usually sufficient More than three levels are rare {FILE,INDEX} may be contiguous or not

17. Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150

18. Deletion from sparse index – delete record 40 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 If the deleted entry does not appear in the index do nothing

19. Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 – delete record 30 40 If the deleted entry appears in the index replace it with the next search-key value

20. Deletion from sparse index 20 10 40 30 60 50 80 70 10 30 50 70 90 110 130 150 – delete records 30 & 40 50 70 If the next search key value has its own index entry, then delete the entry

21. Deletion from dense index 20 10 40 30 60 50 80 70 10 20 30 40 50 60 70 80

22. Deletion from dense index 20 10 40 30 60 50 80 70 10 20 30 40 50 60 70 80 – delete record 30 40 Deletion from dense primary index file is handled in the same way with deletion from a sequential file

23. Insertion, sparse index case 20 1030 50 4060 10 30 40 60

24. Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 34 34 our lucky day! we have free space where we need it!

25. Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 15 15 20 30 20 Illustrated: Immediate reorganization Variation: – insert new block (chained file) – update index

26. Insertion, sparse index case 20 1030 50 4060 10 30 40 60 – insert record 25 25 overflow blocks (reorganize later...) How often do we reorganize and how expensive is it? B-Trees offer convincing answers

27. Index (sequential) continuous free space 10 20 30 40 50 60 70 80 90 39 31 35 36 32 38 34 33 overflow area (not sequential) Insertion Example

28. Conventional Indexes uAdvantage: wSimple algorithms wIndex is sequential file good for scans uDisadvantage: wInserts expensive, and/or wEventually sequentiality is lost because of overflows reorganizations are needed

29. B + -Tree Index

30. B+ Tree Indexes  Leaf pages contain data entries, and are chained (prev & next)  Non-leaf pages contain index entries and direct searches: P 0 K 1 P 1 K 2 P 2 K m P m index entry Non-leaf Pages Leaf

31. Example B+ Tree uFind 28*? 29*? All > 15* and < 30* uInsert/delete: Find data entry in leaf, then change it. Need to adjust parent sometimes. wAnd change sometimes bubbles up the tree 2*3* Root 17 30 14*16* 33*34* 38* 39* 135 7*5*8*22*24* 27 27*29* Entries < 17Entries >= 17

32. B+ Tree: Most Widely Used Index uInsert/delete at log F N cost; keep tree height- balanced. (F = fanout, N = # leaf pages) uMinimum 50% occupancy (except for root). Each node contains d <= m <= 2d entries. The parameter d is called the order of the tree. uSupports equality and range-searches efficiently. Index Entries Data Entries ("Sequence set") (Direct search)

33. Example B+ Tree uSearch begins at root, and key comparisons direct it to a leaf. uSearch for 5*, 15*, all data entries >= 24*... * Based on the search for 15*, we know it is not in the tree! Root 1724 30 2* 3*5* 7*14*16* 19*20*22*24*27* 29*33*34* 38* 39* 13

34. Inserting into a B+ Tree uFind correct leaf L. uPut data entry onto L. w If L has enough space, done! w Else, must split L (into L and a new node L2) Redistribute entries evenly, copy up middle key. Insert index entry pointing to L2 into parent of L. uThis can happen recursively w To split index node, redistribute entries evenly, but push up middle key. (Contrast with leaf splits.) uSplits “grow” tree; root split increases height. w Tree growth: gets wider or one level taller at top.

35. Inserting 8* into Example B+ Tree uObserve how minimum occupancy is guaranteed in both leaf and index page splits. uNote difference between copy- up and push- up; be sure you understand the reasons for this. 2* 3*5* 7* 8* 5 Entry to be inserted in parent node. (Note that 5 is continues to appear in the leaf.) s copied up and appears once in the index. Contrast 52430 17 13 Entry to be inserted in parent node. (Note that 17 is pushed up and only this with a leaf split.)

36. Example B+ Tree After Inserting 8*  Notice that root was split, leading to increase in height.  In this example, we can avoid split by re-distributing entries; however, this is usually not done in practice. 2*3* Root 17 24 30 14*16* 19*20*22*24*27* 29*33*34* 38* 39* 135 7*5*8*

37. Deleting from a B+ Tree uStart at root, find leaf L where entry belongs. uRemove the entry. w If L is at least half-full, done! w If L has only d-1 entries, Try to re-distribute, borrowing from sibling (adjacent node with same parent as L). If re-distribution fails, merge L and sibling. uIf merge occurred, must delete entry (pointing to L or sibling) from parent of L. uMerge could propagate to root, decreasing height.

38. Example Tree After (Inserting 8*, Then) Deleting 19* and 20*... uDeleting 19* is easy. uDeleting 20* is done with re- distribution. Notice how middle key is copied up. 2*3* Root 17 30 14*16* 33*34* 38* 39* 135 7*5*8*22*24* 27 27*29*

39. ... And Then Deleting 24* uMust merge. uObserve `toss’ of index entry (on right), and `pull down’ of index entry (below). 30 22*27* 29*33*34* 38* 39* 2* 3* 7* 14*16* 22* 27* 29* 33*34* 38* 39* 5*8* Root 30 135 17

40. Non-leaf Re-distribution uTree is shown below during deletion of 24*. (What could be a possible initial tree?) uIn contrast to previous example, can re- distribute entry from left child of root to right child. Root 135 1720 22 30 14*16* 17*18* 20*33*34* 38* 39* 22*27*29*21* 7*5*8* 3*2*

41. After Re-distribution uIntuitively, entries are re-distributed by `pushing through’ the splitting entry in the parent node. uIt suffices to re-distribute index entry with key 20; we’ve re-distributed 17 as well for illustration. 14*16* 33*34* 38* 39* 22*27*29* 17*18* 20*21* 7*5*8* 2*3* Root 135 17 30 20 22

42. Bulk Loading of a B+ Tree uIf we have a large collection of records, and we want to create a B+ tree on some field, doing so by repeatedly inserting records is very slow. uBulk Loading can be done much more efficiently. uInitialization: Sort all data entries, insert pointer to first (leaf) page in a new (root) page. 3* 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* Sorted pages of data entries; not yet in B+ tree Root

43. Bulk Loading (Contd.) uIndex entries for leaf pages always entered into right- most index page just above leaf level. When this fills up, it splits. (Split may go up right-most path to the root.) uMuch faster than repeated inserts, especially when one considers locking! 3* 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* Root Data entry pages not yet in B+ tree 3523126 1020 3* 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* 6 Root 10 12 23 20 35 38 not yet in B+ tree Data entry pages