Files & Indexing

Files of Records uPage or block is OK when doing I/O, but higher levels of DBMS operate on records, and files of records. uFILE : A collection of pages, each containing a collection of records. Must support: w insert/delete/modify records w read a particular record (specified using record id) w scan records (possibly with some conditions on the records to be retrieved)

Alternative File Organizations Many alternatives exist, tradeoffs for each: wHeap files: Suitable when typical access is file scan of all records. wSorted Files: Best for retrieval in search key order Also good for search based on search key w Indexes: Organize records via trees or hashing. Like sorted files, speed up searches for search key fields Updates are much faster than in sorted files.

Unordered (Heap) Files uSimplest file structure contains records in no particular order. uAs file grows and shrinks, disk pages are allocated and de-allocated. uTo support record level operations, we must: w keep track of the pages in a file w keep track of free space on pages w keep track of the records on a page uThere are many alternatives for keeping track of this.

Heap File Implemented as a List uThe header page id and Heap file name must be stored someplace. uProblem: Most pages might be on free space list (holes) Header Page Data Page Data Page Data Page Data Page Data Page Data Page Pages with Free Space Full Pages

Heap File Using a Page Directory uThe entry for a page can include the number of free bytes on the page. uThe directory is a collection of pages; linked list implementation is just one alternative. w Much smaller than linked list of all HF pages! Data Page 1 Data Page 2 Data Page N Header Page DIRECTORY

Indexes uSometimes need to retrieve records by the values in one or more fields, e.g., wFind all students in the “CS” department wFind all students with a gpa > 3 uAn index on a file is a: wDisk-based data structure wSpeeds up selections on the search key fields for the index. wAny subset of the fields of a relation can be index search key wSearch key is not the same as (candidate) key (e.g. doesn’t have to be unique). uAn index wContains a collection of index and data entries wSupports efficient retrieval of all records with a given search key value k.

Given condition(s) on attribute(s) find qualified records Attr = value Condition may also be uAttr>value uAttr>=value  value Qualified records value Goal of Indexing

9 First Question About Indexes uWhat kinds of selections do they support? wSelections of form field constant wEquality selections (op is =) wRange selections (op is one of, =, BETWEEN) wMore exotic selections: 2-dimensional ranges (“east of Troy and west of Schenectady and North of Albany and South of Watervliet”) –Or n-dimensional 2-dimensional distances (“within 2 miles of Sage Hall”) –Or n-dimensional Ranking queries (“10 italian restaurants closest to Troy”) Regular expression matches, genome string matches, etc.

Alternatives for Data Entry k* in Index uThree alternatives:  Actual data record (with key value k)  uChoice is orthogonal to the indexing technique. wtechniques: B+ trees, hash-tables, R trees, … wTypically, index contains auxiliary information that directs searches to the desired data entries uCan have multiple (different) indexes per file. wE.g. file sorted by age, with a hash index on salary and a B+tree index on name.

Basic Indexing Methods uIndexed Sequential File uB-Tree uHash Index

Indexed Sequential File uSearch key (  primary key) uPrimary index (on Sequencing field) wThe index on the attribute (a.k.a. search key) that determines the sequencing of the table uSecondary index wIndex on any other attribute uDense index (all Search Key values in) uSparse index uMulti-level index

Sequential File Tuples are sorted by their primary key Block

Sequential File Dense Index Index file needs much fewer blocks than the data file, hence easier to fit in memory For a given key K, only log 2 n, out of n, index blocks need to be accessed

Sequential File Sparse Index Typically, only one key per data block Find the index record with largest value that is less or equal to the value we are looking

Sequential File Sparse 2nd level Treat the index as a file and build an index on it Two levels are usually sufficient More than three levels are rare {FILE,INDEX} may be contiguous or not

Deletion from sparse index

Deletion from sparse index – delete record If the deleted entry does not appear in the index do nothing

Deletion from sparse index – delete record If the deleted entry appears in the index replace it with the next search-key value

Deletion from sparse index – delete records 30 & If the next search key value has its own index entry, then delete the entry

Deletion from dense index

Deletion from dense index – delete record Deletion from dense primary index file is handled in the same way with deletion from a sequential file

Insertion, sparse index case

Insertion, sparse index case – insert record our lucky day! we have free space where we need it!

Insertion, sparse index case – insert record Illustrated: Immediate reorganization Variation: – insert new block (chained file) – update index

Insertion, sparse index case – insert record overflow blocks (reorganize later...) How often do we reorganize and how expensive is it? B-Trees offer convincing answers

Index (sequential) continuous free space overflow area (not sequential) Insertion Example

Conventional Indexes uAdvantage: wSimple algorithms wIndex is sequential file good for scans uDisadvantage: wInserts expensive, and/or wEventually sequentiality is lost because of overflows reorganizations are needed

B + -Tree Index

B+ Tree Indexes  Leaf pages contain data entries, and are chained (prev & next)  Non-leaf pages contain index entries and direct searches: P 0 K 1 P 1 K 2 P 2 K m P m index entry Non-leaf Pages Leaf

Example B+ Tree uFind 28*? 29*? All > 15* and < 30* uInsert/delete: Find data entry in leaf, then change it. Need to adjust parent sometimes. wAnd change sometimes bubbles up the tree 2*3* Root *16* 33*34* 38* 39* 135 7*5*8*22*24* 27 27*29* Entries < 17Entries >= 17

B+ Tree: Most Widely Used Index uInsert/delete at log F N cost; keep tree height- balanced. (F = fanout, N = # leaf pages) uMinimum 50% occupancy (except for root). Each node contains d <= m <= 2d entries. The parameter d is called the order of the tree. uSupports equality and range-searches efficiently. Index Entries Data Entries ("Sequence set") (Direct search)

Example B+ Tree uSearch begins at root, and key comparisons direct it to a leaf. uSearch for 5*, 15*, all data entries >= 24*... * Based on the search for 15*, we know it is not in the tree! Root * 3*5* 7*14*16* 19*20*22*24*27* 29*33*34* 38* 39* 13

Inserting into a B+ Tree uFind correct leaf L. uPut data entry onto L. w If L has enough space, done! w Else, must split L (into L and a new node L2) Redistribute entries evenly, copy up middle key. Insert index entry pointing to L2 into parent of L. uThis can happen recursively w To split index node, redistribute entries evenly, but push up middle key. (Contrast with leaf splits.) uSplits “grow” tree; root split increases height. w Tree growth: gets wider or one level taller at top.

Inserting 8* into Example B+ Tree uObserve how minimum occupancy is guaranteed in both leaf and index page splits. uNote difference between copy- up and push- up; be sure you understand the reasons for this. 2* 3*5* 7* 8* 5 Entry to be inserted in parent node. (Note that 5 is continues to appear in the leaf.) s copied up and appears once in the index. Contrast Entry to be inserted in parent node. (Note that 17 is pushed up and only this with a leaf split.)

Example B+ Tree After Inserting 8*  Notice that root was split, leading to increase in height.  In this example, we can avoid split by re-distributing entries; however, this is usually not done in practice. 2*3* Root *16* 19*20*22*24*27* 29*33*34* 38* 39* 135 7*5*8*

Deleting from a B+ Tree uStart at root, find leaf L where entry belongs. uRemove the entry. w If L is at least half-full, done! w If L has only d-1 entries, Try to re-distribute, borrowing from sibling (adjacent node with same parent as L). If re-distribution fails, merge L and sibling. uIf merge occurred, must delete entry (pointing to L or sibling) from parent of L. uMerge could propagate to root, decreasing height.

Example Tree After (Inserting 8*, Then) Deleting 19* and 20*... uDeleting 19* is easy. uDeleting 20* is done with re- distribution. Notice how middle key is copied up. 2*3* Root *16* 33*34* 38* 39* 135 7*5*8*22*24* 27 27*29*

... And Then Deleting 24* uMust merge. uObserve `toss’ of index entry (on right), and `pull down’ of index entry (below) *27* 29*33*34* 38* 39* 2* 3* 7* 14*16* 22* 27* 29* 33*34* 38* 39* 5*8* Root

Non-leaf Re-distribution uTree is shown below during deletion of 24*. (What could be a possible initial tree?) uIn contrast to previous example, can re- distribute entry from left child of root to right child. Root *16* 17*18* 20*33*34* 38* 39* 22*27*29*21* 7*5*8* 3*2*

After Re-distribution uIntuitively, entries are re-distributed by `pushing through’ the splitting entry in the parent node. uIt suffices to re-distribute index entry with key 20; we’ve re-distributed 17 as well for illustration. 14*16* 33*34* 38* 39* 22*27*29* 17*18* 20*21* 7*5*8* 2*3* Root

Bulk Loading of a B+ Tree uIf we have a large collection of records, and we want to create a B+ tree on some field, doing so by repeatedly inserting records is very slow. uBulk Loading can be done much more efficiently. uInitialization: Sort all data entries, insert pointer to first (leaf) page in a new (root) page. 3* 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* Sorted pages of data entries; not yet in B+ tree Root

Bulk Loading (Contd.) uIndex entries for leaf pages always entered into right- most index page just above leaf level. When this fills up, it splits. (Split may go up right-most path to the root.) uMuch faster than repeated inserts, especially when one considers locking! 3* 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* Root Data entry pages not yet in B+ tree * 4* 6*9*10*11*12*13* 20*22* 23*31* 35* 36*38*41*44* 6 Root not yet in B+ tree Data entry pages