6. Files of (horizontal) Records The concepts of pages or blocks suffices when doing I/O, but the higher layers of a DBMS operate on records and files of records FILE: A collection of pages, each containing a collection of records. Must support: insert/delete/modify record read a particular record (specified using record id) scan all records (possibly with some conditions on the records to be retrieved) 13

Files Types The three basic file organizations supported by the File Manager of most DBMSs are: HEAP FILES (files of un-ordered records) SORTED or CLUSTERED FILES ( records sorted or clustered on some field(s) ) HASHED FILES (files in which records are positioned based on a hash function on some field(s) 13

Unordered (Heap) Files Simplest file structure contains records in no particular order. As file grows and shrinks, disk pages are allocated and de-allocated. To support record level operations, DBMS must: keep track of the pages in a file keep track of free space on pages keep track of the records on a page There are many alternatives for keeping track of these. 14

Heap File Implemented as a Linked List Data Page Data Page … Data Page Full Pages Header Page Data Page Data Page … Data Page Pages with Free Space The header page id and Heap file name must be stored someplace. Each page contains 2 `pointers’ plus data. 15

Heap File Using a Page Directory Data Page 1 Page 2 Page N Header Page DIRECTORY (linked list of Header blocks Containing page IDs) The entry for a page can include the number of free bytes on the page. The directory is a collection of pages; linked list implementation is just one alternative. 16

Heap File Facts Record insert? Method-1: System inserts new records at the end of the file (need indicator), moves last record into freed slot following a deletion, updates indicator. - doesn't allow support of the RID or RRN concept. Or a deleted record slot can remain empty (until file reorganized) - allows support of RID/RRN concept <- page | record |0 | record |1 | record |2 | |3 | |4 | |5 |3 | <- next open slot indicator 16

Heap File Facts Record insert Method-2: Insert in any open slot. Must maintain a data structure indicating open slots (e.g., bit filter (or bit map) identifies open slots) - as a list or - as a bit_filter <- page record 101001  availability bit filter (0 means available) If we want all records with a given value in particular field, need an "index" Of course index files must provide a fast way to find the particular value entries of interest (the heap file organization for index files would makes little sense). Index files are usually sorted files. Indexes are examples of ACCESS PATHS. 16

Sorted File (Clustered File) Facts File is sorted on one attribute (e.g., using the unpacked record-pointer page-format) Advantages over heap includes: - reading records in that particular order is efficient - finding next record in order is efficient. For efficient "value-based" ordering (clustering), a level of indirection is useful (unpacked, record-pointer page-format) What happens when a page fills up? Use an overflow page for next record? page 3 RID(3,3) 0 RID(3,0) 1 RID(3,4) 2 RID(3,2) 3 RID(3,1) 4 5 20341 unpacked record-pointer page-format slot-directory Ovfl page 9 1 2 3 4 5 | When a page fills up and,e.g., a record must be inserted and clustered between (3,1) and (3,5), one solution is to simply place it on an overflow page in arrival order. Then the overflow page is scanned like an unordered file page, when necessary. Periodically the primary and overflow pages can be reorganized as an unpacked record-pointer extent to improve sequential access speed (next slide for an example) RID(3,6) RID(3,8) 5 16

Sorted File (Clustered File) Facts Reorganizing a Sorted File with several overflow levels. THE BEFORE: Ovfl page 2 1 2 3 4 5 page 3 RID(3,3) 0 RID(3,0) 1 RID(3,4) 2 RID(3,2) 3 RID(3,1) 4 520341 RID(3,8) RID(3,6) Ovfl page 9 RID(3,9) RID(3,5) RID(3,11) RID(3,10) RID(3,15) 534102 RID(3,7) Ovfl page 9 1 2 3 4 5 AFTER: Ovfl page 2 page 3 RID(3,3) 0 RID(3,0) 1 RID(3,4) 2 RID(3,2) 3 RID(3,1) 4 520341 RID(3,5) RID(3,6) RID(3,9) RID(3,8) RID(3,11) RID(3,10) RID(3,7) RID(3,15) In this case re-organization requires only 2 record swaps and 1 slot directory re-write. 341250 16

Hash files A hash function is applied to the key of a record to determine which "file bucket" it goes to ("file buckets" are usually the pages of that file. Assume there are M pages, numbered 0 through M-1. Then the hash function can be any function that converts the key to a number between 0 and M-1 (e.g., for numeric keys, MODM-1 is typical) ). Collisions or Overflows can occur (when a new record hashes to a bucket that is already full). The simplest Overflow method is to use separate Overflow pages: Overflow pages are allocated if needed (as a separate link list for each bucket. Page#s are needed for pointers) or a shared link list. Long overflow chains can develop and degrade performance. Extendible and Linear Hashing are dynamic techniques to fix this problem. Overflow pages(as Single link list) . e.g., h(key) mod M Overflow pages(as separate link list) 1 2 key h M-1 Primary bucket pages 3

Other Static Hashing overflow handling methods Overflow can be handled by open addressing also (more commonly used for internal hash tables where a bucket is a allocation of main memory, not a page. In Open Addressing, upon collision, search forward in the bucket sequence for the next open record slot. 1 2 3 4 5 6 rec e.g., h(key) mod M Then to search, apply h. If not found, search sequentially ahead until found (circle around to search start point)! rec rec rec key h h(rec_key)=1 Collision! 2? no 3? yes rec bucket pages 3

Other overflow handling methods Overflow can be handled by re-hashing also. In re-hashing, upon collision, apply next hash function from a sequence of hash functions.. Then to search, apply h. If not found, apply next hash function until found or list exhausted. 1 2 3 4 5 6 h0(key) rec rec rec then h1 then h2 ... h rec These methods can be combined also. bucket pages 3

Extendible Hashing Idea: Use directory of pointers to buckets, split just the bucket that overflowed double the directory when needed Directory is much smaller than file, so doubling it is cheap. Only one page of data entries is split. No overflow page! Trick lies in how hash function is adjusted! 5

Example blocking factor(bfr)=4 (# entries per bucket) 13* 00 01 10 11 2 LOCAL DEPTH DIRECTORY Bucket A Bucket B Bucket C Bucket D 10* 1* 21* 4* 12* 32* 16* 15* 7* 19* 5* GLOBAL DEPTH = gd To find the bucket for a new key value, r, take just the last global depth bits of h(r), not all of it! (last 2 bits in this example) (for simplicity we let h(r)=r here) E.g., h(5)=5=101binary thus it's in bucket pointed in the directory by 01. Local depth of a bucket: # of bits used to determine if an entry belongs to bucket Global depth of directory: Max # of bits needed to tell which bucket an entry belongs to (= max of local depths) DATA PAGES Apply hash function, h, to key value, r Follow pointer of last 2 bits of h(r). Insert: If bucket is full, split it (allocate 1 new page, re-distribute over those 2 pages). 6

Example how did we get there? LOCAL DEPTH 1 Bucket A 4* GLOBAL DEPTH = gd 1 Bucket B First insert is 4: h(4) = 4 = 100binary in bucket pointed in the directory by 0. 1 DIRECTORY DATA PAGES 6

Example LOCAL DEPTH 1 Bucket A 4* 12* 32* 16* GLOBAL DEPTH = gd 1 1 Bucket B 1* Insert: 12, 32, 16 and 1 1 h(12) = 12 = 1100binary in bucket pointed in the directory by 0. h(32) = 32 = 10 0000binary in bucket pointed in the directory by 0. DIRECTORY h(16) = 16 = 1 0000binary in bucket pointed in the directory by 0. h(1) = 1 = 1binary in bucket pointed in the directory by 1. DATA PAGES 6

Example LOCAL DEPTH 1 Bucket A 4* 12* 32* 16* GLOBAL DEPTH = gd 1 1 Bucket B 1* 5* 21* 13* Insert: 5, 21 and 13 1 h(5) = 5 = 101binary in bucket pointed in the directory by 1. h(21) = 21 = 1 0101binary in bucket pointed in the directory by 1. h(13) = 13 = 1101binary in bucket pointed in the directory by 1. DIRECTORY DATA PAGES 6

Example LOCAL DEPTH 2 Bucket A 4* 12* 32* 16* GLOBAL DEPTH = gd 1 2 1 9th insert: 10 h(10) = 10 = 1010binary in bucket pointed in the directory by 0. Collision! Bucket B 1 1* 5* 21* 13* 1 2 Bucket C 10* Split bucket A into A and C. Double directory (by copying what is there DIRECTORY and adding a bit on the left). Reset one pointer. DATA PAGES Redistribute values among A and C (if necessary Not necessary this time since all 2's bits correct: 4 = 100 12 = 1100 32 =100000 16 = 10000 10 = 1010 6

Example Inserts: 15, 7 and 19 h(15) = 15 = 1111binary LOCAL DEPTH 2 Bucket A 4* 12* 32* 16* GLOBAL DEPTH = gd 1 Inserts: 15, 7 and 19 2 1 2 h(15) = 15 = 1111binary Bucket B 1 1* 5* 21* 13* h(15) = 7 = 111binary 1 h(19) = 15 = 1 0011binary 2 Bucket C 10* DATA PAGES 1 Bucket D 15* 7* 19* DIRECTORY Split bucket B into B and D. No need to double directory because the local depth of B is less than the global depth. Reset one pointer Redistribute values among B and D (if necessary, not necessary this time). Reset local depth of B and D 6

Insert h(20)=20=10100  Bucket pointed to by 00 is full! Split A. Double directory and reset 1 pointer. LOCAL DEPTH 3 Bucket E (`split image' of Bucket A) 2 Bucket A Redistribute contents of A GLOBAL DEPTH 4* 12* 32* 16* 4* 12* 2 2 Bucket B 1 00 01 10 11 1* 5* 21* 13* 00 01 10 11 2 Bucket C 10* 2 Bucket D 15* 7* 19* 7

Points to Note 20 = binary 10100. Last 2 bits (00) tell us r belongs in either A or A2, but not which one. Last 3 bits needed to tell which one. Local depth of a bucket: # of bits used to determine if an entry belongs to this bucket. Global depth of directory: Max # of bits needed to tell which bucket an entry belongs to (= max of local depths) When does bucket split cause directory doubling? Before insert, local depth of bucket = global depth. Insert causes local depth to become > global depth; directory is doubled by copying it over and `fixing’ pointer to split image page. Use of least significant bits enables efficient doubling via copying of directory!) 8

Comments on Extendible Hashing If directory fits in memory, equality search answered with one disk access; else two. Directory grows in spurts, and, if the distribution of hash values is skewed, directory can grow large. Multiple entries with same hash value cause problems! Delete: If removal of data entry makes bucket empty, can be merged with its `split image’. As soon as each directory element points to same bucket as its (merged) split image, can halve directory. 9

Linear Hash File Starts with M buckets (numbered 0, 1, ..., M-1 and initial hash function, h0=modM (or more general, h0(key)=h(key)modM for any hash ftn h which maps into the integers Use Chaining to shared overflow-pages to handle overflows. At the first overflow, split bucket0 into bucket0 and bucketM and rehash bucket0 records using h1=mod2M. Henceforth if h0 yields value0, rehash using h1=mod2M At the next overflow, split bucket1 into bucket1 and bucketM+1 and rehash bucket1 records using h1=mod2M. Henceforth if h0 yields value1, use h1 ... When all of the original M buckets have been split (M collisions), then rehash all overflow records using h1. Relabel h1 as h0, (discarding the old h0 forever) and start a new "round" by repeating the process above for all future collisions (i.e., now there are buckets 0,...,(2M-1) and h0 = MOD2M). To search for a record, let n = number of splits so far in the given round, if h0(key) is not greater than n, then use h1, else use h0. 11

Linear Hash ex. M=5 h0(27)mod5(27)=2 C! 00,5, mod10rehash 0; n=0 Bucket pg 0 45 1 99 2 23 3 78 4 98 | | | 21 Linear Hash ex. M=5 | | | 15|LOWE |ZAP |ND 02|BAID |NY |NY 23 22|ZHU |SF |CA | | | 25|CLAY |OUTBK|NJ 45 5 21 | | | 6 101 7 104 8 105 33|GOOD |GATER|FL | | | 78 9 107 | | | 8|SINGH |FGO |ND Insert h0(27)mod5(27)=2 C! 00,5, mod10rehash 0; n=0 27|JONES |MHD |MN 14|THAISZ|KNOB |NJ 98 24|CROWE |SJ |CA Insert h0(8)mod5(8)=3 8|SINGH |FGO |ND 11|BROWN |NY |NY 99 21|BARBIE|NY |NY Insert h0(15)mod5(15)=0n 15|LOWE |ZAP |ND h1(15)mod10(15)=5 | | | 101 | | | Insert h0(32)mod5(32)=2 ! 11,6, mod10rehash 1; n=1 32|FARNS |BEEP |NY | | | 104 | | | Insert h0(39)mod5(39)=4 ! 22,7; mod10rehash 2; n=2 39|TULIP |DERLK|IN | | | 105 Insert h0(31)mod5(31)=1 ! 33,8; mod10rehash 3; n=3 31|ROSE |MIAME|OH | | | | | | 107 Insert h0(36)mod5(36)=1 ! 44,9; mod10rehash 4; n=4! 36|SCHOTZ|CORN |IA | | | 27|JONES |MHD |MN 39|TULIP |DERLK|IN OF 36|SCHOTZ|CORN |IA 32|FARNS |BEEP |NY 31|ROSE |MIAME|OH 11

LHex. 2nd rnd M=10 h0mod10 h0(10)mod10(10)=0 Bucket pg 0 45 1 99 0 45 1 99 2 23 3 78 4 98 | | | 25|CLAY |OUTBK|NJ 21 LHex. 2nd rnd M=10 h0mod10 | | | 15|LOWE |ZAP |ND 02|BAID |NY |NY 23 22|ZHU |SF |CA | | | 10|RADHA |FGO |ND 45 5 21 | | | h027=7 6 101 7 104 h032=2 Collision! rehash mod20 8 105 33|GOOD |GATER|FL | | | 78 9 107 h039=9 | | | 10 109 11 110 h031=1 Collision! rehash mod20 14|THAISZ|KNOB |NJ 98 24|CROWE |SJ |CA h036=6 Insert h0(10)mod10(10)=0 10|RADHA |FGO |ND 11|BROWN |NY |NY | | | 99 21|BARBIE|NY |NY | | | 101 ETC. | | | | | | 104 | | | 110 | | | | | | OF 8|SINGH |FGO |ND | | | 105 27|JONES |MHD |MN | | | 32|FARNS |BEEP |NY | | | 107 39|TULIP |DERLK|IN | | | 31|ROSE |MIAME|OH | | | 109 | | | 36|SCHOTZ|CORN |IA 11

Summary Hash-based indexes: best for equality searches, cannot support range searches. Static Hashing can lead to performance degradation due to collision handling problems. Extendible Hashing avoids performance problems by splitting a full bucket when a new data entry is to be added to it. (Duplicates may require overflow pages.) Directory to keep track of buckets, doubles periodically. Can get large with skewed data; additional I/O if this does not fit in main memory. 16

Summary Linear Hashing avoids directory by splitting buckets round-robin, and using overflow pages. Overflow pages not likely to be long. Duplicates handled easily. Space utilization could be lower than Extendible Hashing, since splits not concentrated on `dense’ data areas. skewed occurs when the hash values of the data entries are not uniform! . . . v1 v2 v3 v4 v5 . . . vn Count skew count values . . . v1 v2 v3 v4 v5 . . . vn Dist & Count skew count values v1 v2 v3 v4 v5 . . . vn Distribution skew count values 17