15.5 Two-Pass Algorithms Based on Hashing By Derek Lee

General The essential idea behind all the hash-based algorithms is: If the data is too big to store in main-memory buffers, hash all the tuples of the argument or arguments using an appropriate hash key. For the common operations, there is a way to choose the hash key so all tuples that need to be considered together when perform the operation fall into same bucket. Reduces the size of the operand(s) by a factor equal to the number of buckets

15.5.1 Partitioning Relations by Hashing Set partition R into M-1 buckets of roughly equal size. Associate one buffer with each bucket. Each tuple t in the block is hashed to bucket h(t) and copied to the appropriate buffer. If that buffer is full, write it out of disk, initialize another block for same bucket At the end, write out last block of each bucket if not empty

15.5.1 Contd initialize M-1 buckets using M-1 empty buffers; FOR each block b of relation R DO BEGIN read block b into the Mth buffer; FOR each tuple t in b DO BEGIN IF the buffer for bucket h(t) has no room for t THEN BEGIN copy the buffer to disk; initialize a new empty block in that buffer; END; copy t to the buffer for bucket h(t); END ; FOR each bucket DO IF the buffer for this bucket is not empty THEN write the buffer to disk;

15.5.2 A Hash-Based Algorithm for Duplicate Elimination Two copies of the same tuple t will hash to the same bucket. We can examine one bucket at a time, perform δ on that bucket in isolation, and take as the answer the union of δ(Ri), where Ri is the portion of R that hashes to the ith bucket. Use duplicate elimination on each bucket Ri independently, using one-pass algorithm

15.5.2 Contd Number of disk I/O's: 3*B(R) The two-pass, hash-based algorithm work if B(R) <= M(M-1) In order to work: hash function h evenly distributes the tuples among the buckets each bucket Ri fits in main memory (to allow the one-pass algorithm) i.e., B(R) ≤ M2

15.5.3 Hash-Based Grouping and Aggregation In order to make sure that all tuples of the same group wind up in the same bucket, we must choose a hash function that depends only on the grouping attributes of the list L. If groups are large, then we may actually be able to handle much larger relations R than is indicated by the B(R) <= M2 rule.

15.5.4 Hash-Based Union, Intersection, and Difference When the operation is binary, we must make sure that we use the same hash function to hash tuples of both arguments. R U S we hash both R and S to M-1 R ∩or - S we hash both R and S to 2(M-1) One pass algorithm requires 3(B(R)+B(S)) disk I/O’s. Two pass hash based algorithm requires min(B(R)+B(S))≤ M2

15.5.5 The Hash-Join Algorithm The only difference of the join operation from the other operations is that we must use as the hash key just the join attributes, then we can be sure that if tuples of R and S join, they will wind up in corresponding buckets Ri and Si for some i. A one-pass join of all pairs of the corresponding buckets completes this algorithm, we call Hash-Join

15.5.6 Saving Some Disk I/O’s m * B(S) / k + k – m <= M If there is more memory available on the first pass than we need to hold one block per bucket, then we have some opportunities to save disk I/O. Hybrid hash-join: Avoid writing some of buckets to disk and then reading the again. When we hash S, we can choose to keep m of the k buckets entirely in main memory, while keeping only one block for each of the other k-m buckets , that is: m * B(S) / k + k – m <= M

15.5.6 Contd When we read the tuples of the other relation, R, to hash that relation into buckets, we keep in memory: 1. The m buckets of S that were never written to disk, and 2. One block for each of the k-m buckets of R whose corresponding buckets of S were written to disk.

15.5.7 Summary The requirement for sort-based and hah-based algorithm are almost the same. There are differences between them: 1. Hash-based for binary operation have a size requirement that depends only on the smaller of two arguments rather than on the sum of the argument sizes. 2. Sort-based sometimes allow us to produce a result in sorted order and take advantage of that sort later.

15.5.7 Summary 3. Hash-based depend on the buckets being of equal size. 4. In sort-based, sorted sublists may be written to consecutive blocks of the disk if we organize the disk properly. 5. If M is much larger than the number of sorted sublists, then we may read in several consecutive blocks at a time from a sorted sublist, again saving some latency and seek time.

15.5.7 Summary 6. If we can choose the number of buckets to be less than M in a hash-based, we can write out several blocks of a bucket at once.