@ Carnegie Mellon Databases Improving Hash Join Performance Through Prefetching Shimin Chen Phillip B. Gibbons Todd C. Mowry Anastassia Ailamaki ‡ Carnegie Mellon University Intel Research Pittsburgh ‡

@ Carnegie Mellon Databases Hash Join  Simple hash join:  Build hash table on smaller (build) relation  Probe hash table using larger (probe) relation  Random access patterns inherent in hashing  Excessive random I/Os  If build relation and hash table cannot fit in memory Build Relation Probe Relation Hash Table

@ Carnegie Mellon Databases I/O Partitioning  Avoid excessive random disk accesses  Join pairs of build and probe partitions separately  Sequential I/O patterns for relations and partitions  Hash join is CPU-bound with reasonable I/O bandwidth Build Probe

@ Carnegie Mellon Databases  Partition: divides a 1GB relation into 800 partitions  Join: 50MB build partition 100MB probe partition  Detailed simulations based on Compaq ES40 system  Most of execution time is wasted on data cache misses  82% for partition, 73% for join  Because of random access patterns in memory Hash Join Cache Performance

@ Carnegie Mellon Databases Cache partitioning: generating cache-sized partitions  Effective in main-memory databases [Shatdal et al., 94], [Boncz et al., 99], [Manegold et al.,00]  Two limitations when used in commercial databases 1) Usually need additional in-memory partitioning pass  Cache is much smaller than main memory  50% worse than our techniques 2) Sensitive to cache sharing by multiple activities Employing Partitioning for Cache? Main Memory CPU Cache Build Partition Hash Table

@ Carnegie Mellon Databases Our Approach: Cache Prefetching  Modern processors support:  Multiple cache misses to be serviced simultaneously  Prefetch assembly instructions for exploiting the parallelism  Overlap cache miss latency with computation  Successfully applied to  Array-based programs [Mowry et al., 92]  Pointer-based programs [Luk & Mowry, 96]  Database B + -trees [Chen et al., 01] Main Memory CPU L2/L3 Cache L1 Cache pref 0(r2) pref 4(r7) pref 0(r3) pref 8(r9)

@ Carnegie Mellon Databases Challenges for Cache Prefetching  Difficult to obtain memory addresses early  Randomness of hashing prohibits address prediction  Data dependencies within the processing of a tuple  Naïve approach does not work  Complexity of hash join code  Ambiguous pointer references  Multiple code paths  Cannot apply compiler prefetching techniques

@ Carnegie Mellon Databases Our Solution  Dependencies are rare across subsequent tuples  Exploit inter-tuple parallelism  Overlap cache misses of one tuple with computation and cache misses of other tuples  We propose two prefetching techniques  Group prefetching  Software-pipelined prefetching

@ Carnegie Mellon Databases Outline  Overview  Our Proposed Techniques  Simplified Probing Algorithm  Naïve Prefetching  Group Prefetching  Software-Pipelined Prefetching  Dealing with Complexities  Experimental Results  Conclusions

@ Carnegie Mellon Databases Simplified Probing Algorithm foreach probe tuple { (0)compute bucket number; (1)visit header; (2)visit cell array; (3)visit matching build tuple; } Hash Bucket Headers Hash Cell (hash code, build tuple ptr) Build Partition

@ Carnegie Mellon Databases Naïve Prefetching foreach probe tuple { (0)compute bucket number; prefetch header; (1)visit header; prefetch cell array; (2)visit cell array; prefetch matching build tuple; (3)visit matching build tuple; } tim e Cache miss latency Data dependencies make it difficult to obtain addresses early

@ Carnegie Mellon Databases Group Prefetching foreach group of probe tuples { foreach tuple in group { (0)compute bucket number; prefetch header; } foreach tuple in group { (1)visit header; prefetch cell array; } foreach tuple in group { (2)visit cell array; prefetch build tuple; } foreach tuple in group { (3)visit matching build tuple; } } a group

@ Carnegie Mellon Databases Software-Pipelined Prefetching Prologue; for j=0 to N-4 do { tuple j+3: (0)compute bucket number; prefetch header; tuple j+2: (1)visit header; prefetch cell array; tuple j+1: (2)visit cell array; prefetch build tuple; tuple j: (3)visit matching build tuple; } Epilogue; prologue epilogue j j

@ Carnegie Mellon Databases Dealing with Multiple Code Paths  Previous compiler techniques cannot handle this A BCF DG E cond  Multiple code paths:  There could be 0 or many matches  Hash buckets could be empty or full  Keep state information for tuples being processed Record state Test state to decide: Do nothing, if state = B Execute D, if state = C Execute G, if state = F

@ Carnegie Mellon Databases Dealing with Read-write Conflicts  Use busy flag in bucket header to detect conflicts  Postpone hashing 2 nd tuple until finish processing 1 st  Compiler cannot perform this transformation Hash Bucket Headers Build Tuples  In hash table building:

@ Carnegie Mellon Databases More Details In Paper  General group prefetching algorithm  General software-pipelined prefetching algorithm  Analytical models  Discussion of important parameters:  group size, prefetching distance  Implementation details

@ Carnegie Mellon Databases Outline  Overview  Our Proposed Techniques  Experimental Results  Setup  Performance of Our Techniques  Comparison with Cache Partitioning  Conclusions

@ Carnegie Mellon Databases Experimental Setup  Relation schema: 4-byte join attribute + fixed length payload  No selection, no projection  50MB memory available for the join phase  Detailed cycle-by-cycle simulations  1GHz superscalar processor  Memory hierarchy is based on Compaq ES40

@ Carnegie Mellon Databases Joining a Pair of Build and Probe Partitions  Our techniques achieve X speedups over original hash join A 50MB build partition joins a 100MB probe partition 1:2 matching Number of tuples decreases as tuple size increases

@ Carnegie Mellon Databases Varying Memory Latency processor to memory latency execution time (M cycles) A 50MB build partition joins a 100MB probe partition 1:2 matching 100 B tuples  150 cycles: default parameter  1000 cycles: memory latency in future  Our techniques achieve 9X speedups over baseline at 1000 cycles  Absolute performances of our techniques are very close

@ Carnegie Mellon Databases Comparison with Cache Partitioning  Cache partitioning: generating cache sized partitions [Shatdal et al., 94], [Boncz et al., 99], [Manegold et al.,00]  Additional in-memory partition step after I/O partitioning  At least 50% worse than our prefetching schemes A 200MB build relation joins a 400MB probe relation 1:2 matching Partitioning + join

@ Carnegie Mellon Databases Robustness: Impact of Cache Interference  Cache partitioning relies on exclusive use of cache  Periodically flush cache: worst case interference  Self normalized to execution time when no flush  Cache partitioning degrades 8-38%  Our prefetching schemes are very robust

@ Carnegie Mellon Databases Conclusions  Exploited inter-tuple parallelism  Proposed group prefetching and software-pipelined prefetching  Prior prefetching techniques cannot handle code complexity  Our techniques achieve dramatically better performance  X speedups for join phase  X speedups for partition phase  9X speedups at 1000 cycle memory latency in future  Absolute performances are close to that at 150 cycles  Robust against cache interference  Unlike cache partitioning  Our prefetching techniques are effective for hash joins

@ Carnegie Mellon Databases Thank you !

@ Carnegie Mellon Databases Back Up Slides

@ Carnegie Mellon Databases Is Hash Join CPU-bound ?  Quad-processor Pentium III, four disks  A 1.5 GB build relation, a 3GB probe relation  Main thread: GRACE hash join  Background I/O thread per disk: I/O prefetching and writing  Hash join is CPU-bound with reasonable I/O bandwidth  Still large room for CPU performance improvement Partition phaseJoin phase 550MHz CPUs, 512MB RAM, Seagate Cheetah X15 36LP SCSI disks (max transfer rate 68MByte/sec), Linux B tuples, 4B keys, 1:2 matching Striping unit = 256KB 10 measurements, std < 10% mean or std < 1s

@ Carnegie Mellon Databases Hiding Latency within A Group  Hide cache miss latency across multiple tuples within a group  Group size can be increased to hide most cache miss latency for hash joins  Generic algorithm and analytical model (please see paper)  There are gaps between groups Group size = 3 Group size =

@ Carnegie Mellon Databases D=2 Prefetching Distance  Prefetching distance (D) : the number of iterations between two subsequent code stages for a single tuple  Increase prefetching distance to hide all cache miss latency  Generic algorithm and analytical model (please see paper) D=1

@ Carnegie Mellon Databases Group Pref: Multiple Code Paths  We keep state information for tuples in a group  One of the states decides which code path to take A BCF DG E cond A B st=1 cond C st=2 F st=3 DG st 23 EEmpty st 2 or 3other

@ Carnegie Mellon Databases Prefetching Distance = D Prologue; for j=0 to N-3D-1 do { tuple j+3D: compute hash bucket number; prefetch the target bucket header; tuple j+2D: visit the hash bucket header; prefetch the hash cell array; tuple j+D: visit the hash cell array; prefetch the matching build tuple; tuple j: visit the matching build tuple to compare keys and produce output tuple; } Epilogue;

@ Carnegie Mellon Databases Experiment Setup  We have implemented our own hash join engine:  Relations are stored in files with slotted page structure  A simple XOR and shift based hash function is used  GRACE hash join (baseline), Simple prefetching, Group prefetching, Software-pipelined prefetching  Experiment Design:  Same schema for build and probe relations:  4-byte key + fixed length payload  No selection and projection  50MB memory available for the joins

@ Carnegie Mellon Databases Simulation Platform  Detailed cycle-by-cycle simulations  Out-of-order processor pipeline  Integer multiply and divide latency are based on Pentium4  Memory hierarchy is based on Compaq ES40  memory system parameters in near future  better prefetching support  Supports TLB prefetching

@ Carnegie Mellon Databases Simulation Parameters Processor Pipeline Parameters Clock Rate1 GHz Issue Width4 insts/cycle Functional Units2 Int, 2 FP, 1 Int Divide, 2 Mem, 1 Branch Reorder Buffer Size128 insts Integer Multiply/Divide15/56 cycles All Other Integer1 cycle Branch Prediction Scheme gshare Memory Parameters Line Size64 bytes Primary Inst Cache64 KB, 2-way set assoc. Primary Data Cache64 KB, 4-way set-assoc. Miss Handlers32 for data, 2 for inst DTLB64 entries, fully-assoc. DTLB Miss Handlers1 Page Size8 KB Unified Secondary Cache1 MB, 4-way set assoc. Primary-to-Secondary Miss Latency 15 cycles (plus contention) DTLB Miss Latency20 cycles Primary-to-Memory Miss Latency 150 cycles (plus contention) Main Memory Bandwidth1 access per 10 cycles

@ Carnegie Mellon Databases Simulator vs. Real Machine  Better prefetching support:  Never ignore prefetching  Our prefetches are not hints!  TLB prefetching  When a prefetch incurs a TLB miss, perform TLB loading  More miss handlers:  32 for data

@ Carnegie Mellon Databases Varying Group Size and Prefetching Distance  Too small: latencies are not fully hidden  Too large: many prefetched cache lines are replaced by other memory references  Similar performance even when latency increases to 1000 cycles! Varying params for the 20B case in the previous figure Hash table probing

@ Carnegie Mellon Databases Breakdowns of Cache Misses to Understand the Tuning Curves

@ Carnegie Mellon Databases Cache Performance Breakdowns  Our schemes indeed hide most of the data cache miss latencies  Overheads lead to larger portions of busy times Join (100B tuples) Partition (800 partitions)

@ Carnegie Mellon Databases Partition Phase Performance  When the number of partitions is small, use simple prefetching  When the number of partitions is large, use group or software- pipelined prefetching  Combined: X speedups over the baseline

@ Carnegie Mellon Databases Cache Partitioning Schemes  Two ways to employ cache partitioning in GRACE join:  Direct cache: generate the cache partitions in the I/O partition phase  Two-step cache: generate the cache partitions in the join phase as a preprocessing step  Requires to generate larger number of smaller I/O partitions  Bounded by available memory  Bounded by requirements of underlying storage managers  So cache partitioning may not be used when joining relations are very large  Not robust with multiple activities going on  Require exclusive use of (part of) cache  Performance penalty due to cache conflicts

@ Carnegie Mellon Databases Comparison with Cache Partitioning I/O partition phase Join phaseOverall performance  Direct cache suffers from larger number of partitions generated in the I/O partition phase  Two-step cache suffers from the additional partition step  Our schemes are the best (slightly better than direct cache)

@ Carnegie Mellon Databases Robustness: Impact of Cache Interference  Performance degradation when the cache is periodically flushed  The worst cache interference  Direct cache and 2-step cache degrade 15-67% and 8-38%  Our prefetching schemes are very robust ‘100’ corresponds to the join phase execution time when there is no cache flush

@ Carnegie Mellon Databases Group Pref vs. Software-pipelined Pref  Hiding latency:  Software-pipelined pref is always able to hide all latencies (according to our analytical model)  Book-keeping overhead:  Software-pipelined pref has more overhead  Code complexity:  Group prefetching is easier to implement  Natural group boundary provides a place to do necessary processing left (e.g. for read-write conflicts)  A natural place to send outputs to the parent operator if pipelined operator is needed

@ Carnegie Mellon Databases  Try to hide the latency within the processing of a single tuple  Example: hash table probing  Does not work:  Dependencies essentially form a critical path  Randomness makes prediction almost impossible Hash Bucket Headers Hash Cell Array Build Partition Challenges in Applying Prefetching

@ Carnegie Mellon Databases Naïve Prefetching foreach probe tuple { compute bucket number; prefetch header; visit header; prefetch cell array; visit cell array; prefetch matching build tuple; visit matching build tuple; } Cache miss latency time Data dependencies make it difficult to obtain addresses early

@ Carnegie Mellon Databases Group Prefetching foreach group of probe tuples { foreach tuple in group { compute bucket number; prefetch header; } foreach tuple in group { visit header; prefetch cell array; } foreach tuple in group { visit cell array; prefetch matching build tuple; } foreach tuple in group { visit matching build tuple; } } time a group

@ Carnegie Mellon Databases Software-Pipelined Prefetching Prologue; for j=0 to N-4 do { tuple j+3: compute bucket number; prefetch header; tuple j+2: visit header; prefetch cell array; tuple j+1: visit cell array; prefetch matching build tuple; tuple j: visit matching build tuple; } Epilogue; prologue epilogue j j+3