1. PermJoin: An Efficient Algorithm for Producing Early Results in Multi-join Query Plans Justin J. Levandoski Mohamed E. Khalefa Mohamed F. Mokbel University of Minnesota Department of Computer Science Join Algorithms for Emerging Environments General Approaches PermJoin Architecture Scientific Simulation Sensor Networks Moving Object Environments Remote Source A Query Remote Source B Remote Source C Web-Based Data Aggregation  Goal: Produce early query feedback in new and emerging environments  Constraints o Streaming data: not all data available beforehand o Sources may block o Traditional join algorithms optimized to produce entire result  Applications o Web-based environment with slow and bursty input with streaming data o Sensor networks o Scientific simulations taking days to produce large-scale results with need for early results We introduce an efficient algorithm for Producing Early Results in Multi-join query plans (PermJoin, for short). While most previous research focuses only on the case of a single join operator, PermJoin addresses query plans with multiple join operators. PermJoin is optimized to maximize the early overall throughput and to adapt to fluctuations in data arrival rates. PermJoin is a non-blocking operator that is capable of producing join results even if one or more data sources block due to slow or bursty network behavior. Furthermore, PermJoin distinguishes itself from all previous techniques as it: (1) employs a new flushing policy to write in-memory data to disk, once memory allotment is exhausted, in a way that helps increase the probability of producing early result throughput multi-join queries, and (2) employs a novel state manager module that adaptively switches operators between joining in-memory data and disk-resident data in order to maximize overall throughput. Single-join query plans: Hash-merge Join Multi-join query plans: PermJoin Join AB Join ABC Join ABCD Source A Source B Source C Source D Join Result  Main Idea o Collect statistics during query runtime for input sources and data on disk  Memory Flushing o Consider data at each operator equally, flush data least beneficial to query plan  State Manager o Place each operator in optimal state to produce high throughput: in-memory, on- disk, or temporary blocking Source B Join Result On-Disk Sorting and Merging In-Memory Hashing Source Unblocked Both Sources Block or End of Data Memory Full or End of Data Source A Data Flow Disk In-Memory Join Join disk-resident data Join Result State Manager Low Priority Incoming Data Buffer Observed and Collected Statistics Flush Join operator can be in three states In-memory Joining memory-resident data On-disk Joining disk-resident data Low priority Producing results only if resources are available Consider incoming tuple Rs If operator not in-memory Rs temporarily buffered If operator in-memory Rs joined with memory-resident data immediately Flushing: AdaptiveGlobalFlush State Manager In-Memory Join 1 2 N … 1 2 N … Hash Table AHash Table B Source A hash(A) (1)(2) Source B hash(B) (1)(2) Join Result In-Memory: Symmetric-Hash Join Incoming tuple r at Source A Compute hash(r) Probe table B (produce results) Store in table A Symmetric operator for Source B On-Disk: Sort-Merge Join Join different groups Example: a 1,1 and b 1,2 No need to join similar groups Example: a 1,1 and b 1,1 On-Disk Join 4 h1h1 7916 a 1,1 a 1,2 1 h1h1 426 b 1,1 b 1,2 7 Source ASource B Before MergeIn-memory Results: (4,4), (6,6) After MergeOn-Disk Results: (1,1), (7,7) 1 h1h1 46791 h1h1 2467 Source ASource B  Used once hash table memory exhausted  Consider all operators in query plan  Flush partition groups o Symmetric partitions from both sources  Based on three characteristics of in-memory data o Global contribution: the ability to produce overall results o Arrival patterns: changes in data arrival rates at each partition group o Data properties: join attribute distribution or whether data is sorted  Continuous process  Traverse query plan to determine optimal state for each operator  Fundamentally different than flush algorithm o Flushing evicts least-valuable in-memory data o Due to changing nature of query, on-disk data may become valuable later in query runtime o State manager attempts to find this data to increase early throughput, changing each operator to the appropriate state