Hopkins Storage Systems Lab, Department of Computer Science Network-Aware Join Processing in Global-Scale Database Federations X. Wang, R. Burns, A. Terzis Johns Hopkins University A. Deshpande University of Maryland

Hopkins Storage Systems Lab, Department of Computer Science Time/Cost Trade-off for Reaching Isla Mujeres You are here Playa Tortuga Puerto Juarez Downtown Isla Mujeres 5 min, 7p 35 min, 150p 20 min, 7p 5 min, 30p 30 min, 70p 55 min, 107 pesos 40 min, 157 pesos

Hopkins Storage Systems Lab, Department of Computer Science Outline Target Application – Join scheduling in SkyQuery – Incorporating network structure Balanced network utilization metric – Exploit high throughput paths – Limitations Algorithms – Two-approximate, MST-based solution – Heuristic extensions (clustering, semi-joins, bushy plans)

Hopkins Storage Systems Lab, Department of Computer Science SkyQuery Publicly accessible federation of sky surveys (a virtual telescope) Autonomous, heterogeneous, and geographically distributed sites (30 across NA, EA, EU) Data intensive workload – Terabyte data sets – Hundred megabyte intermediate join results – Queries take ten to over a hundred seconds – Network transfers consume up to 70% of the time Principal federated query is cross-match

Hopkins Storage Systems Lab, Department of Computer Science Cross-Match Queries Join by increasing cardinality (count *) – Minimal I/O – Fewer bytes on the network Query Mediator Probe Query Result Count: 30Count: 100Count: 800

Hopkins Storage Systems Lab, Department of Computer Science Incorporating Network Structure

Hopkins Storage Systems Lab, Department of Computer Science Balanced Network Utilization Metric Exploit excess capacity and avoid long haul paths Minimizes aggregative time on the network Similar metrics used for stream-processing, multicast, and optimal link layer routing (Bertsekas & Gallager) Minimizes response time for serial schedules – Avoid over utilizing resources for bushy schedules – Does not account for I/O

Hopkins Storage Systems Lab, Department of Computer Science How to Extract Network Structure?

Hopkins Storage Systems Lab, Department of Computer Science Volatility in TCP Throughput

Hopkins Storage Systems Lab, Department of Computer Science Limitations Perfect join selectivity assumption – Observations against the same sky – Allows for polynomial-time solutions No attribute aggregation – Address heuristically Local optimizations at the mediators – Decentralized to achieve scale using aggregate stats – Routing at the application layer – Improve end performance and preserve I/O

Hopkins Storage Systems Lab, Department of Computer Science Spanning Tree Approximation (STA) B C A D E F G H

Hopkins Storage Systems Lab, Department of Computer Science STA: Find MST B C A D E F G H

Hopkins Storage Systems Lab, Department of Computer Science STA: Join Using Paths on the MST B C A D E F G H

Hopkins Storage Systems Lab, Department of Computer Science STA: Shortcutting in Metric Regions B C A D E F G H

Hopkins Storage Systems Lab, Department of Computer Science C-STA: Clustering TCP Throughput

Hopkins Storage Systems Lab, Department of Computer Science C-STA: Clustering TCP Throughput

Hopkins Storage Systems Lab, Department of Computer Science C-STA: Combine STA & Count * B C A D E F G H

Hopkins Storage Systems Lab, Department of Computer Science STA-SJ: Semi-joins and Attribute Agg. B C A D E F G H Aggregation Join Attr.

Hopkins Storage Systems Lab, Department of Computer Science STA-BP: Exploring Bushy Plans Poly-time DP Algorithm that explores bushy plans using MST paths – Evaluates regions in parallel when beneficial (avoids sending data down the tree) – May operate on larger intermediate results Intuition: Do not need to traverse STA paths twice if sites have low cardinality R ≤ 2R R > 2R

Hopkins Storage Systems Lab, Department of Computer Science Experiments: Network Utilization

Hopkins Storage Systems Lab, Department of Computer Science Experiments: I/O Overhead

Hopkins Storage Systems Lab, Department of Computer Science Experiments: Algorithms Compared

Hopkins Storage Systems Lab, Department of Computer Science Discussion DP solution w/o selectivity, aggregation, MST- based assumptions – T: O(n3 n ), S: O(n2 n ) – Applicability beyond SkyQuery (distributed OLAP/DSS) – May tolerate exponential complexity – Value in capturing network structure Don’t address multi-query optimization – Incomplete info about link layer – Global knowledge incurs high overhead

Hopkins Storage Systems Lab, Department of Computer Science Which Path to Choose? You are here Playa Tortuga Puerto Juarez Downtown Isla Mujeres 20 min, 7p 5 min, 30p 30 min, 70p 55 min, 107 pesos

Hopkins Storage Systems Lab, Department of Computer Science Questions ???