1. Workload-Aware Data Partitioning in Community-Driven Data Grids
Tobias Scholl, Bernhard Bauer, Jessica Müller, Benjamin Gufler, Angelika Reiser, and Alfons Kemper
Department of Computer Science, Technische Universität München
Germany

2. Should I Split or Replicate?
Many challenges and opportunities in e-science for database research
High-throughput data management
Correlation of distributed data sources
Community-driven data grids
Dealing with data skew and query hot spots
Workload-awareness by employing cost model during partitioning
Should I Split or Replicate?
EDBT 2009 – Workload-Aware Data Partitioning

3. Query Load Balancing via Partitioning
EDBT 2009 – Workload-Aware Data Partitioning

4. Query Load Balancing via Partitioning
EDBT 2009 – Workload-Aware Data Partitioning

5. Query Load Balancing via Partitioning
EDBT 2009 – Workload-Aware Data Partitioning

6. Query Load Balancing via PartitioningX
EDBT 2009 – Workload-Aware Data Partitioning

7. Query Load Balancing via Replication
EDBT 2009 – Workload-Aware Data Partitioning

8. Query Load Balancing via Replication
EDBT 2009 – Workload-Aware Data Partitioning

9. Query Load Balancing via Replication
EDBT 2009 – Workload-Aware Data Partitioning

10. The AstroGrid-D Project
German Astronomy Community Grid
Funded by the German Ministry of Education and Research
Part of D-Grid
EDBT 2009 – Workload-Aware Data Partitioning

11. Up-Coming Data-Intensive Applications
Alex Szalay, Jim Gray (Nature, 2006): “Science in an exponential world”
Data rates
Terabytes a day/night
Petabytes a year
LHC
LSST
LOFAR
Pan-STARRS
LOFAR
LHC
EDBT 2009 – Workload-Aware Data Partitioning

12. The Multiwavelength Milky Way
EDBT 2009 – Workload-Aware Data Partitioning

13. Research Challenges
Directly deal with Terabyte/Petabyte-scale data sets
Integrate with existing community infrastructures
High throughput for growing user communities
EDBT 2009 – Workload-Aware Data Partitioning

14. Current Sharing in Data Grids
Data autonomy
Policies allow partners to access data
Each institution ensures
Availability (replication)
Scalability
Various organizational structures [Venugopal et al. 2006]:
Centralized
Hierarchical
Federated
Hybrid
EDBT 2009 – Workload-Aware Data Partitioning

15. Community-Driven Data Grids (HiSbase)
EDBT 2009 – Workload-Aware Data Partitioning

16. “Distribute by Region – not by Archive!”
EDBT 2009 – Workload-Aware Data Partitioning

17. “Distribute by Region – not by Archive!”
EDBT 2009 – Workload-Aware Data Partitioning

18. “Distribute by Region – not by Archive!”
EDBT 2009 – Workload-Aware Data Partitioning

19. “Distribute by Region – not by Archive!”
EDBT 2009 – Workload-Aware Data Partitioning

20. Mapping Data to Nodes 2009-03-24
EDBT 2009 – Workload-Aware Data Partitioning

21. Workload-Aware Training Phase
Incorporate query traces during training phase
Base partitioning scheme on
Data load
Query load
Challenges
Balance query load without losing data load balancing
Approximate real query hot spots from query sample
EDBT 2009 – Workload-Aware Data Partitioning

22. Dealing with Query Hot Spots
Query skew triggered by increased interest in particular subsets of the data
Two well-known query load balancing techniques:
Data partitioning
Data replication
Finding trade-offs between both
EDBT 2009 – Workload-Aware Data Partitioning

23. When to Split (Partition) or to Replicate
Considers partition characteristics
Amount of data (few/many data points)
Number of queries (few/many queries)
Extent of regions and queries (small/big queries)
Data points
Few Queries
Many Queries
Small
Big
Few ─
SPLIT
REPLICATE
Many
EDBT 2009 – Workload-Aware Data Partitioning

24. Region Weight Functions
Data only (#objects in a region)
Queries only (#queries in a region)
Scaled queries
Approximate “real” extent of hot spot
Avoid overfitting to training query set
Heat of a region (#objects * #queries)
Extents of regions and queries
Replicate when many big queries
big
small
EDBT 2009 – Workload-Aware Data Partitioning

25. Evaluation Pobs Weight functions: data, heat, extent
Data sets (observational, simulation)
Workloads (SDSS query log, synthetic)
Partitioning Scheme Properties
Load distribution
Communication overhead
Throughput Measurements
Distributed setup
FreePastry simulator
Pobs
EDBT 2009 – Workload-Aware Data Partitioning

26. Load Distribution Uniform data set from the Millennium simulation
Workload with extreme hot spot
In the following:
1024 partitions
Heat of a region (#data * #queries)
Normalized across all partitioning schemes
EDBT 2009 – Workload-Aware Data Partitioning

27. Query-unaware Training
EDBT 2009 – Workload-Aware Data Partitioning

28. Training with Scaled Queries (scaled 50x)
EDBT 2009 – Workload-Aware Data Partitioning

29. Training with Scaled Queries (scaled 400x)
EDBT 2009 – Workload-Aware Data Partitioning

30. Heat-based, Extent-based Training
EDBT 2009 – Workload-Aware Data Partitioning

31. Communication Overhead for Pobs
EDBT 2009 – Workload-Aware Data Partitioning

32. Throughput for Pobs
EDBT 2009 – Workload-Aware Data Partitioning

33. Load Balancing During Runtime
Complement workload-aware partitioning with runtime load-balancing
Short-term peaks
Master-slave approach
Load monitoring
Long-term trends
Based on load monitoring
Histogram evolution
EDBT 2009 – Workload-Aware Data Partitioning

34. HiSbase Related Work On-line load balancing
Hundreds of thousands to millions of nodes
Reacting fast
Treating objects individually
HiSbase
EDBT 2009 – Workload-Aware Data Partitioning

35. Should I Split or Replicate?
Many challenges and opportunities in e-science for database research
High-throughput data management
Correlation of distributed data sources
Community-driven data grids
Dealing with data skew and query hot spots
Workload-awareness by employing cost model during partitioning
EDBT 2009 – Workload-Aware Data Partitioning

36. Thank You for Your Attention
Get in Touch
Database systems group, TU München
Web site:
The HiSbase project
Thank You for Your Attention
EDBT 2009 – Workload-Aware Data Partitioning

37. Queries Intersecting Multiple Regions
EDBT 2009 – Workload-Aware Data Partitioning

38. Regions Without Queries
EDBT 2009 – Workload-Aware Data Partitioning

39. Throughput for Pobs (300 nodes, sim.)
EDBT 2009 – Workload-Aware Data Partitioning

40. Throughput for Pobs (1000 nodes, sim.)
EDBT 2009 – Workload-Aware Data Partitioning

41. Throughput (Region-Uniform Queries)
EDBT 2009 – Workload-Aware Data Partitioning