Network-Aware Operator Placement for Stream-Processing Systems CS253 project presentation Min Chen, Danhua Guo {michen, 12/4/2006

2 Roadmap Motivation Motivation Background Knowledge Background Knowledge Project Architecture Project Architecture Evaluation Evaluation Conclusion Conclusion

Motivation Stream-based overlay network (SBON) Stream-based overlay network (SBON) –Operator placement (RELAXATION) Designed for large-scale DSPS Designed for large-scale DSPS Network efficiency Network efficiency Low delay Low delay –Cost space

Motivation Cost Space Cost Space –Idea: Problem -> Virtual space -> physical node –Latency (& Load) dimensions

Background Knowledge Network Usage: A Blended Metric Network Usage: A Blended Metric –Bandwidth-delay product –Individual applications perspective Small delay Small delay –Network perspective Scaling bandwidth by latency Scaling bandwidth by latency

Architecture Overview Overview

Operator Placement Algorithm RELAXATION RELAXATION –Adapt to changing conditions –Decentralized –Considers impacts of shared placement decisions

Operator Placement Algorithm Algorithm Algorithm –Virtual Operator Placement

Operator Placement Algorithm Algorithm Algorithm –Physical Operator Mapping Find k nodes whose coordinates are near the target cost space coordinate. Sort the list by distance to the target coordinate. Walk the sorted list, returning the first node already running the operator. Failing that, return the nearest node that meets the application’s resource criteria

Operator Placement Result Δ=0.1 and Ft = 1 Δ=0.1 and Ft = 1

Evaluation Based on PlanetLab trace files Based on PlanetLab trace files –Efficiency compared with other algorithm Optimal: exhaustive search Optimal: exhaustive search Producer: place the operator at producer Producer: place the operator at producer Consumer: place the operator at consumer Consumer: place the operator at consumer Random: place the operator at a random node Random: place the operator at a random node

Evaluation Set up Set up –Bandwidth: 2KB/s –# of Nodes: 226 nodes from PlanetLab –# of Queries: 1000 –Query set up: 4 producers, 1 comsumer, 1 unpinned operator per query

Evaluation Result Result

Evaluation Delay penalty after placing 1000 queries Delay penalty after placing 1000 queries Algorithm Network Usage PenaltyDelay Penalty Optimal0.00%1.01% Relaxation1.34%7.71% Producer24.91%50.99% Consumer27.36%0.00% Random54.14%57.26%

Conclusion Our experiment shows that SBON is efficient in terms of Network Usage (Bandwidth, Latency)

Reference Peter Pietzuch et. al. “Network-Aware Operator Placement for Stream-Processing Systems” Peter Pietzuch et. al. “Network-Aware Operator Placement for Stream-Processing Systems” Peter Pietzuch, Jonathan Ledlie, and Margo Seltzer, “, In Proceedings of WORLDS 2005, San Francisco, CA, December 2005 Peter Pietzuch, Jonathan Ledlie, and Margo Seltzer, “Supporting Network Coordinates on PlanetLab”, In Proceedings of WORLDS 2005, San Francisco, CA, December 2005