SHADOWSTREAM: PERFORMANCE EVALUATION AS A CAPABILITY IN PRODUCTION INTERNET LIVE STREAM NETWORK ACM SIGCOMM CING-YU CHU

MOTIVATION Live streaming is a major Internet application today Evaluation of live streaming Lab/testbed, simulation, modeling Scalability realism Live testing

CHALLENGE Protection Real views’ QoE Masking failures from real viewers Orchestration Orchestrating desired experimental scenarios (e.g., flash-crowd) Without disturbing QoE

MODERN LIVE STREAMING Complex hybrid systems Peer-to-peer network Content delivery network BitTorrent-like Tracker  peers watching same channel  overlay network topology Basic unit: pieces

MODERN LIVE STREAMING Modules P2P topology management CDN management Buffer and playpoint management Rate allocation Download/upload scheduling Viewer-interfaces Share bottleneck management Flash-crowd admission control Network-friendliness

METRICS Piece missing ratio Pieces not received by the playback deadline Channel supply ratio Total bandwidth capacity (CDN+P2P) to total streaming bandwidth demand

MISLEADING RESULTS SMALL-SCALE EmuLab: 60 clients vs. 600 clients Supply ratio Small: 1.67 Large: 1.29 Content bottleneck!

MISLEADING RESULTS SMALL-SCALE With connection limit CDN server’s neighbor connections are exhausted by those clients that join earlier

MISLEADING RESULTS MISSING REALISTIC FEATURE Network diversity Network connectivity Amount of network resource Network protocol implementation Router policy Background traffic

MISLEADING RESULTS MISSING REALISTIC FEATURE LAN-like network vs. ADSL-like network Hidden buffers ADSL has larger buffer but limited upload bandwidth

SYSTEM ARCHITECTURE

STREAMING MACHINE self-complete set of algorithms to download and upload pieces Multiple streaming machines experiment (E) Play buffer

R+E TO MASK FAILURES Another streaming machine For protection repair (R)

R+E TO MASK FAILURES Virtual playpoint Introducing a slight delay To hide the failure from real viewers R = rCDN Dedicated CDN resources Bottleneck

R = PRODUCTION Production streaming engine Fine-tuned algorithms (hybrid architecture) Larger resource pool More scalable protection Serving clients before experiment starts

PROBLEM OF R = PRODUCTION Systematic bias Competition between experiment and production Protect QoE  higher priority for production  underestimate experiment

PCE R = P + C C: CDN (rCDN) with bounded resource P: production δ

PCE rCDN as a filter It “lowers” the piece missing ratio curve of experiment visible by production down by δ

IMPLEMENTATION Modular process for streaming machines Sliding window to partition downloading tasks

STREAMING HYPERVISOR Task window management: sets up sliding window Data distribution control: copies data among streaming machines Network resource control: bandwidth scheduling among stream machines Experiment transition

STREAMING HYPERVISOR

TASK WINDOW MANAGEMENT Informs a streaming machine about the pieces that it should download

DATA DISTRIBUTION CONTROL Data store Shared data store Each streaming machine  pointer

NETWORK RESOURCE CONTROL Production bears higher priority LED-BAT to perform bandwidth estimation Avoid hidden buffer network congestion

EXPERIMENT ORCHESTRATION Triggering Arrival Experiment Transition Departure

SPECIFICATION AND TRIGGERING Testing behavior pattern Multiple classes Each class Arrival rate function during interval Duration function L Triggering condition t start

ARRIVAL Independent arrivals to achieve global arrival pattern Network-wide common parameters t star t, t exp and λ(t) Included in keep-alive message

EXPERIMENT TRANSITION Current t 0, join at a e,i  [t 0, a e,i ] Connectivity Transition Production  neighbor’s production (not in test) Production rejoins

EXPERIMENT TRANSITION Playbuffer State Transition Legacy removal

DEPARTURE Early departure Capturing client state  snapshot Using disconnection message Substitution Arrival process again Only equal or more frequent than the real viewer departure pattern

EVALUATION Software Framework Experimental Opportunities Protection and Accuracy Experiment Control Deterministic Replay

SOFTWARE FRAMEWORK Compositional Run-time Block-based architecture Total ~8000 lines of code Flexibility

EXPERIMENTAL OPPORTUNITIES Real traces from 2 living streaming testing channel (impossible in testbed) Flash-crowd No client departs

PROTECTION AND ACCURACY EmuLab (weakness) Multiple experiment with same settings 300 clients δ ~ 4% Buggy code!

EXPERIMENT CONTROL Trace-driven simulation Accuracy of distributed arrivals Impact of clock synchronization Up to 3 seconds

DETERMINISTIC REPLAY Minimize logged data Hypervisor Protocol packet: whole payload Data packet: only header