Advances in Network-adaptive Video Streaming Bernd Girod J. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. Zhang Information Systems Laboratory Department of Electrical Engineering Stanford University

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 2 Streaming media: a huge success Hundreds of thousands of streaming media servers deployed > 1 million hours of streaming media content produced per month Hundreds or millions streaming media players RealPlayer –Most popular Internet application second only to Internet Explorer [Media Metrix] –More than 400 million unique registered users –More than 200,000 new users per day –Open source code

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 3 Best-effort packet network low bit-rate variable throughput variable loss variable delay Challenges compression rate scalability error resiliency low latency Challenges compression rate scalability error resiliency low latency Internet Media Streaming Streaming client DSL 56K modem Media Server Internet wireless

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 4 Outline What is network-adaptive video streaming? –Better delivery of video packets by considering source coding, signal processing, and packet transport jointly –Application-layer joint source-channel coding techniques for the Internet This talk: review recent advances in 1.Adaptive media playout 2.Rate-distortion optimal packet scheduling 3.Network-adaptive packet dependency management

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 5 Adaptive Media Playout Fixed deadline Flexible deadline Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions 5% packet loss 2 sec average receiver buffer 5% packet loss 2 sec average receiver buffer [Steinbach, Färber, Girod, ICIP 2001]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 6 Modification of Playout Speed Video: adaptation of display rate Audio and speech: Stretching based on time-domain interpolation algorithm WSOLA [Verhelst et al., 1993, Liang 2001] Output packet 1/20/12/33 4 Original packet Pitch- period Template

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 7 Audio Speed Adjustment Waveform Similarity Overlap Add (WSOLA) method allows speed adjustment without changing pitch Speech demo Music demo original slower 30% faster 30% original slower 30% faster 30%

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 8 Reduced Pre-roll Time for Stored Streams G =1.092, B =0.42, T G =20 sec, T B =2 sec, T RTT =220 ms Probability of buffer underflow < 1% Probability of buffer underflow < 1% [Kalman, Steinbach, Girod, ISCAS 2002]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 9 Rate Scalability by Playout Speed Adjustment Server 20 kbps 50 kbps 95 kbps 25 kbps 55 kbps 100 kbps Channel (mean throughput) 85 kbps (29.2 dB, 53.3 kbps) (33.1 dB, 93.6*0.9=84.2 kbps)

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 10 First Things First: Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet Request stream Request stream Rate-distortion preamble Rate-distortion preamble Prefetch times Prefetch times Video data Repeat request Repeat request Repeat request Repeat request Repeat request Repeat request

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 11 Streaming as a Packet Scheduling Problem Which media units should be selected for transmission, and when? Requirements –Meet rate constraint –Meet latency constraint –Maximize reconstruction quality Rate-distortion framework proposed, e.g., in [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001] time pre-encoded media units transmission opportunities server

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 12 Markov Decision Tree for One Packet... N transmission opportunities before deadline send: 1 ack: send: ack: t current t current +  tt current +2  t Action Observation “Policy“ minimizing J = D + R “Policy“ minimizing J = D + R

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 13 Packet Delay Jitter and Loss delay     pdf lead-time loss probability lead-time loss probability loss

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 14 Source Description Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBBPPPI I BBBP A … … … A…

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 15 PB Source Description Each media packet n is labeled by − B n — size [in bits] of data unit n −  d n —distortion reduction if n is decoded − t n — decoding deadline for n PPI I BBPPI I BBBP A … … … A… For video:  d n must be made “state-dependent” to accurately capture concealment For video:  d n must be made “state-dependent” to accurately capture concealment

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 16 R-D Optimized Streaming Performance Foreman 120 frames 10 fps, I-P-P-… H Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms Foreman 120 frames 10 fps, I-P-P-… H Layer SNR scalable 20 frame GOP Copy Concealment 20 % loss forward and back Γ-distributed delay –κ = 10 ms –μ = 50 ms –σ = 23 ms Pre-roll 400ms PSNR [dB] Bit-Rate [kbps]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 17 R-D Optimized Streaming with a Proxy Server Media Server Switch/ Router Proxy Server Client Backbone Network Packet Last hop Receiver-driven RaDiO streaming Sender-driven RaDiO streaming Buffer packets [Chakareski, Chou, Girod, Asilomar 2002, MMSP 2002]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 18 RaDiO Edge: Experiment Video: Foreman, QCIF, 130 frames Compression: H layer temporal scalability 72…144 kbps [Liang, 2002] Backbone –Packet loss rate: 10% –Delay: shifted  -distribution Last hop –Packet loss rate: 1% –Delay: shifted  -distribution

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 19 Streaming with Diversity Media Server Client Channel 1 Channel 2 Channel N Media Server 1 Client Channel 1 Channel 2 Channel N Media Server 2 Media Server N Packet Path Diversity Server Diversity

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 20 R-D Optimized Streaming over 2 Channels Video: Foreman, QCIF, 130 frames Compression: H layer SNR scalability 32/64 kbps 2 identical, independent 2-state Markov channels Good/bad packet loss rates: 3%/15%  -distributed delays short/long Video: Foreman, QCIF, 130 frames Compression: H layer SNR scalability 32/64 kbps 2 identical, independent 2-state Markov channels Good/bad packet loss rates: 3%/15%  -distributed delays short/long [Chakareski, Girod, DCC 2003]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 21 R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) Latency: 100 ms RTT: 100 ms 3 dB 47 % PSNR, in dB Bit-Rate, in kbps

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 22 R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) Latency: 100ms Multiple Deadlines Rate: 68.8 kbps Mean PSNR: 27.0 dB Single Deadline Rate: 89.0 kbps Mean PSNR: 23.9 dB

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 23 Network-adaptive Packet Dependency Management Very low latency: no retransmissions Highly robust compressed representation by network-adaptive management of packet dependencies Utilize ACK/NACK in source coder H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe prediction Time Transmission error NACK

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 24 Error Resilience vs. Coding Efficiency P1 P2 I P5 230 frames of Foreman coded using H.26L TML8.5. Average PSNR=33.4dB

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 25 Rate-Distortion Optimal Reference Picture Selection [Wiegand, Färber, Girod, 2000] [Liang, Girod, 2002]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 26 RD Performance of Optimal Reference Picture Selection LTM buffer: V=5 frames Feedback round-trip time: d fb =7 frames Packet loss rate: p=10% Comparison with P-I scheme, where each NACK triggers insertion of I-frame [Liang, Girod, 2002]

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 27 ORPS Performance over Time Axis

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 28 MaD Sequence at 10% packet loss No Retransmissions Optimal Reference Picture Selection Rate: 320 kbps Mean PSNR: 39.2 dB Adaptive I-Frame Insertion Rate: 320 kbps Mean PSNR: 38.2 dB

Bernd Girod et al.: Advances in Network-adaptive Video Streaming 29 Conclusions Network-adaptive video streaming: jointly optimize compression, error control, packet transport, and decoding Adaptive media playout: “real-time” more flexible than we thought RaDiO streaming can provide virtual priority mechanisms Proxy servers with RaDiO can improve streaming performance Path Diversity can improve streaming performance RaDiO streaming with multiple deadlines for accelerated retroactive decodingof late retransmissions (“catch-up decoding”) Packet dependency management allows robust representations for streaming w/o retransmissions