Project Proposal Due Next Mon. Submit by Email Reminder Project Proposal Due Next Mon. Submit by Email

Problems with Internet Limited Bandwidth Varying Bandwidth Delay Delay Jitter Packet Loss Heterogeneity : NUS.SOC.CS5248 OOI WEI TSANG

Error Recovery

You Are Here Encoder Decoder Middlebox Sender Receiver Network NUS.SOC.CS5248 OOI WEI TSANG

Overview Characteristics of the Internet General techniques Error recovery for audio Effect of loss on MPEG Error recovery for MPEG NUS.SOC.CS5248 OOI WEI TSANG

Loss Characteristics of The Internet

Characteristics of Internet 60-70% of paths do not show any loss Those with loss have an average of 4.5 – 6% packet loss [Paxson97] End-to-end Internet packet dynamics NUS.SOC.CS5248 OOI WEI TSANG

Packet Loss Pattern 1000 30 NUS.SOC.CS5248 OOI WEI TSANG

Characteristic of Internet Bursts of loss are typically short (2-3 consecutively loss packets) Long burst do occur Burst may occur periodically NUS.SOC.CS5248 OOI WEI TSANG

Wireless Link Loss rate measured in my office 10 ~ 20% up to 50% reported! NUS.SOC.CS5248 OOI WEI TSANG

A Survey of Packet-Loss Recovery Techniques for Streaming Audio Perkins, Hodson and Hardman IEEE Network Magazine 1998

Retransmission 1 2 3 4 3 1 2 4 3 NUS.SOC.CS5248 OOI WEI TSANG

Redundant Data 1 2 2 3 3 4 4 5 1 2 2 3 3 4 5 NUS.SOC.CS5248 OOI WEI TSANG

Error Concealment 1 2 3 4 1 2 4 NUS.SOC.CS5248 OOI WEI TSANG

Pros and Cons retransmission redundant data error concealment NUS.SOC.CS5248 OOI WEI TSANG

Retransmission for audio

Retransmission is Bad Need scalable feedback NACK consumes bandwidth May need to retransmit frequently Retransmission maybe useless NUS.SOC.CS5248 OOI WEI TSANG

Retransmission is Good Only retransmit when needed NUS.SOC.CS5248 OOI WEI TSANG

Scalable Retransmission On packet loss T = random(0, RTT) wait for T multicast NACK On receiving NACK from others suppress own NACK NUS.SOC.CS5248 OOI WEI TSANG

Retransmit when group size is small loss rate is low large latency acceptable NUS.SOC.CS5248 OOI WEI TSANG

Redundant Data for audio

Parity FEC 1011001 1000010 0010010 0001001 XOR NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC 1011001 1000010 0001001 0010010 NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC Ordering NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC Ordering NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC Ordering NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC Ordering NUS.SOC.CS5248 OOI WEI TSANG

Parity FEC Ordering NUS.SOC.CS5248 OOI WEI TSANG

Reed-Solomon Code RS(n,k) RS NUS.SOC.CS5248 OOI WEI TSANG

Reed-Solomon Code NUS.SOC.CS5248 OOI WEI TSANG

Media Specific FEC 1 2 2 3 3 4 4 5 1 2 2 3 3 4 5 NUS.SOC.CS5248 OOI WEI TSANG

Interleaving NUS.SOC.CS5248 OOI WEI TSANG

Error Concealment

Effect of Loss on Audio Speech Human ears can interpolate Loss up to length of phoneme can still be tolerable NUS.SOC.CS5248 OOI WEI TSANG

Insertion-based Repair Splice Silence Substitution Noise Substitution Repetition NUS.SOC.CS5248 OOI WEI TSANG

Other Repair Methods Interpolation Regeneration NUS.SOC.CS5248 OOI WEI TSANG

Colin’s Recommendations

Non-Interactive Apps Interleaving FEC Retransmission for unicast only NUS.SOC.CS5248 OOI WEI TSANG

Interactive Applications Media Specific FEC NUS.SOC.CS5248 OOI WEI TSANG

Error Concealment Repeat NUS.SOC.CS5248 OOI WEI TSANG

Packet Loss Effects on MPEG Video Sent over the Public Internet Jill Boyce and Robert Gaglianello ACM Multimedia 1998

Measurement-based Study Need to understand the problem before proposing solution NUS.SOC.CS5248 OOI WEI TSANG

NUS.SOC.CS5248 OOI WEI TSANG

Data Gathering Method From: NYC13 , Austin21, London18 To: Holmdel, NJ NUS.SOC.CS5248 OOI WEI TSANG

Data Gathering Method Video: Two 5-mins MPEG 30 fps 384 kbps and 1 Mbps IBBPBBPBBPBBPBB QSIF 176x112 and SIF 352x240 One row per slice NUS.SOC.CS5248 OOI WEI TSANG

Average Packet Loss NUS.SOC.CS5248 OOI WEI TSANG

Packet Loss vs Time NUS.SOC.CS5248 OOI WEI TSANG

How to Fill Packets B 1 frame 1 packet P 1 slice 1 packet I 1 slice 1 packet NUS.SOC.CS5248 OOI WEI TSANG

Packet Size Distributions NUS.SOC.CS5248 OOI WEI TSANG

Packet Size vs Loss Rate NUS.SOC.CS5248 OOI WEI TSANG

Lesson 1 slice per packet even if larger than MTU If smaller packets means higher loss rate: fill packet until MTU Else one slice per MTU Alt: Change size of slice NUS.SOC.CS5248 OOI WEI TSANG

Frame Effected by Errors NUS.SOC.CS5248 OOI WEI TSANG

Correlation of Packet Loss NUS.SOC.CS5248 OOI WEI TSANG

Overview of Error Recovery for Video

Methods Retransmission Error Concealment FEC Limiting Error Propagation Reference Frame Selection NUS.SOC.CS5248 OOI WEI TSANG

Reference Frame Selection I P B B P I X NUS.SOC.CS5248 OOI WEI TSANG

Methods Retransmission Error Concealment FEC Limiting Error Propagation Reference Frame Selection Changing Temporal Pattern NUS.SOC.CS5248 OOI WEI TSANG

Error Control Techniques for Interactive Low Bitrate Video Transmission over The Internet Injong Rhee SIGCOMM ‘98

Basic Idea “Better Late Than Never!”, or Late packet is still useful NUS.SOC.CS5248 OOI WEI TSANG

MPEG Frame Pattern I B B P B B P I P B B P B B NUS.SOC.CS5248 OOI WEI TSANG

H.261 Error Propagation X X X X X X X X X X I P P P P P P loss I P retransmission NUS.SOC.CS5248 OOI WEI TSANG

H.261 Frame Pattern I P P P P P P X loss NUS.SOC.CS5248 OOI WEI TSANG

H.261 Frame Pattern X X X I P retransmission loss NUS.SOC.CS5248 OOI WEI TSANG

PTDD Periodic Temporal Dependency Distance Large PTDD Later Deadlines Longer Error Propagations Less Temporal Dependencies NUS.SOC.CS5248 OOI WEI TSANG

QAL I P P P Enhancement Layer I P P P Base Layer NUS.SOC.CS5248 OOI WEI TSANG

QAL X I P P P Enhancement Layer I P P P Base Layer + FEC NUS.SOC.CS5248 OOI WEI TSANG

QAL + PTDD I P P P Enhancement Layer I P P P Base Layer NUS.SOC.CS5248 OOI WEI TSANG

Error Propagation

Recovery from Error Propagation

Packet Loss Recovery for Streaming Video N. Feamster, H. Balakrishnan PV’02

Overview An analytical model for packet loss SR-RTP Post Processing NUS.SOC.CS5248 OOI WEI TSANG

Mathematical Model Lets model packet loss effects and see what it can tell us NUS.SOC.CS5248 OOI WEI TSANG

SI : mean num. of packets in I-frames P(I) : prob. frame type is I p : prob. packet loss SI : mean num. of packets in I-frames P(I) : prob. frame type is I P(F): prob. frame is useless NUS.SOC.CS5248 OOI WEI TSANG

SP : mean num. of packets in P-frames Np : number of P-frames in GOP p : prob. packet loss SP : mean num. of packets in P-frames Np : number of P-frames in GOP P(P): prob. frame type is P P(F): prob. frame is useless NUS.SOC.CS5248 OOI WEI TSANG

SB : mean num. of packets in B-frames NB : number of B-frames in GOP p : prob. packet loss SB : mean num. of packets in B-frames NB : number of B-frames in GOP P(B): prob. frame type is B P(F): prob. frame is useless NUS.SOC.CS5248 OOI WEI TSANG

Average Frame Rate NUS.SOC.CS5248 OOI WEI TSANG

Is it accurate? NUS.SOC.CS5248 OOI WEI TSANG

SR-RTP Extensions to RTP for selective reliability NUS.SOC.CS5248 OOI WEI TSANG

SR-RTP Length ADU Sequence Number ADU Length ADU Offset Priority Length ADU Sequence Number ADU Length ADU Offset Priority Layer Number NUS.SOC.CS5248 OOI WEI TSANG

Feamster & Balakrishnan’s ADU 1 ADU = 1 Frame NUS.SOC.CS5248 OOI WEI TSANG

Loss Detection LEN: 50 ADU: 0 ADU Size: 100 ADU Off: 0 LEN: 20 ADU: 0 NUS.SOC.CS5248 OOI WEI TSANG

Retransmission Request Based on priority Example: I-Frame: highest priority P-Frame: higher priority if closer to prev I B-Frame: no retransmition NUS.SOC.CS5248 OOI WEI TSANG

Error Concealment for Video Repeat pixels from previous frame Interpolate pixels from neighbouring region Interpolate motion vectors from previous frame NUS.SOC.CS5248 OOI WEI TSANG

Inferred MV for P Frame I P NUS.SOC.CS5248 OOI WEI TSANG

Inferred MV for I Frame P P I NUS.SOC.CS5248 OOI WEI TSANG

Evaluation Peak Signal to Noise Ratio (PSNR) 2552 over average square of differences in pixel values Calculated in dB (10 log 10) NUS.SOC.CS5248 OOI WEI TSANG

Performance of SR-RTP NUS.SOC.CS5248 OOI WEI TSANG

Today’s Summary

How to recover packet loss Retransmission FEC Error Concealment NUS.SOC.CS5248 OOI WEI TSANG

How to limit the damage of error Interleaving Key frame selection Change reference frame NUS.SOC.CS5248 OOI WEI TSANG

Demo

Relate to CS5248 Really bursty audio loss Error concealment with noise Really bursty video loss, or long video propagation Need to protect important content (content-based unequal error protection) NUS.SOC.CS5248 OOI WEI TSANG