1. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 1 Error Recovery

2. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 2 Problems with Internet Limited Bandwidth Varying Conditions Delay Jitter Packet Loss Delay Heterogeneity :

3. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 3 Network Encoder Sender Middlebox Receiver Decoder You Are Here

4. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 4 Overview Characteristics of the Internet General techniques Error recovery for audio Effect of loss on MPEG Error recovery for MPEG

5. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 5 Loss Characteristics of The Internet

6. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 6 Characteristics of Internet 60-70% of paths do not show any loss Those with loss have an average of 4.5 – 6% packet loss [Paxson97] End-to-end Internet packet dynamics

7. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 7 Packet Loss Pattern 1000 30 Number of Occurrences Burst Length

8. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 8 Packet Loss Pattern 1000 30 Number of Occurrences Burst Length

9. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 9 Characteristics of Internet Bursts of loss are typically short (2-3 consecutively loss packets) Long bursts do occur Bursts may occur periodically

10. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 10 Wireless Link Loss rate measured in various environments (e.g., office) 10 ~ 20% Up to 50% reported!

11. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 11 Gilbert Model GOODBAD

12. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 12 Gilbert Model GOODBAD p q 1-q1-p

13. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 13 A Survey of Packet-Loss Recovery Techniques for Streaming Audio Perkins, Hodson and Hardman IEEE Network Magazine 1998

14. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 14 Three Fundamental Techniques 1. Retransmission 2. Redundant data 3. Error concealment

15. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 15 1. Retransmission 1234 124 3 3 X

16. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 16 2. Redundant Data 1234 124 2 23 345 53

17. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 17 24 3. Error Concealment 1234 1

18. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 18 Retransmission for audio

19. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 19 Pros/Cons of Retransmissions Pros Low bandwidth overhead Cons Additional delay 2-way channel needed

20. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 20 Scalable Retransmission On packet loss T = random(0, RTT) wait for T multicast NACK On receiving NACK from others suppress own NACK

21. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 21 Retransmit when Group size is small Loss rate is low Large latency acceptable

22. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 22 Selective Retransmission Retransmission based on priority of packets Important/urgent packets are retransmitted first Packets are only retransmitted when there is enough time [Papadopoulos and Parulkar, NOSSDAV 1996]

23. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 23 Redundant Data for audio

24. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 24 Parity Forward Error Correction 1011001 1000010 0001001 XOR 0010010

25. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 25 Parity Forward Error Correction 1011001 1000010 0001001 0010010

26. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 26 Parity FEC Ordering

27. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 27 Parity FEC Ordering

28. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 28 Parity FEC Ordering

29. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 29 Parity FEC Ordering

30. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 30 Parity FEC Ordering

31. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 31 Reed-Solomon Code RS(n,k) RS n k (popular: n = 223) (popular: k = 32) Parity symbols Data symbols

32. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 32 Reed-Solomon Code Data block (length n+k, e.g., 255) Can correct up to 16 corrupted symbols per block (32/2)

33. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 33 Media Specific FEC 1234 124 2 23 345 53

34. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 34 Pros/Cons of Redundant Data Pros Small additional delay Only 1-way channel needed Cons Computational complexity Limited correction ability Additional bandwidth overhead

35. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 35 Interleaving

36. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 36 Pros/Cons of Interleaving Pros No bandwidth overhead “Several small errors vs. one large error” Cons Additional delay

37. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 37 Error Concealment

38. Recreate Lost Information NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 38

39. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 39 Effect of Loss on Audio Speech Human ears can interpolate Loss up to length of phoneme can still be tolerable

40. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 40 Insertion-based Repair Splice Silence Substitution Noise Substitution Repetition

41. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 41 Other Repair Methods Interpolation Regeneration

42. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 42 Error Concealment More complex error concealment algorithms provide better performance (i.e., playback quality)

43. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 43 Pros/Cons of Error Concealment Pros No bandwidth overhead No (small) additional delay Cons Computational complexity May result in visual artifacts May result in propagation errors

44. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 44 Colin’s Recommendations

45. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 45 Non-Interactive Apps Interleaving FEC Retransmission for unicast only

46. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 46 Interactive Applications Media Specific FEC

47. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 47 Error Concealment Repeat

48. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 48 Packet Loss Effects on MPEG Video Sent over the Public Internet Jill Boyce and Robert Gaglianello ACM Multimedia 1998

49. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 49 Measurement-based Study Need to understand the problem before proposing solution

50. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 50 Data Gathering Method From: NYC 13, Austin 21, London 18 To: Holmdel, NJ

51. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 51 Data Gathering Method Video: Two 5-mins MPEG 30 fps 384 kbps and 1 Mbps GOP: IBBPBBPBBPBBPBB QSIF 176x112 and SIF 352x240 One row per slice

52. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 52 Average Packet Loss

53. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 53 Frames Affected by Errors (1) % Frames in Error Packet Loss Rate

54. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 54 Frames Affected by Errors (2)

55. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 55 Overview of Error Recovery for Video

56. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 56 Methods Retransmission Interleaving Error Concealment FEC Limiting Error Propagation Reference Frame Selection

57. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 57 Reference Frame Selection I P B B P X I

58. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 58 Methods Retransmission Interleaving Error Concealment FEC Limiting Error Propagation Reference Frame Selection Changing Temporal Pattern

59. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 59 Case Study: HD video conferencing Korean Air, Inha University (Seoul), University of Southern California, Pratt & Whitney

60. © PWICE, 200660 USC-Inha Two-way Experiments USC (Powell Hall of Engineering) July 2005

61. © PWICE, 200661 USC-Inha Two-way Experiments Inha University (Memorial Library) July 2005

62. © PWICE, 200662 KAL Network Infrastructure Korean Air network infrastructure between Incheon, Kimpo, and Pusan KAL network & external ISP (KT) 1 Gbps: ICN-GMP 50 Mbps: GMP-PUS December 2005

63. © PWICE, 200663 Network Measurements End-to-end packet loss rates of the network path between Kimpo (GMP) and Incheon (ICN) observed during November 15-16, 2005 Network is shared with other business traffic Loss rates are quite low November 2005

64. © PWICE, 200664 Hardware Preparation 3 computers purchased in Korea Computers sent to USC with KAL Software setup in the IMSC laboratories Linux installation RCS software installation Retransmission configuration End-to-end equipment test with local JVC cameras and displays Computers sent back to KAL HQ December 2005

65. © PWICE, 200665 Hardware Test at KAL Re-configuration for Sony HDR-HC1 cameras December 2005 Local area network tests

66. © PWICE, 200666 Experiments: ICN ◄▬► GMP Asymmetric environment: 88 Mb/s & 50 Mb/s Conclusion: ICN - GMP: sufficient BW for HD video Sufficient headroom for data applications December 2005 Bandwidth measurements Visual quality Required by RCS

67. © PWICE, 200667 Experiments: ICN ◄▬► PUS Asymmetric environment: 6 Mb/s & 1 Mb/s GMP - PUS: initially insufficient BW for HD video Investigated network routers with KAL IT help Conclusion: replaced router to improve throughput; to be re-tested December 2005 Bandwidth measurements Visual quality

68. © PWICE, 200668 Measurements: Packet Loss Rate Path: ICN - PUS Two packet sizes: 564 bytes and 940 bytes Conclusion: GMP - PUS loss rate very high December 2005 One-way packet loss rate Two-way packet loss rate

69. © PWICE, 200669 Packet Loss Error Recovery Selective retransmission protocol is used to recover from lost data Effectiveness of protocol is tested in lab environment Gilbert Model is used to induce losses into transmission Transmission delay is chosen to be 10 ms (expected latency in Korea)

70. © PWICE, 200670 Packet Loss Error Recovery Selective retransmission protocol 10% Loss Rate 5% Loss Rate 1% Loss Rate With Retransmissions Sept. - Dec. 2005 (ICN-GMP Link)

71. Current Hardware Codecs NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 71 Hauppauge HD-PVR Cavium PureVu™ CNW31XX

72. HD-PVR H.264 Encoder NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 72

73. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 73

74. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 74 Error Control Techniques for Interactive Low Bitrate Video Transmission over The Internet Injong Rhee SIGCOMM ‘98

75. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 75 Basic Idea “Better Late Than Never!”, or Late packet is still useful

76. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 76 MPEG Frame Pattern IBBPBBP IPBBPBB

77. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 77 H.261 Error Propagation IPPPPPP XXXXXX IPPPPPP XXXX retransmission loss

78. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 78 H.261 Frame Pattern IPPPPPP X loss

79. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 79 IPPPPPP H.261 Frame Pattern X loss X X retransmission

80. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 80 PTDD Periodic Temporal Dependency Distance Large PTDD Higher Chance of Repair Longer Error Propagations Less Temporal Redundancy

81. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 81 QAL IPPP IPPP Base Layer Enhancement Layer

82. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 82 QAL IPPP IPPP Base Layer + FEC Enhancement Layer X

83. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 83 QAL + PTDD IPPP IPPP Base Layer Enhancement Layer

84. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 84 Error Propagation

85. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 85 Recovery from Error Propagation

86. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 86 Frame “quality” PSNR =

87. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 87 PSNR vs. Loss Rate Packet Loss Rate PSNR

88. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 88 Today’s Summary

89. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 89 How to recover packet loss Retransmission FEC Error Concealment

90. NUS.SOC.CS5248-2014 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 90 Limit the damage of error Interleaving Key frame selection Change reference frame