Ph.D. Dissertation Defense Modeling and Evaluating Feedback-Based Error Control for Video Transfer PhD Candidate: Yubing Wang - Computer Science, WPI, EMC Corp. Committee: Prof. Mark Claypool - Computer Science, WPI Prof. Robert Kinicki - Computer Science, WPI Prof. Dan Dougherty - Computer Science, WPI Prof. Ketan Mayer-Patel – Computer Science, UNC at Chapel Hill

Ph.D. Dissertation Defense 2 Video Transfer 5 Video Frames Internet Client 4321 Frame Loss Capacity Constraint Server 531 Delay Constraint Too Late Error Propagation 43215

Ph.D. Dissertation Defense 3 Error Control 5 Video Frames Internet Client 4321 Server Retransmission NACK 3 Change Coding Parameter Local Concealment 3

Ph.D. Dissertation Defense 4 Motivation Frame loss degrades video quality Feedback-based error control techniques use information from decoder to repair Feedback indicates damage location. Encoder and decoder cooperate in error control process. Better than error control techniques where no interaction between encoder and decoder Major techniques: RPS, Intra Update, Retransmission Choice and Effectiveness depends on packet loss, RTT, video content and GOP size No systematic exploration and comparison of impact of video and network conditions on the performance of feedback-based error control techniques

Ph.D. Dissertation Defense 5 The Dissertation Analyze video quality with feedback based error control Develop analytical models to predict quality of videos streamed with RPS NACK, RPS ACK, Intra Update or Retransmission Conduct systematic study of effects of reference distance on video quality Validate analytical models through simulations Analysis of loss rate, round-trip time, video content, Group Of Pictures (GOP) Determine choice between RPS NACK, RPS ACK, Intra Update or Retransmission Publications Impact of Reference Distance for Motion Compensation Prediction on Video Quality, MMCN07 An Analytic Comparison of RPS Video Repair, MMCN08 Modeling RPS and Evaluating Video Repair with VQM, IEEE Transactions on Multimedia, 2009, (to appear)

Ph.D. Dissertation Defense 6 Outline Introduction Background RPS ACK RPS NACK Intra Update Retransmission Impact of Reference Distance on Video Quality Analytical Models and Results Model Validations Conclusions

Ph.D. Dissertation Defense 7 Reference Picture Selection (ACK) The decoder acknowledges all correctly received frames Only the acknowledged frames are used as a reference Error propagation is avoided entirely Distance from reference frame is reference distance Reference distance increases with round-trip delay Coding efficiency decreases as reference distance increases Video quality degrades as coding efficiency decreases ACK(1) ACK(2)ACK(3)

Ph.D. Dissertation Defense 8 Reference Picture Selection (NACK) The previous frame is used as a reference for encoding during the error-free transmission. Reference distance is always 1 regardless of RTT The decoder sends a NACK for the erroneous frame along with a reference frame number Error propagation Impact of loss increases with RTT NACK(3)

Ph.D. Dissertation Defense 9 Intra Update Upon receiving a NACK from the decoder, encodes the current frame with intra mode Frame is independently encoded without using any information from previous frames Coding efficiency is reduced because of intra coding NACK(4) Intra-coded

Ph.D. Dissertation Defense 10 Retransmission Retransmission of lost frames needs extra bandwidth Packets arriving after their display times are not discarded but instead are used to reduce error propagation Encoder Decoder NACK(3) 3

Ph.D. Dissertation Defense 11 Outline Introduction Background Impact of Reference Distance on Video Quality Hypothesis Methodology Results and Analysis Analytical Models and Results Model Validations Conclusions

Ph.D. Dissertation Defense 12 Impact of Reference Distance on Video Quality RPS selects one of several previous frames as a reference frame during compression Distance from selected frame is reference distance Higher reference distance, lower quality and vice versa How reference distance affects video quality has not been quantified A systematic study of the effects of reference distance on video quality Data is needed for modeling RPS

Ph.D. Dissertation Defense 13 Hypothesis The y-intersect is determined by motion and scene complexity. High-motion video sequences starts with low quality, degrade slower. Low-motion video sequence starts with high quality, degrade faster. Low Motion: The similarities among frames are high; More macro-blocks are inter-coded; High motion: The similarities among frames are low; More macro-blocks are intra-coded;

Ph.D. Dissertation Defense 14 Methodology Select a set of non-compressed video clips with a variety of motion content. All in YUV 4:2:2, CIF (352x288) Each video sequence contains 300 video frames with a frame rate of 30 fps. Change reference distances for each selected video sequence Encode the video clips using H.264 Measure video quality using Peak-Signal-to-Noise-Ratio (PSNR) Video Quality Metric (VQM) Analyze the results.

Ph.D. Dissertation Defense 15 PSNR vs. Reference Distance Video ClipsabR- Squared Akiyo Container News Silent Mom & Daughter Foreman Mobile Coastguard The relationship between PSNR and reference distance can be characterized using a logarithmic function:

Ph.D. Dissertation Defense 16 VQM vs. Reference Distance Video ClipsabR- Squared Akiyo Container News Silent Mom & Daughter Foreman Mobile Coastguard The relationship between VQM and reference distance can be characterized using a linear function:

Ph.D. Dissertation Defense 17 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Assumptions RPS ACK RPS NACK Intra Update Retransmission Result & Analysis Model Validations Conclusions

Ph.D. Dissertation Defense 18 Assumptions Each GOB is independent from other GOBs in the same frame. An independent video sub- sequence is referred to as a reference chain. Each GOB is carried in a single network packet. Reliable transmission of feedback messages are assumed. Erroneously-decoded GOBs are repaired by local concealment. Make no assumption on specific local concealment techniques Assume independent packet loss with a random loss distribution. In this talk, GOB and Frame is exchangeable.

Ph.D. Dissertation Defense 19 Model Parameters

Ph.D. Dissertation Defense 20 Modeling of RPS ACK The probability of decoding GOB (n) correctly using GOB (n-δ-i) as a reference: The probability of GOB (n) being successfully decoded is: p Packet loss probability Probability of GOB (n-δ-i) being successfully decoded Round-trip time Time-interval between two frames ACK(1) ACK(2)

Ph.D. Dissertation Defense 21 RPS ACK Modeling (cont.) The expected video quality for n-th GOB: Average video quality for a GOB encoded using the GOB that is r GOBs backward. Average video quality for a Intra-Coded GOB Average PSNR value for a GOB that is repaired using local concealment

Ph.D. Dissertation Defense 22 RPS NACK -- Model The probability of GOB (n) being successfully decoded: --- the probability of decoding GOB (n) correctly using GOB (n- δ -i) as a reference 1- p p pp p p p [1] (1) [2] (1) [1] (2) (1) (3)[3] GOB 1 GOB 2 GOB 3 GOB 4 (2) p p root [1] A B C D NACK(1) NACK(2) GOB Dependency Tree

Ph.D. Dissertation Defense 23 Intra Update -- Model The probability of GOB (n) being successfully decoded: -- the probability of decoding GOB (n) correctly using Intra coding NACK Intra-coded 1- p p pp p p p GOB 1 GOB 2 GOB 3 GOB 4 p p A E B root C D F GOB Dependency Tree

Ph.D. Dissertation Defense 24 Retransmission C apacity constraint : The n-th GOB in the reference chain being successfully decoded : The expected video quality for GOB (n):

Ph.D. Dissertation Defense 25 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Assumptions RPS ACK RPS NACK Intra Update Retransmission Result & Analysis Model Validations Conclusions

Ph.D. Dissertation Defense 26 Analytic Experiments Our analytical models consider a number of factors that may affect feedback-based repair performance: Reference distance change Loss probability Round-trip time Bitrate constraint Video content GOP Size Select a set of video clips with a variety of motion content

Ph.D. Dissertation Defense 27 Quality versus Round-Trip Time RPS ACK RPS NACK Quality degrades with round-trip time increase NACK resistant to degradation with round-trip time for low loss ACK degrades uniformly with round-trip time

Ph.D. Dissertation Defense 28 Quality versus Loss Rate RPS ACK RPS NACK Quality degrades with loss rate increase NACK degrades faster with high round trip times ACK uniform degradation

Ph.D. Dissertation Defense 29 RPS NACK vs. RPS ACK Above trend line, ACK better. Below trend line, NACK better Crossover points for low-motion are higher than for high-motion Error propagation more harmful to quality than reference distance

Ph.D. Dissertation Defense 30 Comparison RPS NACK performs best in low loss RPS ACK performs best in high loss RPS ACK performs worst in low loss Retransmission performs worst in high loss Intra Update performs as well as RPS NACK as RTT increases RTT=80 ms RTT=240 ms

Ph.D. Dissertation Defense 31 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Model Validations Methodology Results Conclusions

Ph.D. Dissertation Defense 32 Validation -- Methodology Randomly drop controllable number of frames in input sequence based on given loss probability Based on given round-trip time and randomly selected lost frames, regenerate video sequence Encode video sequence generated in step 2 using H.264 Measure average PSNR and VQM for encoded H.264 video sequence Calculate average PSNR and VQM based upon video quality measured in step 4 1(I) 2(P) 5(P) 6(P) 7(P) RPS NACK, round-trip time = 2 frames, frame 3 is lost

Ph.D. Dissertation Defense 33 Validation – RPS NACK Error bar represents 95% confidence interval As loss probability or round-trip time increases, the variance is increased Simulation results are consistent with values predicted by analytical model for both PSNR and VQM

Ph.D. Dissertation Defense 34 Outline Introduction Background Impact of Ref. Distance on Video Quality Analytical Models and Results Model Validations Conclusions

Ph.D. Dissertation Defense 35 Major Contributions 1.Systematic study of effects of reference distance on video quality for a range of video coding conditions 2.Two utility functions that characterize impact of reference distance on video quality based upon study 3.Modeling prediction dependency among GOBs for RPS NACK and Intra Update using binary tree 4.Analytical models for feedback-based error control techniques including Full Retransmission, Partial Retransmission, RPS ACK, RPS NACK and Intra Update 5.Simulations that verify accuracy of our analytical models 6.Analytic experiments over a range of loss rates, round-trip times and video content using our models

Ph.D. Dissertation Defense 36 Future Work Explore and incorporate other existing video quality metrics or develop a new quality metric Investigate how local concealment may affect the choice of feedback-based repair techniques Investigate the impact of the extra bandwidth consumed by feedback messages on performance Build a videoconference system that automatically adapts to the best repair techniques

Ph.D. Dissertation Defense 37 Conclusions Degree of video quality degradation is affected by video content High-motion video sequences starts with lower quality, degrade slower. Low-motion video sequences starts with higher quality, degrade more rapidly. Mathematical Characterization of the relationship between video quality and reference distance: PSNR: VQM: Analytical models reveal: RPS NACK performs best in low loss RPS ACK performs best in high loss, worst in low loss RPS NACK outperforms RPS ACK over a wider range for low motion videos than for high motion videos Retransmission performs worst in high loss Intra Update performs as well as RPS NACK as RTT increases

Ph.D. Dissertation Defense 38 Acknowledge Prof. Claypool and Prof. Kinicki Prof. Dougherty Prof. Mayer-Patel from UNC at Chapel Hill Faculty/Staff of Computer Science Dept., WPI Huahui Wu, Mingze Li, Feng Li, and everyone from PEDS and CC groups Attendees today My Family