1 ARMOR-Adjusting Repair and Media Scaling with Operations Research for Streaming Video PhD Candidate: Huahui Wu - Computer Science, Worcester Poly. Inst. Committee: Prof. Mark Claypool - Computer Science, Worcester Poly. Inst. Prof. Robert Kinicki - Computer Science, Worcester Poly. Inst. Prof. Craig Wills - Computer Science, Worcester Poly. Inst. Prof. Wu-chi Feng – Computer Science, Portland Stat Univ.

Ph.D. Defense205/01/2006 Acknowledge  Prof. Claypool and Prof. Kinicki  Prof. Wills  Prof. Wu-Chi Feng from Portland State Univ.  Faculty/Staff of Computer Science Dept., WPI  Jae Chung, Feng Li, Mingzhe Li and Rui Lu  User study participants  Attendees today  My Family

Ph.D. Defense305/01/2006 Introduction - Motivation Video Frames Repair by Forward Error Correction (FEC)

Ph.D. Defense405/01/2006 Operations Research Concept More Repair and More Scaling Video Quality Optimal Point  Adjusting Repair and Media Scaling –Given Network and Application Environment –For each valid FEC and scaling combination, measure the video quality –Find the optimal point

Ph.D. Defense505/01/2006 The Dissertation Repair (FEC)ScalingApproachPublications Media Independent No Scaling[NOSSDAV 03] [PV 03 poster] Temporal Scaling [TOMCCAP 05] [ACM MM 06 in Reviewing] Quality Scaling [NOSSDAV 05] [ACM MM 04 Demo] Combination[NOSSDAV 06] Media Dependent Quality Scaling M: Video Quality Model A: Optimization Algorithm U: User Study S: Simulation I: Implementation MAU SI MA US MA MA MA

Ph.D. Defense605/01/2006 Outline  Introduction  Background   Models  Algorithms  User Study  Implementation  Contributions  Conclusions

Ph.D. Defense705/01/2006 Video Compression Standard  MPEG –Popular compression standard –Intra-compression and inter-compression –Three types of frames: I, P and B –Group Of Pictures (GOP)  ARMOR models MPEG dependencies

Ph.D. Defense805/01/2006 Forward Error Correction (FEC)  Media-Independent FEC –Reed-Solomon codes [Reed+ 60]  ARMOR models benefits of FEC for frame transmission

Ph.D. Defense905/01/2006 Media Scaling  Sacrifice data to fit the capacity  Temporal Scaling (TS) –Pre-Encoding Temporal Scaling –Post-encoding Temporal Scaling 

Ph.D. Defense1005/01/2006 Media Scaling (cont.)  Quality Scaling –MPEG uses quantization in coding to save bits –Quantization Value (1~31) –For example: original data = 23, 13, 7, 3  ARMOR models both Temporal Scaling and Quality Scaling Quantization Value After Quantization After DeQuantization 37, 4, 2, 121, 12, 6, 3 63, 2, 1, 018, 12, 6, 0 121, 1, 0, 012, 12, 0, 0

Ph.D. Defense1105/01/2006 Video Quality Measurements  Subjective Measurement –User study, expensive, not practical  Objective Measurements –Playable Frame Rate (R) Good for Temporal Scaling, not for Quality Scaling –Peak Signal Noise Ratio (PSNR) Good for Quality Scaling, not for Temporal Scaling –Video Quality Metric (VQM) [Pinson+ 04] By Institute for Telecommunication science Extracts 7 perception-based features –Only one for frame losses Report a distortion value from 0 (no distortion) to 1 (many)  ARMOR uses both R and VQM  A comprehensive user study is included

Ph.D. Defense1205/01/2006 Outline  Introduction  Background  Models  –Streaming Bitrate Model (cost) –Video Quality Model (benefit)  Algorithms  User Study  Implementation  Contributions  Conclusions

Ph.D. Defense1305/01/2006 Parameters and Variables Video Frames Repair by Forward Error Correction (FEC)

Ph.D. Defense1405/01/2006 Streaming Bitrate Model  Total streaming bitrate, including video packets and FEC packets: where G is the constant GOP rate N PD and N BD are the numbers of transmitting P and B frames depending on Temporal Scaling level l TS

Ph.D. Defense1505/01/2006  Two distortion factors –Frame Loss Caused by Temporal Scaling and network packet loss Appears jerky in the video playout Measured by Playable Frame Rate –Quantization Distortion Caused by a high quantization value with Quality Scaling Appears visually as coarse granularity in every frame Measured by VQM  Overall Quality –Distorted Playable Frame Rate Video Quality Model - Overview [Wu+ 05 TOMCCAP]

Ph.D. Defense1605/01/2006 Playable Frame Rate (R)  Frame Successful Transmission Probability –Where Frame Size  Frame Dependencies  Total Playable Frame Rate

Ph.D. Defense1705/01/2006  Quality scaling distortion varies exponentially with the quantization level  Distorted Playable Frame Rate Distorted Playable Frame Rate (R D ) [Frossard+ 01]

Ph.D. Defense1805/01/2006 ARMOR Algorithm  For each Repair and Scaling combination Estimate video frame sizes (S I, S P, S B ) –Compute streaming bitrate B and make sure it’s under capacity constraint T –Use frame sizes and FEC amount to get successfully frame transmission rate (q I, q P, q B ) Compute playable frame rate (R) Estimate quality scaling distortion (D) –Compute distorted playable frame rate (R D )  Exhaustively search all FEC and Scaling combination and look for the optimal quality

Ph.D. Defense1905/01/2006 Outline  Introduction  Background  Models  Algorithms  User Study   Implementation  Contributions  Conclusions

Ph.D. Defense2005/01/2006 User Study Goals  Accuracy of R D –Correlation with user perceptual quality –Versus PSNR and VQM?  Temporal Scaling versus Quality Scaling –What are the differences?  Adjusted Repair (FEC) versus No Repair –Is Adjusted Repair an effective method for increasing perceptual quality?

Ph.D. Defense2105/01/2006 Video Clips  Compare degraded clips to the original  Original: 30 fps, no quality scaling  Degraded: Combinations of 4 independent factors (2 options each) –Video and Network environment 1.Video content: low motion (News) or high motion (Coastguard) 2.Packet loss rate: low loss (1%) or high loss (4%) –ARMOR Layer 3.Repair: adjusted repair or no repair 4.Scaling: Quality Scaling or Temporal Scaling  2 4 =16 combinations for evaluation

Ph.D. Defense2205/01/2006 User Study Application  Two-week volunteer study  74 users, most CS undergraduate students [ITU-R BT ]

Ph.D. Defense2305/01/2006 Results – Video Quality Metrics (1) User Score versus PSNR Same as original clip Much worse than original clip

Ph.D. Defense2405/01/2006 Results – Video Quality Metrics (2) User Score versus VQM Score (1 – VQM distortion)

Ph.D. Defense2505/01/2006 Results – Video Quality Metrics (3) User Score versus Distorted Playable Frame Rate (R D )

Ph.D. Defense2605/01/2006 Results – Scaling Methods Temporal Scaling versus Quality Scaling User Score ARMOR Prediction (Coastguard) R D

Ph.D. Defense2705/01/2006 Results – Repair Methods Adjusted Repair versus No Repair User ScoreARMOR Prediction (Coastguard) R D

Ph.D. Defense2805/01/2006 Outline  Introduction  Background  Models  Algorithms  User Study  Implementation   Contributions  Conclusions

Ph.D. Defense2905/01/2006 Architecture

Ph.D. Defense3005/01/2006 Experiment Settings Network (NistNet) SettingsMPEG Encoder Settings t RTT 50 msNPNP 3 frames per GOP S1 KbyteNBNB 8 frames per GOP p0.01 to 0.04RFRF 30 frames per sec  Video clip Paris –medium motion and details –two people sitting, talking, with high-motion gestures –1200 CIF (352x288) images –average I / P / B frame sizes: 24.24KB / 5.20 KB / 1.18 KB

Ph.D. Defense3105/01/2006 ARMOR Analytical Results RDRD Results ARMOR Measurement Results RDRD

Ph.D. Defense3205/01/2006 Contributions  Derived a novel video quality metric –Distorted playable frame rate  Family of Video Quality Models with Repair and Scaling –Modeled the playable frames rate –Modeled quantization distortion –Studied four ARMOR variants: Media Independent FEC with Temporal Scaling Media Independent FEC with Quality Scaling Media Independent FEC with Temporal Scaling and Quality Scaling Media Dependent FEC with Quality Scaling  Derived optimization algorithm to maximize the quality of streaming video  Conducted a comprehensive user study –Presented the high correlation between user score and distorted playable frame rate  Implemented a working ARMOR system

Ph.D. Defense3305/01/2006 Conclusions  Distorted playable frame rate has a high correlation with user perceptual quality –Higher than PSNR or VQM  Adjusting repair improves video streaming quality significantly –Better than fixed repair and no repair  Quality Scaling is more effective than Temporal Scaling –But when bandwidth is low and network loss is high, Quality Scaling should be used with Temporal Scaling  Media Dependent FEC is not as effective as Media Independent FEC  ARMOR can be implemented in a real video streaming system and effectively improve streaming quality

34 ARMOR-Adjusting Repair and Media Scaling with Operations Research for Streaming Video PhD Candidate: Huahui Wu - Computer Science, Worcester Poly. Inst. Committee: Prof. Mark Claypool - Computer Science, Worcester Poly. Inst. Prof. Robert Kinicki - Computer Science, Worcester Poly. Inst. Prof. Craig Wills - Computer Science, Worcester Poly. Inst. Prof. Wu-chi Feng – Computer Science, Portland Stat Univ. Questions?

Ph.D. Defense3505/01/2006 Future Work  Study of Variance of Playable Frame Rate  Study of dynamic Group of Pictures  Study of different quantization values for different types of frames  Implementation of MIQS and MITQS systems  Study of other scaling methods  User study of more videos

Ph.D. Defense3605/01/2006 Playable Frame Rate [S4]  Playable Frame Rate (PFR) of I frames

Ph.D. Defense3705/01/2006 Playable Frame Rate [S4] (cont.)  PFR of P frames

Ph.D. Defense3805/01/2006 Playable Frame Rate [S4] (cont.)  PFR of B frames

Ph.D. Defense3905/01/2006 Capacity Constraint  TCP-Friendly Flow [Padhye+ 00]  Bottleneck Capacity –Dial up: 56 Kbps –DSL: 1.5 Mbps (Verizon) –Cable Modem: 3 Mbps/384 Kbps (Charter) –Video is often larger than 1.5 Mbps

Ph.D. Defense4005/01/2006 Results – Video Quality Metrics (2) User Score versus Playable Frame Rate (R)

Ph.D. Defense4105/01/2006 Lines of Codes

Ph.D. Defense4205/01/2006 Related Work  DAVE (Delivery of Adaptive Video) –Describes video content –Supports physical and semantic adaptation –Does not consider capacity constraint and media repair  Priority Drop –Implemented SPEG for media scaling –Uses TCP as transmission protocol

Ph.D. Defense4305/01/2006 Media Scaling (cont.)  Quality Scaling (QS) –Adaptive Quantization Level  24KB, 10KB, 5KB

Ph.D. Defense4405/01/2006 System LayersParameters MPEG ARMOR Network System Layers and Parameters