Adaptive Content-Aware Scaling for Improved Video Streaming. Avanish Tripathi Mark Claypool Presented by: Huahui Wu Worcester Polytechnic Institute

Motivation Internet is more useful with Multimedia TCP is the de facto standard… but  It is not ideal for multimedia UDP can send with constant bit rate  It doesn’t have congestion control Other network protocols e.g. TFRC, RAP  Ideal-rate based with smooth rates  But the sender treats all the data same.

Motivation Random packet drop by routers during congestion decreases quality Application know best data to send for a given rate Need application level solution…

…Media Scaling Need media scaling: Application level data- rate reduction Scaling types: Temporal scaling Quality scaling Best scaling type depends on motion “Content of the stream should influence the choice of scaling mechanism” To the best of our knowledge this idea has not yet been completely evaluated

Methodology: Content-Aware Scaling Develop and verify motion measurement mechanism Define temporal and quality scaling levels Evaluate the potential impact of content-aware scaling (User Study 1) Build system to do content-aware scaling adaptively Evaluate the practical impact of the full system (User Study 2)

Motion Measurement MPEG has three kinds of frames I- Intra encoding P- Predictive encoding B- Bi-directional predictive encoding Subdivided into Macroblocks Intra macroblocks Predictive macroblocks Interpolated macroblocks More when adjacent frames are similar

Motion Measurement Our hypothesis High percent of interpolated macroblocks means low motion Low percent of interpolated macroblocks means high motion Conducted a pilot study to test our hypothesis 18 Video clips, one scene each (5~10s) Visually divide frame into 16 sub-blocks Manually count number of blocks that have motion Correlate with percent of Interpolated macroblocks.

Pilot Study Result: Motion Measurement

Methodology: Content-Aware Scaling Develop and verify motion measurement mechanism Define temporal and quality scaling levels Evaluate the potential impact of content- aware scaling (User Study 1) Build system to do content-aware scaling adaptively Evaluate the practical impact of the full system (User Study 2)

Filtering Extend a system developed at Lancaster university  Frame dropping filter (Temporal Scaling)  Requantization filter (Quality Scaling) Scale level table 1 TypeLevelMethodFpsBwidth(%) NoneN/A Temporal1No B1370 Temporal2No P or B511 Quality1Q = Quality2Q =

Methodology: Content-Aware Scaling Develop and verify motion measurement mechanism Define temporal and quality scaling levels Evaluate the potential impact of content- aware scaling (User Study 1) Build system to do content-aware scaling adaptively Evaluate the practical impact of the full system (User Study 2)

User Study 1 22 graduate and undergraduate students in the department Platform: 3 Pentium III machines with 128MB RAM running Linux Clips were on local hard drives Eighteen clips, 10 second each Some high motion, some low motion Five versions of each clip: Perfect, Temporal Level 1, Temporal Level 2, Quality Level 1, Quality Level 2

User Study 1 Users rated the clips with numbers from

Results Four men sitting at a bar  Low Motion ( 70% interpolated macroblocks)

Results A man on horseback tries to rope a bull  High Motion (27% interpolated macroblocks)

Results Temporal scaling is better for low motion Since the frame changes a little, not all the frames are important. Quality scaling is better for high motion Since the frame changes frequently, user need every frame to catch the motion. Will a video keep low motion or high motion? No!

Methodology: Content-Aware Scaling Develop and verify motion measurement mechanism Define temporal and quality scaling levels Evaluate the potential impact of content- aware scaling (User Study 1) Build system to do adaptive content-aware scaling Evaluate the practical impact of the full system (User Study 2)

Full System Architecture MPEG Server Input Motion Measurement High Low Temporal Filter Quality Filter Internet Feedback Generator Client

System Functionality Server is capable of classifying motion as the movie plays The system is adaptive and scales movies in real-time depending on the loss pattern as received from the feedback module Two more scale levels for finer granularity

Methodology: Content-Aware Scaling Develop and verify motion measurement mechanism Define temporal and quality scaling levels Evaluate the potential impact of content- aware scaling (User Study 1) Build system to do content-aware scaling adaptively Evaluate the practical impact of the full system (User Study 2)

User Study 2 Four clips (2 or more scenes) ~30 seconds Four versions of each video clip Perfect Quality Temporal scaling only Quality scaling only Adaptive scaling Bandwidth distribution functions: how often the rate changes Every 3 seconds (slow) Every 200ms(frequently)

Result Talk show followed by a car commercial Low motion to High motion Bandwidth changes every 3 seconds

Conclusions Build application level solution to the problem of congestion due to unresponsive video streams Developed a mechanism to classify the motion in a video stream Shown that content aware scaling can improve user perceived quality by as much as 30% Developed a system to do adaptive content- aware scaling and proved it with user study

Future Work More accurate bandwidth distribution function to test our system in the real world. More scaling level to get smoother data rate. Hybrid scaling methods (Quality + Temporal) Audio Scaling

Adaptive Content-Aware Scaling for Improved Video Streaming. Avanish Tripathi Mark Claypool Presented by: Huahui Wu Worcester Polytechnic Institute