No-Reference Metrics For Video Streaming Applications International Packet Video Workshop (PV 2004) Presented by : Bhavana CPSC 538 February 21, 2004

Video Quality Assessment What’s Quality ? - Implies Comparison => reference - Implies Comparison => reference Three Techniques : - Full-Reference eg. MSE, PSNR - Reduced Reference - Reduced Reference - No Reference

What is a No-Reference metric ? Estimating end-user’s QoE of a multimedia stream without using an original stream as a reference. Estimating end-user’s QoE of a multimedia stream without using an original stream as a reference. In other words : In other words : “ Quantify quality via blind distortion measurement” “ Quantify quality via blind distortion measurement”

Purpose To evaluate two types of distortions in streaming of compressed video over packet-switched networks To evaluate two types of distortions in streaming of compressed video over packet-switched networks - Compression related : block-edge impairment - Transmission related : packet-loss impairment

Where Can It Be Used ? For real time monitoring. Reference unavailable or expensive to send Reference unavailable or expensive to send Feedback to Streaming Server. Feedback to Streaming Server. Evaluation of Compression Algorithms Evaluation of Compression Algorithms

What are Block-Based Codecs ? Process several pixels of video together in blocks Process several pixels of video together in blocks At high compression rates, strong discontinuities called block edges come up. At high compression rates, strong discontinuities called block edges come up. What’s blockiness ? “Distortion of image characterized by appearance of underlying block encoding structure “Distortion of image characterized by appearance of underlying block encoding structure

Block – based Distortion Idea : A block-edge gradient can be masked by a region of high spatial activity around it. Idea : A block-edge gradient can be masked by a region of high spatial activity around it. Measure two things : Measure two things : - spatial activity around block edges : σ - block-edge gradient : Δ

Calculation of NR Blockiness Metric For each 8 x 8 Block B ij, For each 8 x 8 Block B ij, For each edge I k of B ij, For each edge I k of B ij, divide edge into 3 segments a kl For each segment of a kl calculate σ kl calculate Δ kl

I 4 I1I1 E2E2 E4E4 E3E3 I2I2 E1E1 I3I3 B ij a1a1 a2a2 a3a3 An 8 x 8 block and its edges Three segments a kl of a block edge

NR Blockiness Metric contd’ C B = No. of Blocks for which at least one edge satisfies : C B = No. of Blocks for which at least one edge satisfies : σ kl < ε where ε = 0.1 σ kl < ε where ε = 0.1 Δ kl > τ where τ = 2.0 Δ kl > τ where τ = 2.0 ε = min. spatial activity required to mask gradient τ = max. gradient which is imperceivable. τ = max. gradient which is imperceivable. β F = C B / Total no. of blocks in the frame

Simulation Setup For NR Blockiness Metric Aim : to measure how well the NR Blockiness metric conveys QoE Aim : to measure how well the NR Blockiness metric conveys QoE Codec : MPEG -4, GOP = 30 frames Codec : MPEG -4, GOP = 30 frames Bit Rate => compression level

NR Packet Loss Metric Error Concealment : Replace damaged/lost macroblock with corresponding macroblock from previous frame. Error Concealment : Replace damaged/lost macroblock with corresponding macroblock from previous frame. Idea : Use length of artifact to estimate amount of distortion caused by packet loss Idea : Use length of artifact to estimate amount of distortion caused by packet loss

Calculation of NR Packet Loss Metric For a m x n frame For each 16 x 16 macroblock For each 16 x 16 macroblock Calculate : Calculate : Ê j = strength vector across macroblock edge edge Ê΄ j = strength vector within macroblock near the edge near the edge

Macroblock 1 Macroblock 2 Figure : Calculating Strength vector across and within a macroblock

Convert strength vectors into binary vectors Convert strength vectors into binary vectors E j (k) = 1 if Ê j > τ = 0 otherwise = 0 otherwise E ΄ j (k) = 1 if Ê΄ j > τ = 0 otherwise = 0 otherwise where τ = 15

If the sum of differences between the two binary edge vectors is substantial, then there is distortion If the sum of differences between the two binary edge vectors is substantial, then there is distortion Packet loss metric for j th macroblock H j = ∑ | E j (k) - E΄ j (k) | if ∑ | E j (k) - E΄ j (k) | > ζ H j = ∑ | E j (k) - E΄ j (k) | if ∑ | E j (k) - E΄ j (k) | > ζ = 0 otherwise where ζ = 10% of frame width (n) where ζ = 10% of frame width (n) Packet loss metric for whole frame F = ∑ H j 2 F = ∑ H j 2

Simulation Setup for NR Packet Loss Metric Bit Rate = 1.5 Mbps Bit Rate = 1.5 Mbps Frame Rate = 30 fps Frame Rate = 30 fps Frame Size = 352 x 240 Frame Size = 352 x 240 Used NTT DoCoMo packet loss generating software. Used NTT DoCoMo packet loss generating software.

Limitations Of NR-metrics Blockiness metric might fail in the presence of strong de-blocking filters which might otherwise introduce blur Blockiness metric might fail in the presence of strong de-blocking filters which might otherwise introduce blur Metric predictions lose meaning in presence of other distortions like blur, noise etc. Metric predictions lose meaning in presence of other distortions like blur, noise etc.

Future Directions VQEG standardization efforts VQEG standardization efforts HVS based approaches HVS based approaches Statistical models for natural scenes Statistical models for natural scenes NR QA schemes for NR QA schemes for - Non-block based compression schemes such Wavelet-based -Targeting full range of artifacts

References No Reference Image and Video Quality Assessment tm tm tm Objective video Quality Assessment _hvd_bookchapter.pdf _hvd_bookchapter.pdf _hvd_bookchapter.pdf Perceptual Video Quality and Blockiness Metrics for Multimedia Streaming Applications pdf