Li Liu, Robert Cohen, Huifang Sun, Anthony Vetro, Xinhua Zhuang BMSB

Introduction Existing New Techniques Weighted Prediction (WP) Localized Weighted Prediction for Video Coding Second Order Prediction on H.264/AVC Proposed Second-Order Prediction for Inter Coding Proposed Reduced Resolution Update for Intra Coding Experimental Results Conclusion 2

Increasing popularity of high definition TV, video delivery on mobile devices, and other multimedia applications creates new demands for video coding standards. Both MPEG and VCEG launched their next-generation video coding project, which potentially could be either an extension of H.264/AVC or a brand new standard. In January 2010, MPEG and VCEG have established a Joint Collaborative Team on Video Coding (JCT-VC) to develop the proposed High Efficiency Video Coding (HEVC) standard. 3

To provide a software platform to gather and evaluate these new techniques, a Key Technique Area (KTA) platform was developed based on JM11. Intra Prediction : BIP, MDDT Inter prediction : increasing resolution to 1/8-pel, MB size to 64x64 Quantization : RDOQ, AQMS Transform : 16x16 transform In-loop Filter : QALF, BALF Internal bit-depth increase : 12 bits of internal bit depth for 8-bit source 4

5

6 [1] Peng Yin, Alexis Michael Tourapis, Jill Boyce, “Localized Weighted Prediction for Video Coding,” IEEE Circuits and Systems, 2005.

7

8

Implement: Step1: decide if LWP should be used for current slice. If not, perform normal coding as H.264 does, otherwise, go to step2. Step2 : for each MB, first calculate the mean of the reconstructed neighboring pixel of the current MB. Then perform ME and mode decision using LWP. 9

The predicted blocks generated by MCP will result in low coding efficiency when the video containing complex movements such as shape transforming, rotation or fading. Weighed prediction in H.264/AVC is presented to deal with the fading sequences with global illumination change between frames. Utilizes only temporal correlation but no spatial correlation. Can’t handle motion like shape transforming and rotation. This paper proposes a Second Order Prediction (SOP) to exploit remaining signal correlation after MCP. 10 [2] Shangwen Li, Sijia Chen, Jianpeng Wang and Lu Yu, “Second Order Prediction on H.264/AVC,” Picture Coding Symposium, 2009.

Slight rotation Visible residual Residual exhibit high spatial correlation ※ All-black blocks indicate the MBs applying P-skip mode in the bit-stream 11

Residual Subjective-Textured MBs (RST MBs) : MBs with relatively large residuals. More than twice 12

Apply intra-prediction of H.264/AVC to residuals of inter- prediction. The reconstructed pixel values of an SOP MB are derived as follow : Reconstructed pixel-value = Motion-compensated prediction (first-predictor) + Prediction of first order residuals (second-predictor) + Second order residuals (need to be coded) It seems straightforward to use the previously reconstructed first-order residuals of the neighboring blocks as reference for the current block. 13

The discontinuity caused by motion between blocks will prohibit the efficient utilization of the remaining correlation of the first- order residuals. 14

15

SOP may take 4x4 or 8x8 block as its second prediction unit. Nine 4x4 intra prediction modes of AVC Baseline profile Nine 8x8 intra prediction modes of AVC High profile Transform of the second prediction residuals takes the same block size as the second prediction. The block size might be chosen adaptively based on rate-distortion criterion. 16

Coding of the additional side information of SOP Indicator of SOP An SOP flag to indicate the usage of SOP at MB level. Mode indicator of the second prediction mode A second prediction mode is calculated for each MB, and the coding procedure is the same as that of 4x4 or 8x8 intra-prediction modes encoding in H.264/AVC. The decision of whether an MB will be coded in SOP mode follow the rate-distortion criterion. 17

Environment Low-delay IPPP encoding on H.264/AVC JM10.1 Baseline and Benchmarking with its P-picture coding Stable improvement at most 0.41dB gain 18

It is not efficient for blocks whose size is smaller than 8x8, as too much side information needs to be coded. Partition size larger than 8x8 will be divided into multiple 8x8 sub-blocks, each with its own second-order prediction mode. 19

Reduced resolution update (RRU) is a technique that aims to save coding bits by resize image/prediction residuals to a reduced spatial resolution. At low bit rates, it’s known that down-sampling an image to a low resolution, then compressing the lower resolution, and interpolating the result to the original resolution can improve the overall PSNR. 20

JPEG Blocks of 8x8 pixels Allocate too few bits (4 bits per block on average) Only DC coefficients are coded Blocking artifacts Reduced resolution 21

22

Modified the framework of H.264/AVC so that residual after intra prediction can be optionally down-sampled before the transform and quantization steps. For instance : a 16x16 block can be down-sampled by a factor of 2 so that only an 8x8 block needs to be encoded Decoder shall up-sample the down-sampled residual to reconstruct full resolution picture. The choice of RRU should be considered under RDO. 23

Second-Order Prediction for Inter Coding Gains are not significant The optimal motion vector position may be different from first-order motion vector. Perform motion vector search for each individual second-order prediction mode. Increase computational complexity. 24 Environment H.264/AVC JM15.1 Compare with original H.264/AVC inter coding First frame : I picture Remaining : P pictures Only 4x4 DCT is allowed QP : 23,28,33,38

Proposed Reduced Resolution Update for Intra Coding 25 Environment H.264/AVC JM15.1 Compare with original H.264/AVC intra coding 16x16 blocks Down-sampling : 5-tap filter [ ]/8 Up-sampling : 7-tap filter [ ]/16 All frames are I frames QP : 23,28,33,38 Each sequence is coded using RRU

RRU improves the coding efficiency for medium content complexity. H.264/AVC is efficient for flat areas. RRU may bright too much loss for areas with high frequency content. 26

RRU works well for 16x16 blocks, the contribution of RRU to overall intra coding shall depend on the percentage 16x16 block size is used over 4x4 and 8x8 modes. 27

Both the new techniques listed and our experiments on second- order prediction and RRU prove that there is still room for performance improvement of current coding standard. The Call for Evidence for HVC provided results that averaged a 15-25% gain in coding efficiency. 28