1. Computational Controlled Mode Selection for H.264/AVC June 2006 1 Computational Controlled Mode Selection for H.264/AVC Ariel Kit & Amir Nusboim Supervised by: Dr. Ofer Hadar Mr. Evegeny Kaminsky Department of Communication Systems Engineering Ben Gurion University of the Negev Be’er-Sheva, Israel

2. Computational Controlled Mode Selection for H.264/AVC June 2006 2 Presentation Content H.264 Advanced Video Coding OverviewH.264 Advanced Video Coding Overview Project objectivesProject objectives Introduction to mode selectionIntroduction to mode selection Method 1: Cost map estimationMethod 1: Cost map estimation Method 2: Fast Mode SelectionMethod 2: Fast Mode Selection The FMS AlgorithmThe FMS Algorithm ResultsResults Future workFuture work

3. Computational Controlled Mode Selection for H.264/AVC June 2006 3 H.264/AVC Overview Written by the JVT (Joint Video Team) – A Joint effort of the ITU-T and ISO. Previous standards: MPEG1 ISO, MPEG2 JVT, H.261 ITU, MPEG4 ISO, H.263 ITU. Substantially lower bit rates (half of less). Project’s related Features: MM ulti-picture motion compensation using up to 32 reference pictures (frames). VV ariable block-size motion compensation with block sizes from 16x16 to 4x4. WW eigh-scaled predictors based on motion vectors are used to enhance compression. QQ uarter-pixel precision for motion prediction and compensation.

4. Computational Controlled Mode Selection for H.264/AVC June 2006 4 H.264/AVC Overview (cont.)

5. Computational Controlled Mode Selection for H.264/AVC June 2006 5 Motivation The performance gain of H.264 comes at a price of increased computational complexity. Therefore there is a need to develop a low complexity implementation of H.264 that can offer performance and flexibility without excessive computational cost. “Time is Money!”

6. Computational Controlled Mode Selection for H.264/AVC June 2006 6 Project’s Goals Implementing a fast mode selection algorithm that is based on different macroblock’s properties. The algorithm is computationally controlled for usage in limited resources application such as cellular phones and other portable video equipment.

7. Computational Controlled Mode Selection for H.264/AVC June 2006 7 Motion Estimation/Compensation Motion Estimation/ Compensation Intra Prediction Memory (Reference Frames) Lossless Compression Deblocking Filter QT Q -1 T -1 + - + + Video Input H.264 Encoder 10010101…

8. Computational Controlled Mode Selection for H.264/AVC June 2006 8 Mode Selection Motion Estimation/ Compensation In the mode selection process the encoder must decide which mode is best used for a specific macroblock (16x16 pixels). Decision making is critical for the inter prediction, if a macroblock is involved in a precise movement of various objects, we need to split up the macroblock according to the movement of the object to get an accurate motion compensation based on motion vectors.

9. Computational Controlled Mode Selection for H.264/AVC June 2006 9 Mode Selection (cont.) Mode selection is based on RDO (Rate Distortion Optimization) of the cost function per each mode:  Distortion – difference between reconstructed macroblock to the original macroblock.  λ – Lagrange multiplier  Rate – Bits used to encode the macroblock. Best Mode=Smallest Cost J=Distortion + MODE x Rate

10. Computational Controlled Mode Selection for H.264/AVC June 2006 10 Method 1: Cost Map Estimation Estimating cost using motion vectors and predictors to the reference frame. Cost information for a reference frame is saved as a cost map (table) with quarter pixel resolution. Cost is estimated using RDO and Best Cost is saved to the map. The algorithm was simulated using Matlab; based on information received from the Joint Model Reference software (C code) used to simulate the H.264 standard encoder\decoder. JM v10.2 Matlab 7

11. Computational Controlled Mode Selection for H.264/AVC June 2006 11 Method 1: Algorithm Steps Step 0: Create a cost Map of the reference image based on RDO chosen modes (quarter pixel resolution). Step 1: Calculate motion vectors for the current mode. Step 2: Use calculated motion vectors to find macroblock’s cost in the reference Cost Map. Step 3: If cost is larger than the previous Cost go to Step 5. Step 4: Repeat step 2 for all modes (different partitions). Step 5: Choose Best mode with the lowest cost. 8976

12. Computational Controlled Mode Selection for H.264/AVC June 2006 12 Method 1: Results Foreman, QP=28News, QP=28Hall Monitor, QP=28 Foreman, QP=0 Improvement RateTest sequence 2%Foreman 3%Foreman (QP=0) 1%Hall Monitor 1.5%News Improvement mean - Cost Estimation vs. skip prediction

13. Computational Controlled Mode Selection for H.264/AVC June 2006 13 Method 1: Conclusions Results are compared to skip prediction which is the case where cost is predicted using motion vectors predicators using macroblock’s neighbors and motion copy. Improvement is achieved only for mode 16x16 prediction. The improvement is much smaller than expected due to the quarter resolution averaging made in the Joint Model software. The method is not applicable for fast mode selection in the current constellation.

14. Computational Controlled Mode Selection for H.264/AVC June 2006 14 Method 2: Fast Mode Selection Method uses macroblock’s spatial and temporal properties in order to reduce computational complexity.  Temporal Stationarity – macroblocks with zero motion and small differences from the reference measured by threshold.  Spatial Homogeneity – homogenous macroblocks doesn’t need to be sub-partitioned; homogenous is measured by applying a threshold to the macroblock Sobel edge map.  Based on Reference: K. P. Lim, S. Wu, D. J. Wu, S. Rahardja, X. Lin, F. Pan, Z. G. Li, “Fast INTER Mode Selection,” ITU-T, Doc. I020, Sep. [2003].

15. Computational Controlled Mode Selection for H.264/AVC June 2006 15 Method 2: Temporal Stationarity Frame i-1 Frame i Image Difference

16. Computational Controlled Mode Selection for H.264/AVC June 2006 16 Method 2: Spatial Homogeneity 1-01 2-02 1-01 01 2-02 1-01 01 2-02 1-01 01 2-02 1-01 121 000 2-1- 121 000 2-1- 121 000 2-1- 121 000 2-1- >Th Horizontal Sobel Vertical Sobel Edge Map Absolute Value Absolute Value Sobel Masks are used for edge detection

17. Computational Controlled Mode Selection for H.264/AVC June 2006 17 Method 2: Algorithm Steps Edge Detection Zero Motion? MB Difference Similar? RDO (16x16, 16x8/8x16) 16x16 Homogenous? RDO (16x16,16x8,8x16) 8x8 Homogenous? RDO (8x8) RDO (8x4,4x8,8x8) RDO (16x16) Yes No Yes No After RDO, mode with the lowest cost Is chosen as the best mode. Homogeneity and Similarity are decided by using statistical thresholds. Skip mode is always checked.

18. Computational Controlled Mode Selection for H.264/AVC June 2006 18 Method 2: Experiments The algorithm was applied into the JM v10.2 reference software (Code was written in Microsoft Visual C v6). 4 test sequences were used: * Sequences Length – 150 Frames * Video Rate – 30 Frames per second

19. Computational Controlled Mode Selection for H.264/AVC June 2006 19 Method 2: Results Test Sequence Complexity Reduction (%) PSNR Luma Attenuation (dB) Bit Rate Reduction (%) 48.2640.080.777 47.911-0.0923.0974 53.509033.658 34.4270.049.099 * Configuration: Fast Motion Estimation + Early Skip + 4 Reference Frames MV Search +/- 32, Hadamard Transform, GOP structure: IPPP

20. Computational Controlled Mode Selection for H.264/AVC June 2006 20 Method 2: Results (cont.) Test Sequence Complexity Reduction (%) PSNR Luma Attenuation (dB) Bit Rate Reduction (%) 26.0670.13-2.89 34.5550.11.945 32.6360.07-1.89 33.6260.13-2.12 * Configuration: RDO High Complexity (Regular) MV Search +/- 32, Hadamard Transform, GOP structure: IPPP

21. Computational Controlled Mode Selection for H.264/AVC June 2006 21 Method 2: Results (cont.)

22. Computational Controlled Mode Selection for H.264/AVC June 2006 22 Future Work Adding a Complexity parameter. Case of High Complexity – FMS (Rate Control) Case of Medium Complexity – FMS Case of Low Complexity – FMS advanced stages are skipped when application resources run out. Complexity parameter is calculated from average macroblock encoding time.

23. Computational Controlled Mode Selection for H.264/AVC June 2006 23