1. T RUE -M OTION E STIMATION WITH 3-D R ECURSIVE S EARCH B LOCK M ATCHING Gerard de Haan, Paul W. A. C. Biezen Henk Huijgen Olukayode A. Ojo (Philips Research Laboratories, 5600 JA Eindhoven, the Netherlands.) This paper appears in: Circuits and Systems for Video Technology, IEEE Transactions on Page 368–379.388,Oct 1993 1

2. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 2

3. I NTRODUCTION What is true motion? Why do we find the true motion? Consumer display scan rate conversion[1]-[8]. Common drawback is decreased dynamic resolution. Motion compensation techniques[9]-[12] are too expensive for consumer television applications. 3

4. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 4

5. 1-D Recursive Search: similar to 2-D logarithmic search[22] The candidate set (CS i ) & prediction vector (D i-1 ): Indicate with S rather than D i-1 as the spatial prediction vector (pel-recursive algo. [23][24] ): R ECURSIVE S EARCH M ETHOD FOR T RUE ME(1/5) 5

6. 2-D Recursive Search: two spatial prediction vectors A 1-D recursive algorithm cannot cope with discontinuities in the velocity plane. Assumption (1): The discontinuities in the velocity plane are spaced at a distance that enables convergence of the recursive block matcher in between two discontinuities. Two estimators and the selection criterion: As described in 1-DRS, updating, respectively, prediction vectors: R ECURSIVE S EARCH M ETHOD FOR T RUE ME(2/5) 6

7. 2-D Recursive Search solves the run-in problem at the boundaries of moving objects. The best implementation of 2-DC results with predictions from blocks 1 and 3 for estimators a and b, respectively: R ECURSIVE S EARCH M ETHOD FOR T RUE ME(3/5) where (X,Y) is the size of block. 7

8. 3-D Recursive Search: temporal prediction vectors Assumption (2): The displacements between two consecutive velocity planes, due to movements in the picture, are small compared to the block size. Rather than choosing the additional estimators c and d, applying temporal prediction vectors as additional candidates: These convergence accelerators (CA) are taken from a block shifted diagonally over “ r ” blocks. R ECURSIVE S EARCH M ETHOD FOR T RUE ME(4/5) 8

9. 3-D RS candidate set CS a & CS b : The CA's are particularly advantageous at the top of the screen, where the spatial process starts converging. The CA's improve the temporal consistency. R ECURSIVE S EARCH M ETHOD FOR T RUE ME(5/5) 9

10. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 10

11. U PDATING S TRATEGY The asynchronous cyclic search (ACS): N bl is the output of a block counter lut is a look-up table function The pseudorandom look-up table (for p=9): symmetrical distribution around 0 with p updates 11 0 improves the performance for small stationary image parts but disturbs the convergence.

12. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 12

13. F URTHER E MPHASIS ON S MOOTHNESS (1/2) The risks which jeopardize the smoothness: 1) An element of the update sets may equal a multiple of the basic period of the structure. 2) "The other" estimator may not be converged, or may be converged to wrong value that does not correspond to the actual displacement. 3) Directly after a scene change, the convergence accelerators (CAs) yield the threatening candidate. Improve the result for risks 1) & 3): Add penalties to the error function related to the length of the difference vector between the candidates to be evaluated: 13

14. Respectively, 0.4%, 0.8%, and 1.6% of the maximum error value, for the cyclic update(S n ), the convergence accelerator (CA), and the fixed 0 candidate vector. The last candidate(0) especially requires a large penalty. Improve the result for risk 2): The situation occurs if a periodic part enters the picture from the blanking or appears from behind an other object. Advantage of two independent estimators would be lost. F URTHER E MPHASIS ON S MOOTHNESS (2/2) 14

15. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 15

16. B LOCK E ROSION TO E LIMINATE B LOCKING E FFECTS Improve the result for: Eliminating the visible block structures in the picture. Eliminating fixed block boundaries from the vector field without blurring contours. Finally assigned to the pixels in the quadrant: E F 16 H -1-1

17. O VERVIEW Introduction Recursive Search Method for True ME 1-D Recursive Search 2-D Recursive Search 3-D Recursive Search Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments Modified Mean Square Prediction Error(M2SE) Smoothness Conclusion 17

18. E VALUATION R ESULTS & E XPERIMENTS (1/4) Modified Mean Square Prediction Error(M2SE):↓, quality↑ s identifies the test sequence 1~5 P. L is the number of pixels in the image excluding margin. Smoothness Indicator: S(t)↑, smoothness↑ N b is the number of blocks in a field. 18

19. Experiments: E VALUATION R ESULTS & E XPERIMENTS (2/4) 19

20. E VALUATION R ESULTS & E XPERIMENTS (3/4) Captured from: Frame Rate Up-Conversion,陳秉昱,January 8,2006 20

21. E VALUATION R ESULTS & E XPERIMENTS (4/4) Captured from: Frame Rate Up-Conversion,陳秉昱,January 8,2006 21

22. C ONCLUSION The newly designed motion estimation algorithm is emerging as the most attractive of the tested block- matching algorithms in the application of consumer field rate conversion. The bidirectional convergence principle enabled combination of the conflicting demands for smoothness and yet steep edges in the velocity field. Using new test criteria, the suitability of motion estimators for television with motion compensated field rate doubling was tested. 22