1. Yung-Lin Huang, Yi-Nung Liu, and Shao-Yi Chien Media IC and System Lab Graduate Institute of Networking and Multimedia National Taiwan University Signal Processing Systems (SIPS), 2010 IEEE

2. Outline  Introduction Markov Random Field  Motion Vector Analysis  Motion Vector Pre-processing  Predictor Selection  Simplified Belief Propagation  Experimental Results  Conclusion

3. Introduction (1/4)  Instead of heuristic approaches, TME can be formulated as a pixel-labeling problem:

4. Introduction (2/4)  Markov Random Field : Given an undirected graph G = (V, E) A set of random variables(label) X = (X v ) v ∈ V  Markov properties:

5. Introduction (3/4)  Assigning each pixel a label, can be justified in terms of maximum a-posterior estimation of a MRF model: posterior ∝ likelihood * prior  With negative log probabilities, where the max-product becomes a min-sum.  The max-product algorithm can be used to find an approximate minimum cost labeling of energy functions. E d (the data term) & E s (the smoothness term)

6. Introduction (4/4)  The cost energy function of a Markov Random Field model to estimate the optimal labels { l p } of corresponding pixels : E d : the data term that measures the penalty between the labels and the data E s : the smoothness term that penalizes the coherence between labels P : the set of all pixels N : the 4-nearest neighbor pixels

7. Motion Vector Analysis(1/3)  Optical flow datasets are used here because the ground truth (GT) MV maps are provided:

8.  Check the existence of true MV by similarity:  The existence of TMV: W,H: the width, height of the test sequence  In Fig. 4. Both TH x and TH y are set to 1, and PSR ranges from 0 to 64. Motion Vector Analysis(2/3)

9.  The ME strategy(FastFS, FS or EPZS) has little effect on the experimental results.  There are still MVs with true motion trajectory in the H.264-coded MV field. Motion Vector Analysis(3/3)

10. Proposed Algorithm

11. Motion Vector Pre-processing  In the proposed algorithm, the block size is fixed in each scale, so the MVs of variable block sizes must be split and merged.  The block merging method takes not only the macroblock types (from H.264) but also neighboring MVs into consideration.  Although the global optimization might modify these bad MVs, the pre-processing costs less efforts.

12. Proposed Algorithm

13. Predictor Selection(1/2)  In Fig. 4, the probability that true MV exists is high with enough PSR.Fig. 4  We choose PSR=32, when the block size is 16, the range of ±32 pixels ±2 blocks.  The strategy of predictor selection and the MRF model of the proposed algorithm are shown in Fig. 5(a). Nine predictors are selected.

14. Predictor Selection(2/2)

15. Proposed Algorithm

16. Simplified Belief Propagation (1/3)  The multi-scale concept from [7],instead of pixel-based operation, 4x4 block is taken as the smallest unit.  The belief propagation is operated from the highest scale (16x16 block) to the lowest scale (4x4 block).  f t : video frame at time t

17.  The basic concept of belief propagation is to perform message passing operation iteratively and approximate global minimum by local messages.  Each pixel requires O(k 2 ) computation for full-search candidates.  The proposed algorithm requires only O(k) computation with predictor selection [7] for each pixel. Simplified Belief Propagation (2/3)

18. Simplified Belief Propagation (3/3) [7] Pedro F. Felzenszwalb and Daniel P. Huttenlocher, “Efficient belief propagation for early vision,” Int. J. Comput.Vision, vol. 70, no. 1, pp. 41–54, 2006.  Loopy Belief Propagation approach for MRF:  Messages with the truncated linear model: Time complexity: O(nk 2 T) n: the number of pixels k: the number of possible labels T: the number of iterations Time complexity: O(k) n: the number of pixels k: the number of possible labels T: the number of iterations

19. Experimental Results  Frame Rate Up Conversion compare with: Bidirectional overlapped block motion estimation (OBME), MV field smoothing with median filter.  Proposed algorithm has higher PSNR about the camera motion video (mobile calendar) because of the global MV field optimization.  OBME requires full search(FS) with an enlarged search range.  The proposed algorithm has relative lower computational complexity.

20.  Motion Vector Field: Experimental Results

21.  Motion Vector Field: Experimental Results

22. Conclusion  In this paper, a MRF-based true motion estimation obtained from H.264/AVC scheme is proposed.  The MV field of H.264/AVC is optimized using belief propagation efficiently.  In the future works, more reusable decoding information and hardware implementation will be involved.