Byung Cheol Song Shin-Cheol Jeong Yanglim Choi Video Super-Resolution Algorithm Using Bi-directional Overlapped Block Motion Compensation IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 3, MARCH

 Introduction  Motion-compensated SR  Basic concept  Hierarchical motion estimation  Bi-directional OBMC  Result  Learning-based SR  Hybrid super-resolution  Experimental Results O UTLINE 2

 For the last few decades, many image interpolation algorithms have been developed to display high quality scaled images on cutting-edge digital consumer application such as HDTV,DSC.  Traditional interpolation methods usually suffer from several types of visual degradation.  To overcome the problem, many algorithms are proposed. But there are many problem.(Ex. structurally weak against textures, computational complexity) I NTRODUCTION 3

 Brandi et al. presented an interesting SR approach.  They defined the so-called key frames (KF) that sparsely exist in a video sequence and have HR resolution. The remaining frames in the video sequence, i.e., non-key frames(NKF) had LR resolution  Brandi et al. took advantage of the fact that few KF (encoded at HR) may provide enough HF information to the up-scale NKF (encoded at LR)  However, it rarely found true motion information because it made use of the conventional full-search motion estimation.  This paper presents a hybrid SR algorithm where each LR patch is adaptively selected between a temporally super-resolved patch and a spatially super-resolved patch using adjacent HR KFs.  Temporally super-resolved patch  MSR  Spatially super-resolved patch  LSR I NTRODUCTION 4

 A target frame in LR video sequence is interpolated by using forward and backward HR key-frame(KF). M OTION -C OMPENSATED SR Basic Concept 5 The KF interval N can be constant or variable. These paper employ cubic convolution for Up-Scaling.

 First, the MV for overlapping M×M matching block is searched by using a rate-constrained ME.  The conventional rate-constrained ME find the best v for each matching block by minimizing the rate as well as distortion  In order to maximize the ME performance and to concurrently reduce the computational burden, we adopt a rate-constrained fast full search algorithm presented in [15]. M OTION -C OMPENSATED SR Hierarchical Motion Estimation 6 λ : Lagrange multipier

 Assume that MVs between up-scaled LR frames are statistically very similar to those between corresponding HR frame.  Replace the unknown MVs for HR frames with the MVs obtains from the up-scaled LR frames.  Employ BOBMC based on bi-directional MVs.the BOBMC is performed on a 4×4 block basis.  Each MV may be a forward or backward MV. The direction is determined according to SAD M OTION -C OMPENSATED SR Bi-directional OBMC 7

M OTION -C OMPENSATED SR Result originalMSR Fan’s [12] Brandi’s [14] 8

 It cannot often provide acceptable visual quality due to non- translational motion, occlusion, inaccurate motion estimation and limited motion search range.  When such temporal motion compensation does not work well, we employ a learning-based SR in order to avoid the degradation of visual quality. M OTION -C OMPENSATED SR Drawback 9

L EARNING -B ASED SR 10 K-means clustering LMS algorithm

C OMPARE MSR & LSR MSRLSR 11 Background is well- motion-compensated Seldom be well- motion-compensated Better Quality

H YBRID S UPER -R ESOLUTION 12 The boundary between the temporally super-resolved patches and spatially super- resolved patches, we can observe blocking artifact because they are derived from different frames. We apply a simple smoothing filter to the boundary pixels

 PSNR performance of the MSR according to N E XPERIMENTAL R ESULTS 13 Without/with little global motion  N seldom affects With large global motion

E XPERIMENTAL R ESULTS 14 N = 30 M =16 L = 4 Search range =±64 MSRHybrid SR [1] X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Trans. Image Process., vol. 10, no. 10, pp. 1521–1527, Oct [12] W. Fan and D. Y. Yeung, “Image hallucination using neighbor embedding over visual primitive manifolds,” in Proc. CVPR, 2007, pp. 1–7. [14] F. Brandi, R. Queiroz, and D. Mukherjee, “Super-resolution of video using key frames and motion estimation,” in Proc. IEEE ICIP, Oct.2008, pp. 321–324. Fan’s[12] Barsiu’s Brandi’s[14] NEDI[1] BLI Bi-cubic

E XPERIMENTAL R ESULTS 15 N = 30 M =16 L = 4 Search range =±64

E XPERIMENTAL R ESULTS 16 N = 30 M =16 L = 4 Search range =±64