1 Video Frames Interpolation Using Adaptive Warping Ying Chen Lou Major Advisor: M.J.T. Smith Co-advisor: Edward Delp Nov. 15, 2010.

Slides:



Advertisements
Similar presentations
Packet Video Error Concealment With Auto Regressive Model Yongbing Zhang, Xinguang Xiang, Debin Zhao, Siwe Ma, Student Member, IEEE, and Wen Gao, Fellow,
Advertisements

Wen-Hsiao Peng Chun-Chi Chen
Tae-Shick Wang; Kang-Sun Choi; Hyung-Seok Jang; Morales, A.W.; Sung-Jea Ko; IEEE Transactions on Consumer Electronics, Vol. 56, No. 2, May 2010 ENHANCED.
Byung Cheol Song Shin-Cheol Jeong Yanglim Choi Video Super-Resolution Algorithm Using Bi-directional Overlapped Block Motion Compensation IEEE TRANSACTIONS.
Forward-Backward Correlation for Template-Based Tracking Xiao Wang ECE Dept. Clemson University.
Ai-Mei Huang And Truong Nguyen Image processing, 2006 IEEE international conference on Motion vector processing based on residual energy information for.
1 Adaptive slice-level parallelism for H.264/AVC encoding using pre macroblock mode selection Bongsoo Jung, Byeungwoo Jeon Journal of Visual Communication.
{ Fast Disparity Estimation Using Spatio- temporal Correlation of Disparity Field for Multiview Video Coding Wei Zhu, Xiang Tian, Fan Zhou and Yaowu Chen.
An Improved 3DRS Algorithm for Video De-interlacing Songnan Li, Jianguo Du, Debin Zhao, Qian Huang, Wen Gao in IEEE Proc. Picture Coding Symposium (PCS),
Wavelet-Domain Video Denoising Based on Reliability Measures Vladimir Zlokolica, Aleksandra Piˇzurica and Wilfried Philips Circuits and Systems for Video.
A New Block Based Motion Estimation with True Region Motion Field Jozef Huska & Peter Kulla EUROCON 2007 The International Conference on “Computer as a.
Limin Liu, Member, IEEE Zhen Li, Member, IEEE Edward J. Delp, Fellow, IEEE CSVT 2009.
Ai-Mei Huang and Truong Nguyen Image Processing (ICIP), th IEEE International Conference on 1.
CMPT-884 Jan 18, 2010 Error Concealment Presented by: Cameron Harvey CMPT 820 October
Yen-Lin Lee and Truong Nguyen ECE Dept., UCSD, La Jolla, CA Method and Architecture Design for Motion Compensated Frame Interpolation in High-Definition.
Natan Jacobson, Yen-Lin Lee, Vijay Mahadevan, Nuno Vasconcelos, Truong Q. Nguyen IEEE, ICME 2010.
Reji Mathew and David S. Taubman CSVT  Introduction  Quad-tree representation  Quad-tree motion modeling  Motion vector prediction strategies.
Ai-mei Huang And Truong Nguyen IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS. (WOWMOM), 2008 IEEE, WORLD OF WIRELESS, MOBILE AND MULTIMEDIA NETWORKS.
Recursive End-to-end Distortion Estimation with Model-based Cross-correlation Approximation Hua Yang, Kenneth Rose Signal Compression Lab University of.
1 Static Sprite Generation Prof ︰ David, Lin Student ︰ Jang-Ta, Jiang
Probabilistic video stabilization using Kalman filtering and mosaicking.
1 Single Reference Frame Multiple Current Macroblocks Scheme for Multiple Reference IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY Tung-Chien.
Decision Trees for Error Concealment in Video Decoding Song Cen and Pamela C. Cosman, Senior Member, IEEE IEEE TRANSACTION ON MULTIMEDIA, VOL. 5, NO. 1,
Scalable Wavelet Video Coding Using Aliasing- Reduced Hierarchical Motion Compensation Xuguang Yang, Member, IEEE, and Kannan Ramchandran, Member, IEEE.
Motion Vector Refinement for High-Performance Transcoding Jeongnam Youn, Ming-Ting Sun, Fellow,IEEE, Chia-Wen Lin IEEE TRANSACTIONS ON MULTIMEDIA, MARCH.
Virtual Control of Optical Axis of the 3DTV Camera for Reducing Visual Fatigue in Stereoscopic 3DTV Presenter: Yi Shi & Saul Rodriguez March 26, 2008.
Optical Flow Estimation
Source-Channel Prediction in Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Laboratory ECE Department University of California,
Motion Estimation Using Low- Band-Shift Method for Wavelet- Based Moving Picture Hyun-Wook Park, Senior Member, IEEE, and Hyung-Sun Kim IEEE Transactions.
4/24/2002SCL UCSB1 Optimal End-to-end Distortion Estimation for Drift Management in Scalable Video Coding H. Yang, R. Zhang and K. Rose Signal Compression.
Jump to first page The research report Block matching algorithm Motion compensation Spatial transformation Xiaomei Yu.
09/24/02ICIP20021 Drift Management and Adaptive Bit Rate Allocation in Scalable Video Coding H. Yang, R. Zhang and K. Rose Signal Compression Lab ECE Department.
An Introduction to H.264/AVC and 3D Video Coding.
Video Compression Concepts Nimrod Peleg Update: Dec
January 26, Nick Feamster Development of a Transcoding Algorithm from MPEG to H.263.
Lossy Compression Based on spatial redundancy Measure of spatial redundancy: 2D covariance Cov X (i,j)=  2 e -  (i*i+j*j) Vertical correlation   
Windows Media Video 9 Tarun Bhatia Multimedia Processing Lab University Of Texas at Arlington 11/05/04.
Research & Development 1 High Frame Rate Television Mike Armstrong, Steve Jolly, Richard Salmon BBC R&D HPA Technical Retreat 18 February 2009.
Robust global motion estimation and novel updating strategy for sprite generation IET Image Processing, Mar H.K. Cheung and W.C. Siu The Hong Kong.
Low-Power H.264 Video Compression Architecture for Mobile Communication Student: Tai-Jung Huang Advisor: Jar-Ferr Yang Teacher: Jenn-Jier Lien.
Marc CHAUMONT ICIP 2003 Fully scalable object based video coder based on analysis- synthesis scheme Marc Chaumont, Nathalie Cammas 1 and Stéphane Pateux.
Sub pixel motion estimation for Wyner-Ziv side information generation Subrahmanya M V (Under the guidance of Dr. Rao and Dr.Jin-soo Kim)
Image Enhancement [DVT final project]
MOTION ESTIMATION IMPLEMENTATION IN RECONFIGURABLE PLATFORMS
Guillaume Laroche, Joel Jung, Beatrice Pesquet-Popescu CSVT
Advances in digital image compression techniques Guojun Lu, Computer Communications, Vol. 16, No. 4, Apr, 1993, pp
Ai-Mei Huang, Student Member, IEEE, and Truong Nguyen, Fellow, IEEE.
Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image.
Segmentation of Vehicles in Traffic Video Tun-Yu Chiang Wilson Lau.
Unified Loop Filter for High-performance Video Coding Yu Liu and Yan Huo ICME2010, July 19-23, Singapore.
Block-based coding Multimedia Systems and Standards S2 IF Telkom University.
Flow Control in Compressed Video Communications #2 Multimedia Systems and Standards S2 IF ITTelkom.
COMPARATIVE STUDY OF HEVC and H.264 INTRA FRAME CODING AND JPEG2000 BY Under the Guidance of Harshdeep Brahmasury Jain Dr. K. R. RAO ID MS Electrical.
Sub-Band Coding Multimedia Systems and Standards S2 IF Telkom University.
Representing Moving Images with Layers J. Y. Wang and E. H. Adelson MIT Media Lab.
Stereo Video 1. Temporally Consistent Disparity Maps from Uncalibrated Stereo Videos 2. Real-time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral.
Motion Estimation Multimedia Systems and Standards S2 IF Telkom University.
Principles of Video Compression Dr. S. M. N. Arosha Senanayake, Senior Member/IEEE Associate Professor in Artificial Intelligence Room No: M2.06
An H.264-based Scheme for 2D to 3D Video Conversion Mahsa T. Pourazad Panos Nasiopoulos Rabab K. Ward IEEE Transactions on Consumer Electronics 2009.
Fine-granular Motion Matching for Inter-view Motion Skip Mode in Multi-view Video Coding Haitao Yanh, Yilin Chang, Junyan Huo CSVT.
Fast disparity motion estimation in MVC based on range prediction Xiao Zhong Xu, Yun He ICIP 2008.
Ai-Mei Huang And Truong Nguyen Image processing, 2006 IEEE international conference on Motion vector processing based on residual energy information for.
Multi-Frame Motion Estimation and Mode Decision in H.264 Codec Shauli Rozen Amit Yedidia Supervised by Dr. Shlomo Greenberg Communication Systems Engineering.
Computational Controlled Mode Selection for H.264/AVC June Computational Controlled Mode Selection for H.264/AVC Ariel Kit & Amir Nusboim Supervised.
Conversion of Standard Broadcast Video Signals for HDTV Compatibility
Overview of the Scalable Video Coding
Quad-Tree Motion Modeling with Leaf Merging
Representing Moving Images with Layers
Anisotropic Double Cross Search Algorithm using Multiresolution-Spatio-Temporal Context for Fast Lossy In-Band Motion Estimation Yu Liu and King Ngi Ngan.
Representing Moving Images with Layers
Presentation transcript:

1 Video Frames Interpolation Using Adaptive Warping Ying Chen Lou Major Advisor: M.J.T. Smith Co-advisor: Edward Delp Nov. 15, 2010

2 Outline Background Generic motion model Video spatial interpolation Video compression Video frame rate up-conversion Summary and future work

3 Motivation Spatial interpolation –Conversion from SDTV to HDTV –Zooming of region of interest (ROI) Surveillance/forensics Medical imaging Satellite imaging Temporal interpolation (Frame rate up-conversion) –3:2 pull down 24Hz -> 30Hz –Avoid flicker and blurring on LCD –Loss of frames in transmission Frame Rate Up-Converter

4 Challenges and Goal Core: motion estimation To derive a generic motion model which can be used for different applications –Motion needs to be accurate Ill-posed problem (aperture problem) –Suitable for different types of motion Translational (panning) Zoom in/out Rotational –Low to moderate computational intensive Local window used for ME

5 Illustrative Example Block Matching vs. Optical Flow Motion Block Matching Optical Flow

6 Motion Estimation Method (Warping) I[n 1, n 2, k] = I[n 1 + d 1 [n 1, n 2, k], n 2 + d 2 [n 1, n 2, k], k + δk] OFE where

7 Assumes that the pixel displacement functions within some region R of an image I[n 1,n 2,k] can be written as: and are vectors composed of the displacement parameters to be estimated The bilinear displacement parameter are computed by minimizing the mean squared error function (MSEF) Warping Method (cont’d)

8 Quad-tree Divisions Makes the algorithm adaptive and more efficient Quad-treeUniform

9 Application I Spatial Interpolation Motivation Characteristics of lecture videos –Large static backgrounds –Little to medium motion of foreground Possible to store several high resolution frames and retrieve them later Simplest scenario –periodically transmit one high resolution frame and the remaining frames are in the form of low resolution Extend the proposed method to other types of video

10 Interpolation –Bilinear –Adaptive Synthesis Filter Banks Warping –Block matching-based and optical flow-based –Quadtree splitting Proposed method I: Full-band Warping (FWWA)

11 High Motion Breakdown

12 Robustness Issue Address the robustness issue –Obtain reliable motion vectors –Maintain the sharpness Challenges –No corresponding pixels in the reference frame –Ambiguity of improvement in sharpness and distortion objectively

13 Proposed method II The Composite Algorithm Incorporate advanced spatial interpolation algorithms Bidirectional warping –Forward and backward warping –To solve “no corresponding pixels in the reference frame” Hierarchical motion structure –To solve “ambiguity of improvement in sharpness and geometric distortion ”

14 Hierarchical Motion Structure Z F 3 – forward warped frame Z F 3’ – downsampled forward warped frame Z B 3 – backward warped frame Z B 3’ – downsampled backward warped frame X3 – original MxN low resolution frame

15 Experiment Setup Assessment of the composite algorithm –Compare with bilinear, bicubic, NEDI, VA, and the full-band warping algorithm (FWWA) Investigate the impact of different spatial interpolation methods on the composite warping algorithm –Incorporate different spatial interpolation methods in the first step –The remaining stages are the same 3 sets of video sequences, CIF, 50 frames –Low motion (‘talking head’ lecture video ) –High detail –High motion

16 Assessment of the Composite Algorithm Every 5 th frame as a high resolution reference frame Decimate frames to get low resolution frames –Use a 21-tap lowpass filter –Downsample factor 2 Use original frames as ground truth Competing methods –Bilinear, Bicubic simple image interpolation methods –New-edge Directed Interpolation (NEDI) an advanced image interpolation method –A super resolution method proposed by Patrick Vandewalle (VA) 1 –Full band warping (FWWA) [1] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to superresolution”, EURASIP Journal on Applied Signal Processing, 2006

17 Objective Results AkiyoCarphoneNewsSilenceMother Bilinear Bicubic NEDI VA FWWA Comp SalesBusFlowerTempeteMobile StefanTableFootball

18 Talking Head Video Original frameBilinear interpolation FWWA Comp

19 High Detail Video Original frameBilinear interpolation FWWA Comp

20 High Motion Video Original frameBilinear interpolation FWWA Comp

21 (a) (b) (c) (c) (d) (a) Bilinear (b) NEDI (c) VA (d) Bicubic + Comp

22 Demo

23 Different Spatial Interpolation AkiyoCarphoneNewsSilenceMother Bilinear Bicubic NEDI VA Bi-Comp Cu-Comp NEDI-Comp VA-Comp

24 Conclusion The composite methods achieved good spatial interpolation results Accommodate complex motion Outperform competing methods subjectively and objectively –Improvement comes mostly from the warping process –A combination of bicubic interpolation and warping results in best overall performance Complexity not too high Subjective and objective results are satisfactory Particularly perform well for lecture videos and high detail videos

25 Application II Video Compression H.264/AVC Retain edges but remove texture at low bitrates

26 Goal and Proposed Method Goal –Propose a coding method which keeps the high frequency components –Achieve as high visual quality as possible –Maintain integrating of H.264/AVC coder which is well engineered –Be robust Three assumptions –Smaller resolution requires fewer bits –Sequences with low motion don’t need full resolution coding –Key frames more bits Proposed method –Adaptive warping –Spatio-temporal

27 Overall System encoderdecoder

28 Experiment Setup New algorithm compared against H.264 in the following setup H.264 codec –Every N th frame as an I frame, the rest coded as P or B frames Proposed method –Every N th frame used as the reference frame –Other frames are decimated (LL subband) and coded –Total bit rate is sum of full resolution reference frame and the quarter resolution LL subbands Bit rates are the same in both cases

29 Result (1)

30 Subjective Result (a) H.264 (b) proposed method 3 rd frame for Salesman 80kbits/s

31 Conclusion The proposed method achieve better visual quality at low bitrates The gap decreases as the bitrates increase At high bitrates, H.264 has more bits to spend on high frequency components and thus achieves better quality The nature of the proposed method works better for sequences with more details Room remained to be improved Explore tradeoffs in spatio-temporal decimation rates More frequently for video with large motion and less often for video with small motion For long lecture video, we can choose full coverage of reference frame and no anymore later

32 Application III Video Frame Rate Up-Conversion Overview of FRUC –No motion vectors Frame repetition, frame average –Use motion vectors Use motion vectors from the decoder directly Advantage: Low complexity Disadvantage: Not true motion Perform motion estimation again Advantage: true motion Disadvantage: High computational complexity Frame Rate Up-Converter

33 Goal and Proposed Method Goal (1) True motion vectors (2) Relative low complexity Challenges (1) Highly accurate MVs (2) Low percentage of MV re-estimation (3) Occlusion (4) Blocking artifacts Approach –Decoded video sequences from the decoder –Additional information from the decoder System diagram Previous Reconstructed frame X1 Current residual frame, reconstructed frame X2, and its MVs MV Reliability Check MV Re-estimation Motion Compensated Interpolation Interpolated Frame Small Block Merging

34 MV reliability check Categoried into 3 groups

35 Small Block Merging Avoid broken edges and want to maintain object structure

36 MV Re-estimation Key in the system Accurate MVs are required for FRUC Low complexity A combination of Optical Flow-based and Block Matching- based motion estimation (Warping method)

37 Motion Compensated Interpolation N=2, k=1

38 Occlusion Uni-directional interpolation N=2, k=1

39 Overlapped Boundary Motion Compensation (OBMC) Goal: To reduce the blocking artifacts. Selectively perform OBMC to reduce the computational complexity (BAD > T)

40 Experiment Setup JM 11.0 GOP: IPP…P, 15 th I frame, fixed QP Code odd frame and skip every other frame 15fps Transform 8x8=2 mode on Search range = 16 Standard CIF Video sequences used: –Akiyo, News, Salesman, Foreman, Carphone –Flower Garden, Tempete –Football, Table Tennis Test against DMCFI, correlation-based motion selection 1 [1] Ai-Mei Huang and T. Nguyen, “Correlation-based motion vector processing for motion compensated interpolation”, ICIP 2008

41 Visual Result (1) 384kb/s (a) Orig (b) DMCFI 20.54dB (c ) Correlat ion- based 20.48dB (d) Proposed 24.19dB

42 Visual Result (2) 512kb/s (a) Orig (b) DMCFI 20.01dB (c ) Correlat ion- based 20.17dB (d) Proposed 20.13dB

43 Conclusion Proposed a FRUC method that combines optical flow and block matching-based motion estimation Reduced computational complexity Reduced blocking artifacts Achieve better visual quality for low motion video sequences and perform on par with other methods for high motion video sequences

44 Summary Provide a generic framework to achieve –Spatial enlargement of video frames –Video compression –Frame rate up-conversion Achiever higher objective and subjective results Improve the robustness by using FW, BW and hierarchical motion structure

45 Future Work Continue to refine the model Apply to higher resolution video Incorporate Subjective Video Quality Analysis Reference frame recycling –Adaptively select the position of high quality reference frame

46 Q & A

47 Application I Spatial Interpolation Related Work Frame restoration Frame interpolation –Bilinear, bicubic, spline, … –Adaptive Synthesis Filter Banks –New edge-directed interpolation Superresolution (SR)

48 Different Spatial Interpolation AkiyoCarphoneNewsSilenceMother Bilinear Bicubi NEDI VA Bi-Comp Cu-Comp NEDI-Comp VA-Comp SalesBusFlowerTempeteMobile StefanTableFootball