Robust global motion estimation and novel updating strategy for sprite generation IET Image Processing, Mar H.K. Cheung and W.C. Siu The Hong Kong Polytechnic Univ. ( 香港理工大學 )

Outlines  Overview / Introduction  Proposed system New global motion estimation  Combing short- and long-term estimation  Dynamic reference frame 2-pass sprite blending  Preserving frame resolution loss Sprite updating  Overcoming illumination variations & object changing  Experimental results  Conclusions

Overview

 Sprite High resolution image Composed of information belonging to an object visible throughout a video sequence  Background of a scene

Overview  Sprite background of frame 1 (Dimension: 352x288) background of frame 20 Sprite (Dimension: 2670x1072)

Overview  Core of sprite generation Global motion estimation (GME)  Finding a set of parameters representing camera motion between frames Image registration Iterative minimization Blending  Temporal (weighted) averaging, median, updating

Introduction

 Global motion estimation Image registration Short-term motion estimation  Estimation between consecutive frames  Easy and accurate Long-term motion estimation  Estimation between frames with temporal distance  Harder  Required to perform sprite coding Single sprite for all frames in sequence

Introduction  Global motion estimation (cont.) Short- to long-term estimation  Converting short-term motion parameters to long- term parameters  Error propagation Directly long-term estimation  Estimation every frames directly to a specified base frame (reference frame) No error propagation  Search range may be huge  Hard to find overlapping area

Introduction  Global motion estimation (cont.) Hierarchical estimation  Rough estimation to find coarse parameters  Refining parameters Using coarse parameters as initials Iterative minimization  Some existing methods Dufaux and Konrad Szeliski Smolic et. al. Lu et. al.

Introduction  Restrictions Background must be really static  Background objects must be still  No illumination variations Dynamic sprite

Introduction  Classification Static sprite  Build offline before coding individual frames  Quality degradation as frame increases Motion estimation errors Illumination variations Background object changes Dynamic sprite  Built dynamically online in both encoder and decoder while coding individual frames Sprite is updated using reconstructed frame  Short-term estimation is employed Error accumulated

Introduction  Proposed system New global motion estimation  Directly estimating the relative motion between current image and a chosen reference frame Give accurate, stable and robust estimation Alleviate error accumulation  Hierarchical 3-levels approach Coarse-to-fine approach Sprite updating  Updating sprite only if necessary Sprite update frames are generated and sent

Proposed system

 Short-term GME to long-term GME Frame 1 A 11 Frame m A m1 Frame m+1 GME reference frame …… A (m+1)1 A m1 + A (m+1)m Registration Error = A (m+1)m  A m(m-1)  …  A 21 A (m+1)k = A (m+1)m  A m1 Registration Error Registration errors are ACCUMULATED More Error

Proposed system  Directly measure to reference frame GME Frame 1 A 11 Frame m A m1 Frame m+1 reference frame …… A (m+1)1 A m1 initial guess Registration Error Registration errors are COMPENSATED

Proposed system  Weakness Reference frame is temporally far from current frame  Frame contents may change largely Background objects activities Lighting conditions changes Overlapping area could be smaller  Unfavorable to GME

Proposed system  Combining the advantages Dividing video into groups of consecutive frames 1st frame of each group is selected as reference  Frames in a group Each frame is directly measured to the 1st frame  Smaller registration error  Merging groups GMEs of reference frames of all groups are merged  Registration error is slightly increased R1R2R3 …… ++ A (R1)(R1) A (R2)(R1) A (R3)(R2) A (R2)(R1) A (R3)(R1)

Proposed system  Proposed GME structure Motion Estimation Frame k A k1 Frame m A mk A m1 Frame m+1 Frame z Chosen to be reference frame …… A (m+1)k A mk

Proposed system  Dynamic reference frame 1st frame is the initial reference frame Assigning current frame as new reference frame if  Displaced frame difference between registered current frame and the reference frame it large Reference frame is not like current frame  Relative displacement between current frame and the reference frame is large Overlapping area is too small or where Nr is a parameter between 0 and 1 (Nr=0.1 in practical)

Proposed system  Advantages Accuracy  Accurate than short-term and directly long-term estimation Very few memory usage  Estimations are performed frame-to-frame  Sprite building is not necessary

Proposed system  GME Reference frame (frame k) Frame z Three step search Block-based partial distortion search Fast gradient method A (m+1)k A mk +

Proposed system  Motion model Perspective motion model  8 motion parameters to be determined  Three-step matching 3-level pyramids for frame k and z are built using Gaussian down-sampling filter [ ¼, ½, ¼ ] frame k: reference frame frame z: transformed current frame m+1

Proposed system  Block-matching Affine parameters are estimated by solving over- fitting equations Results of block-based motion estimation are used to construct the equations  Parameter estimation Fast gradient descent method by Keller and Averbuch where

Proposed system  Two-passed blending to avoid resolution loss First pass: 1st frame as base frame  All frames are projected into 1st frame  Frame with minimal area of projected frame is selected as new base frame Avoiding resolution loss  No real pixel blending applied Second pass: new base frame  All frames are projected into new base frame  Simple temporal average blending With bilinear interpolation

Proposed system  Dynamic sprite updating Overcoming illumination variations  Single value in sprite can not represent intensity variations over the time Accumulation of GME error blurring the frame  GME error in a reference frame will inherit into all of frames in the group

Proposed system  Studying the generated intensity error an edge pixel a pixel from homogeneous area a pixel from texture area translation in x-direction # of pixel with significant error

Proposed system  Distribution of intensity error correlates roughly to the panning motion Errors tends to be clustered in the temporal domain  Errors of homogeneous and texture regions are tend to randomly around zero

Proposed system  Sprite updating Selecting frames with significant change in panning direction/speed

Proposed system  Sprite updating (cont.) Reconstruct next N frame from the sprite Blend the N error frames into a sprite-sized buffer (the sprite update frame) Compute the N error frames Encode and send the sprite update frame to the decoder MPEG4 I-VOP frame

Experimental results

 Testing Constructing sprite Reconstructing frames from sprite Compute PSNR  Comparison Short-term motion estimation  Estimating between current and previous frame Long-term motion estimation  Estimating between current frame and sprite  No parameters predicting Long-term motion estimation by MPEG-4 VM Long-term motion estimation by Smolic et. al.

Experimental results Short-term Long-term

Experimental results MPEG-4 VM Proposed method

Experimental results  PSNR Proposed MPEG-4 Short-term Long-term Smolic et. al.

Experimental results  Average PSNR (dB) Short- term Long- term MPEG-4 VM Smolic et. al. Proposed (affine) Proposed (per- spective) Stefan (150) Foreman (150) Coast Guard (150) Stefan (300) Failure

Experimental results  Selecting threshold Nr Proposed method is better than simple short-term and long-term estimation Short-term 0.1 Long-term

Experimental results  Performance of sprite updating SequenceUpdate framesAverage PSNR (dB)Size of updates (kB) stefan stefan0,51,108,174,206 * stefan0,60,120,180, stefan0,51,108,106 * stefan0,80,160, coast guard coast guard0,76 * foreman foreman0,10,25,64,110 * foreman0,30,60,90, * Update frames is figured out from the major camera operations of the sequences

Conclusions

 New global motion estimation method Directly estimation from current frame to a chosen reference frame Combing advantages of short-term and long-term estimation  Error accumulation prevented  Keeping reference frame close to current frame  Sprite updating Encoding & sending sprite update frames  Errors of a group of reconstructed frames Reducing sprite blurring