1 Robust Video Stabilization Based on Particle Filter Tracking of Projected Camera Motion (IEEE 2009) Junlan Yang University of Illinois,Chicago

2 Reference [1]A tutorial on particle filters for online nonlinear non- Gaussian Bayesian tracking [4]probabilistic video stabilization using kalman filtering and mosaicking [5]Fast electronic digital image stabilization for off-road navigation [18]condensation conditional density propagation for visual tracking

3 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

4 Introduction Video Stabilization –Camera motion estimation Particle filter –Tracking projected affine model of camera motion SIFT algorithm ( 范博凱 ) –Detect feature points in both images Removing undesired (unintended) motion –Kalman filter

5 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

6 Example of camera motion Motion Camera X Y Z (x 0,y 0,z 0 ) at time t 0 P Camera X Y Z (x 1,y 1,z 1 ) at time t 1 P

7 Generating Camera model Related of two vectors

8 Building 2-D affine model Projection of P in time t0 and t1

9 Building 2-D affine model Rewriting the related of two projected vectors 2-D affine model

10 Building 2-D affine model Global motion estimation is to determine the six parameters for every successive frame

11 Why do she use 2-D affine model to represent camera motion?  A pure 2-D model 2-D translation vector and one rotation angle  3-D model Giant complexity

12 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

13 Particle Filtering Estimation Markov discrete-time state-space model state vector at time k observations z, and the posterior density is

14 To approximate the posterior

15 Estimation of current state

16 Importance density q(.) Traditionally – prior density This paper takes into account the current observation z k. The proposed important density whose mean vector obtained from the current observation z k Why do she use the particle filtering estimation ?

17 Advantage of particle filtering estimation With Low error variance Proof : In large particle numbers condition, the estimation gives lower error variance than

18 Covariance matrix of errors

19 Lemma 1: where

20

21 Lemma 1: Strong law of large number

22

23

24 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

25 Complete system of video stabilization At time k

26 Getting six parameters SIFT algorithm – Find corresponding pairs At time k It needs three pairs to determine a unique solution YXA

27 (a) SIFT correspondence from frame 200,201 in outdoor sequence STREET

28 Generate particles Important density q(.) is a six-dimensional Gaussian distribution Particles In experience, N set to only 30 with better quality than prior distribution set N = 300

29 Quality of the particles N particles have N proposals of transformation matrix,and N Inverse transform to frame k have N candidate image A i Compare these images with k-1 frame A 0

30 Similar with A 0 and A i Mean square error –Difference of gray-scale from pixel to pixel Feature likelihood –Distance of all corresponding feature points

31 Particle filtering for global motion estimation Weight for each particle Estimation of current state where

32 Accumulative motion At time k-1 to k At time 0 to k Where s is scaling factor, R is rotation matrix and T is translation displacement

33

34 Intentional Motion estimation and motion compensation Compensate for the unwanted motion

35 Complete system of video stabilization At time k

36 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

37 (a)Original image, (b) Matched-feature-based motion estimation (MFME) (c) p-norm cost function-based motion estimation (CFME) (d) proposed method (PFME)

38 (a)Original image, (b) MFME (c) CFME (d) PFME

39 (a)Original image, (b) MFME (c) CFME (d) PFME

40 (a)Original video sequence (ground truth) (b) unstable video sequence (c) PFME

41 (a) Motion in horizontal direction (b) Motion in vertical direction Ty?Ty?

42 Comparison of average MSE and PSNR for stabilized output PSNR = peak signal to noise ratio Large PSNR has low distortion

43 Outline Introduction Camera Model Particle Filtering Estimation Complete System of Video Stabilization Simulation and Results Conclusion

44 Conclusion We demonstrated experimentally that the proposed particle filtering scheme can be used to obtain an efficient and accurate motion estimation in video sequences.

45 Contributed of this paper Constraining rotation matrix projected onto the plane ?(depth change) Show using particle filtering can reduce the error variance compared to estimation without particle filtering Using both Intensity-based motion estimation method (PFME) and feature-based motion estimation (SIFT) method