1. Dynamosaicing Dynamosaicing Mosaicing of Dynamic Scenes 2007. 11. 16 (Fri) Young Ki Baik Computer Vision Lab Seoul National University

2. Dynamosaicing: Mosaicing of Dynamic scenes  References Dynamosaicing: Mosaicing of Dynamic Scenes Alex Rav-Acha, Yael Pritch, Dani Lischinski and Shumuel Peleg (PAMI 2007.10) www.vision.huji.ac.il/dynmos

3. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? Are you getting the whole picture? = 50 x 35° Compact camera FOV = 50 x 35°

4. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? Are you getting the whole picture? = 50 x 35° Compact camera FOV = 50 x 35° = 200 x 135° Human FOV = 200 x 135°

5. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? Are you getting the whole picture? = 50 x 35° Compact camera FOV = 50 x 35° = 200 x 135° Human FOV = 200 x 135° Panoramic Mosaic = 360 x 180° Panoramic Mosaic = 360 x 180°

6. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? How does mosaicing work? ?

7. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? How does mosaicing work? New wide FOV camera New image plane

8. Dynamosaicing: Mosaicing of Dynamic scenes  What is Mosaicing in Vision? Are you getting the whole picture?

9. Dynamosaicing: Mosaicing of Dynamic scenes  Dynamosaicing Purpose: The wide scene movie from the normal video data Synthesis of new movie

10. Dynamosaicing: Mosaicing of Dynamic scenes  Dynamosaicing Contribution: Found out new interesting application. (50%) Proposed new constancy for image registration. (20%) Introduced various video editing trick with simple idea. (20%) Applied Min-cut algorithm to video data in order to overcome some artifacts. (10%)

11. Dynamosaicing: Mosaicing of Dynamic scenes  Dynamosaicing How does Dynamosaicing work? Video alignment An initial task of Dynomosaicing Evolving time front For making wide screen movie and editing videos Time front optimization : Graph-cut (Min-cut) For seamless dynamosaicing

12. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Space time volume txtxk Stationary Camera Panning Camera x : Image x axis t : Time axis k : Frame index

13. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Purpose of Video Alignment Changing video data from panning camera like the video taken by stationary camera It is similar to conventional 2D static mosaicing. (Image registration) tx Aligned video txk Unaligned video

14. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation What is the differences? Dynamosaicing can treats Dynamic scenes. Conventional mosaicing Only treat static objects. Dynamosaicing Permit the presence of moving objects. Brightness constancy Dynamic constancy

15. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation What is the differences? Brightness constancy - Only consider intensity values of images

16. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation What is the differences? Brightness constancy - Only consider intensity values of images

17. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Assumption : If If two different space time blocks are similar … Then their continuations are also similar!!!!

18. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Aligned space time volume and space time block t x k-1 Assumption : we have already computed the aligned data up to k-1. we have already computed the aligned data up to k-1. x y t t k-1

19. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Aligned space time volume and space time block t x k-1 Space time block is same as a 3D window. x y t t k-1

20. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Estimating k-th frame (Extrapolation) Using previous aligned time volume I est (x,y) I est : Estimated image for k-th frame If you want to predict the value of x, y on I est …

21. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Estimating k-th frame (Extrapolation) Space time block is assigned to target (x,y) position of k-1 frame. k-1 x y t x t k-1 7x7x7space time block 7x7x7space

22. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Estimating k-th frame (Extrapolation) Matching all space time block of space time volume between k-2 frame and initial frame k-1 x y t xtk-1

23. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Estimating k-th frame (Extrapolation) Matching cost : SSD (sum of square differences) W : space-time block

24. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Estimating k-th frame (Extrapolation) If you found out best matched W… ~ Intensity value of next frame is assigned to the value of wanted position on k-th frame. x t k-1 k

25. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Registration k-th frame Matching between real k-th frame and estimated k frame… I est (x,y) I cap (x,y)

26. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Dynamic Constancy Updating space time volume x t k I cap (x,y) of k-th frame

27. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Masking Unpredictable region There exist some unpredictable region… Raising hand, changing direction… W 2d : 2D window I x, I y : color differences with neighborhood W 2d : 2D window I x, I y : color differences with neighborhood

28. Dynamosaicing: Mosaicing of Dynamic scenes  Video Alignment using Video Extrapolation Overall algorithm Obtain The motion of the first k frames by using conventional optical flow.Obtain The motion of the first k frames by using conventional optical flow. Align all frame in the space time volume.Align all frame in the space time volume. Compute the motion parameters between captured and estimated frames.Compute the motion parameters between captured and estimated frames. Estimate the next new frame by extrapolation from the previous frames.Estimate the next new frame by extrapolation from the previous frames.

29. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front What is evolving time front? For making wide screen movie and editing videos txtxk Stationary Camera Panning Camera

30. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Stationary camera case New video are created by editing time front tx Stationary Camera t x

31. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Stationary camera case New video are created by editing time front

32. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Stationary camera case New video are created by editing time front

33. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Panning camera case New wide FOV video are created by editing time front as follows… tx Panning Camera t x

34. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Panning camera case New wide FOV video are created by editing time front as follows… t x Panning Camera

35. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Panning camera case New wide FOV video are created by editing time front as follows… t x Panning Camera

36. Dynamosaicing: Mosaicing of Dynamic scenes  Evolving time front Panning camera case New wide FOV video are created by editing time front as follows…

37. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization What is the problem of time front ? There exist some artifacts from seams at the middle of moving objects. Linear stitching Using Min-cut

38. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization What is the problem of time front ? There exist some artifacts from seams at the middle of moving objects. t x t x Linear stitching Using Min-cut

39. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Assumtion S l (x,y) = V(x,y, M k (x,y) ) txt x Source space time volume Target space time volume k l SlSlSlSl MkMkMkMk V

40. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function E(M) = E shape (M) + αE stitch-3d (M) α : balances between the two E (α =0.3 when gray values were between 0~255) (α =0.3 when gray values were between 0~255)

41. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function E(M) = E shape (M) + αE stitch-3d (M) t x Source space time volume k M’ k V E shape (M) MkMkMkMk E shape (M k )=Σ D(M’ k, M k ) M’ k : Predefined(or user defined) time front function M’ k : Predefined(or user defined) ~~~~ time front function M k : Desired time front function

42. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function E(M) = E shape (M) + αE stitch-3d (M) t x k V MkMkMkMk E stitch-3d (M) E stitch-3d (M)=Σ (x,y) Σ (x’,y’)Є N(x’,y’) Σ (M(x,y)~k~M(x’,y’)) Σ (M(x,y)~k~M(x’,y’)) 0.5*ssd(V (x,y,k), V(x,y,k+1)) 0.5*ssd(V (x,y,k), V(x,y,k+1)) +0.5*ssd(V (x’,y’,k), V(x’,y’,k+1)) +0.5*ssd(V (x’,y’,k), V(x’,y’,k+1))

43. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function Preserve user defined time front function Preserve the color for each pair of spatial neighboring output pixels (x,y) and (x’,y’). E(M) = E shape (M) + αE stitch-3d (M) E shape (M) E stitch-3d (M)

44. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut We can just apply cost function to Min-cut. We can make seamless video with 3D Min-cut. In order to accomplish more reliable results, We need to consider relation between past and future.

45. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Eovolving Time Front as 4D Min-cut Cost function Temporal consistency of M in both past and future. E(M) = E shape (M) + αE stitch-4d (M) E temporal (M) E stitch-4d (M)= Σ l E stitch-3d (M) +Σ (x,y,t) E temporal (x,y,t) +Σ (x,y,t) E temporal (x,y,t)

46. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Eovolving Time Front as 4D Min-cut Cost function Preserving Temporal Consistency E(M) = E shape (M) + αE stitch-4d (M) E temporal (x,y,t)= Σ ( M t(x,y)~k~ M t+1(x,y)) 0.5*ssd(V (x,y,k-1), V(x,y,k)) 0.5*ssd(V (x,y,k-1), V(x,y,k)) +0.5*ssd(V (x’,y’,k), V(x’,y’,k+1)) +0.5*ssd(V (x’,y’,k), V(x’,y’,k+1))

47. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Eovolving Time Front as 4D Min-cut We can just apply cost function to Min-cut. We can make seamless video with 4D Min-cut. Stitch Stitch Shape Shape Temporal 3D Min-cut 4D Min-cut

48. Dynamosaicing: Mosaicing of Dynamic scenes  Final results

49. Dynamosaicing: Mosaicing of Dynamic scenes  Final results

50. Dynamosaicing: Mosaicing of Dynamic scenes  Conclusion Video alignment An initial task of Dynomosaicing Evolving time front For making wide screen movie and editing videos Time front optimization : Graph-cut (Min-cut) For seamless dynamosaicing  Discussion Contribution Found out new interesting application. Proposed new constancy for image registration. Introduced various video editing with simple idea. Applied Min-cut algorithm to overcome some artifact.

51. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function M(x,y) : Desired time front function M’(x,y) : Predefined(or user defined) time front function || || : l 1 norm E(M) = E shape (M) + αE stitch-3d (M)

52. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Single Time Front as 3D Min-cut Cost function E(M) = E shape (M) + αE stitch-3d (M) M(x,y) <= M(x’, y’)

53. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Eovolving Time Front as 4D Min-cut Cost function Temporal consistency of M in both past and future. E(M) = E shape (M) + αE stitch-4d (M) E temporal (M)

54. Dynamosaicing: Mosaicing of Dynamic scenes  Time front optimization Eovolving Time Front as 4D Min-cut Cost function Preserving Temporal Consistency E(M) = E shape (M) + αE stitch-4d (M) M t (x, y) <= M t+1 (x, y)