1. Computational Photography and Video: Video Synthesis Prof. Marc Pollefeys

2. Last Week: HDR

3. ScheduleComputational Photography and Video 24 FebIntroduction to Computational Photography 3 MarMore on Camera,Sensors and ColorAssignment 1 10 MarWarping, Mosaics and MorphingAssignment 2 17 MarBlending and compositingAssignment 3 24 MarHigh-dynamic rangeAssignment 4 31 MarVideo SynthesisProject proposals 7 AprEaster holiday – no classes 14 AprTBDPapers 21 AprTBDPapers 28 AprTBDPapers 5 MayProject update 12 MayTBDPapers 19 MayPapers 26 MayPapers 2 JuneFinal project presentation

4. Breaking out of 2D …now we are ready to break out of 2D But must we go to full 3D? 4D?

5. Today’s schedule Tour Into the Picture 1 Video Textures 2 1 Slides borrowed from Alexei Efros, who built on Steve Seitz’s and David Brogan’s 2 Slides from Arno Schoedl

6. We want more of the plenoptic function We want real 3D scene walk-throughs: Camera rotation Camera translation Can we do it from a single photograph? on to 3D…

7. Camera rotations with homographies St.Petersburg photo by A. Tikhonov Virtual camera rotations Original image

8. Camera translation Does it work? synthetic PP PP1 PP2

9. Yes, with planar scene (or far away) PP3 is a projection plane of both centers of projection, so we are OK! PP1 PP3 PP2

10. So, what can we do here? Model the scene as a set of planes! Now, just need to find the orientations of these planes.

11. Some preliminaries: projective geometry Ames Room alt.link

12. Silly Euclid: Trix are for kids! Parallel lines???

13. (0,0,0) The projective plane Why do we need homogeneous coordinates? – represent points at infinity, homographies, perspective projection, multi-view relationships What is the geometric intuition? – a point in the image is a ray in projective space (sx,sy,s) Each point (x,y) on the plane is represented by a ray (sx,sy,s) –all points on the ray are equivalent: (x, y, 1)  (sx, sy, s) image plane (x,y,1) y x z

14. Projective lines What does a line in the image correspond to in projective space? A line is a plane of rays through origin –all rays (x,y,z) satisfying: ax + by + cz = 0 A line is also represented as a homogeneous 3-vector l lp

15. l Point and line duality – A line l is a homogeneous 3-vector – It is  to every point (ray) p on the line: l p=0 p1p1 p2p2 What is the intersection of two lines l 1 and l 2 ? p is  to l 1 and l 2  p = l 1  l 2 Points and lines are dual in projective space given any formula, can switch the meanings of points and lines to get another formula l1l1 l2l2 p What is the line l spanned by rays p 1 and p 2 ? l is  to p 1 and p 2  l = p 1  p 2 l is the plane normal

16. Ideal points and lines Ideal point (“point at infinity”) – p  (x, y, 0) – parallel to image plane – It has infinite image coordinates (sx,sy,0) y x z image plane Ideal line l  (0, 0, 1) – parallel to image plane

17. Vanishing points Vanishing point – projection of a point at infinity – Caused by ideal line image plane camera center ground plane vanishing point

18. Vanishing points (2D) image plane camera center line on ground plane vanishing point

19. Vanishing points Properties – Any two parallel lines have the same vanishing point v – The ray from C through v is parallel to the lines – An image may have more than one vanishing point image plane camera center C line on ground plane vanishing point V line on ground plane

20. Vanishing lines Multiple Vanishing Points – Any set of parallel lines on the plane define a vanishing point – The union of all of these vanishing points is the horizon line also called vanishing line – Note that different planes define different vanishing lines v1v1 v2v2

21. Vanishing lines Multiple Vanishing Points – Any set of parallel lines on the plane define a vanishing point – The union of all of these vanishing points is the horizon line also called vanishing line – Note that different planes define different vanishing lines

22. Computing vanishing points Properties – P  is a point at infinity, v is its projection – They depend only on line direction – Parallel lines P 0 + tD, P 1 + tD intersect at P  V P0P0 D

23. Computing vanishing lines Properties – l is intersection of horizontal plane through C with image plane – Compute l from two sets of parallel lines on ground plane – All points at same height as C project to l points higher than C project above l – Provides way of comparing height of objects in the scene ground plane l C

24. Fun with vanishing points

25. “Tour into the Picture” (SIGGRAPH ’97) Horry, Anjyo, Arai Create a 3D “theatre stage” of five billboards Specify foreground objects through bounding polygons Use camera transformations to navigate through the scene

26. The idea Many scenes (especially paintings), can be represented as an axis- aligned box volume (i.e. a stage) Key assumptions: – All walls of volume are orthogonal – Camera view plane is parallel to back of volume – Camera up is normal to volume bottom How many vanishing points does the box have? – Three, but two at infinity – Single-point perspective Can use the vanishing point to fit the box to the particular Scene!

27. Fitting the box volume User controls the inner box and the vanishing point placement (# of DOF???) Q: What’s the significance of the vanishing point location? A: It’s at eye level: ray from COP to VP is perpendicular to image plane.

28. High Camera Example of user input: vanishing point and back face of view volume are defined

29. High Camera Example of user input: vanishing point and back face of view volume are defined

30. Low Camera Example of user input: vanishing point and back face of view volume are defined

31. Low Camera Example of user input: vanishing point and back face of view volume are defined

32. High CameraLow Camera Comparison of how image is subdivided based on two different camera positions. You should see how moving the vanishing point corresponds to moving the eyepoint in the 3D world.

33. Left Camera Another example of user input: vanishing point and back face of view volume are defined

34. Left Camera Another example of user input: vanishing point and back face of view volume are defined

35. Right Camera Another example of user input: vanishing point and back face of view volume are defined

36. Right Camera Another example of user input: vanishing point and back face of view volume are defined

37. Left CameraRight Camera Comparison of two camera placements – left and right. Corresponding subdivisions match view you would see if you looked down a hallway.

38. 2D to 3D conversion First, we can get ratios leftright top bottom vanishing point back plane

39. Size of user-defined back plane must equal size of camera plane (orthogonal sides) Use top versus side ratio to determine relative height and width dimensions of box Left/right and top/bot ratios determine part of 3D camera placement leftright top bottom camera pos 2D to 3D conversion

40. Depth of the box Can compute by similar triangles (CVA vs. CV’A’) Need to know focal length f (or FOV) Note: can compute position on any object on the ground – Simple unprojection – What about things off the ground?

41. DEMO Now, we know the 3D geometry of the box We can texture-map the box walls with texture from the image link to web page with example code TIP demo

42. Foreground Objects Use separate billboard for each For this to work, three separate images used: – Original image. – Mask to isolate desired foreground images. – Background with objects removed

43. Foreground Objects Add vertical rectangles for each foreground object Can compute 3D coordinates P0, P1 since they are on known plane. P2, P3 can be computed as before (similar triangles)

44. Foreground TIP movie UVA example

45. See also… Tour into the picture with water surface reflection Tour into the picture with water surface reflection Tour into the Video: Tour into the Video – by Kang + Shin

46. Today’s schedule Tour Into the Picture 1 Video Textures 2 1 Slides borrowed from Alexei Efros, who built on Steve Seitz’s and David Brogan’s 2 Slides from Arno Schoedl

47. Markov Chains probability of going from state i to state j in n time steps: and the single-step transition as: The n-step transition satisfies the Chapman-Kolmogorov equation,Chapman-Kolmogorov equation that for any 0<k<n:

48. Regular Markov chain: class of Markov chains where the starting state of the chain has little or no impact on the p(X) after many steps. Markov Chains

49. Markov Chain What if we know today and yestarday’s weather?

50. Text Synthesis [ Shannon,’48] proposed a way to generate English- looking text using N-grams: – Assume a generalized Markov model – Use a large text to compute prob. distributions of each letter given N-1 previous letters – Starting from a seed repeatedly sample this Markov chain to generate new letters – Also works for whole words WE NEEDTOEATCAKE

51. Mark V. Shaney (Bell Labs) Results (using alt.singles corpus): – “As I've commented before, really relating to someone involves standing next to impossible.” – “One morning I shot an elephant in my arms and kissed him.” – “I spent an interesting evening recently with a grain of salt”

52. Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech Link to local versionGondry Example

53. Still photos

54. Video clips

55. Video textures

56. Problem statement video clipvideo texture

57. Our approach How do we find good transitions?

58. Finding good transitions Compute L 2 distance D i, j between all frames Similar frames make good transitions frame ivs. frame j

59. Markov chain representation Similar frames make good transitions

60. Transition costs Transition from i to j if successor of i is similar to j Cost function: C i  j = D i+1, j

61. Transition probabilities Probability for transition P i  j inversely related to cost: P i  j ~ exp ( – C i  j /  2 ) high  low 

62. Preserving dynamics

64. Cost for transition i  j C i  j = w k D i+k+1, j+k

65. Preserving dynamics – effect Cost for transition i  j C i  j = w k D i+k+1, j+k

66. Dead ends No good transition at the end of sequence

67. Future cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

68. Future cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

69. Future cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

70. Future cost Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k

71. Propagate future transition costs backward Iteratively compute new cost F i  j = C i  j +  min k F j  k Q-learning Future cost

72. Future cost – effect

73. Finding good loops Alternative to random transitions Precompute set of loops up front

74. Visual discontinuities Problem: Visible “Jumps”

75. Crossfading Solution: Crossfade from one sequence to the other.

76. Morphing Interpolation task:

77. Morphing Interpolation task: Compute correspondence between pixels of all frames

78. Morphing Interpolation task: Compute correspondence between pixels of all frames Interpolate pixel position and color in morphed frame based on [Shum 2000]

79. Results – crossfading/morphing

80. Jump Cut Crossfade Morph video

81. Crossfading

82. Frequent jump & crossfading

83. Video portrait Useful for web pages

84. Combine with IBR techniques Video portrait – 3D

85. Region-based analysis Divide video up into regions Generate a video texture for each region

86. Automatic region analysis

87. User selects target frame range User-controlled video textures slowvariablefast

88. Video-based animation Like sprites computer games Extract sprites from real video Interactively control desired motion ©1985 Nintendo of America Inc.

89. Video sprite extraction Blue screen matting and velocity estimation

90. Video sprite control Augmented transition cost:

91. Video sprite control Need future cost computation Precompute future costs for a few angles. Switch between precomputed angles according to user input [GIT-GVU-00-11]

92. Interactive fish

93. Summary Video clips  video textures – define Markov process – preserve dynamics – avoid dead-ends – disguise visual discontinuities

94. Discussion Some things are relatively easy

95. Discussion Some are hard

96. A final example

97. Michel Gondry train video http://youtube.com/watch?v=qUEs1BwVXGA

98. Video Sprites Link to Video Sprites summary

99. Trainable Videorealistic Speech Animation SIGGRAPH 2002 paper by Tony Ezzat, Gadi Geiger, and Tomaso Poggio – At Center for Biological and Computational Learning, MIT MikeTalk link (from 1998) MikeTalk link Mary101 (web page of this research) Mary101