View Morphing by Steven M. SeitzCharles R. Dyer Irwin Chiu Hau Computer Science McGill University Winter 2004 Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: n.edu/homes/seitz/vmorp h/vmorph.htm

View Morphing What is view morphing? –What is it? –Why do we care about it? –Where do we see them? –Image Morphing vs View interpolation vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Virtual Cameras Source: View Morphing; Steven M. Seitz, Charles R. Dyer

View Morphing: Key Idea Beier-Neely morph is NOT shape-preserving! –distortions –un-natural Comp 767: Advanced Topics in Graphics: Image-Based Rendering A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer

View Morphing: Key Idea View morphing uses 3D shape preserving morph! –no distortions –natural A morph is 3D shape preserving if the results of two different views represent new views of the same object Comp 767: Advanced Topics in Graphics: Image-Based Rendering A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer View morphing fixes these intermediate steps!

Why do we care? View morphing is efficient Produces new views without –3D modelling –Taking additional photos View morphing creates impressive effects –Camera motion –Image morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering trueSpace Source:

Image Morphing vs View interpolation vs View Morphing View Morphing is an extension to Image Morphing (Beier and Neely, 1992) –Produces physically plausible new views of a scene View Morphing (Seitz and Dyer, 1996) is an improvement over View Interpolation (Chen and Williams, 1993) –Addresses non-rigid transformations problems –Does not require depth values –Creates realistic image transitions Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: n.edu/homes/seitz/vmorp h/vmorph.htm

How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer

How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View morphing in 1 steps: Assume parallel views! 1.Compute a morph between the parallel images

Parallel Views Basic Theory Comp 767: Advanced Topics in Graphics: Image-Based Rendering image point p 0 = (x 0,y 0 ) scene point P = (X,Y,Z) p 0 = Π 0 P where Π 0 is a projection matrix Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer image point p 1 = (x 1,y 1 ) scene point P = (X,Y,Z) p 1 = Π 1 P where Π 1 is a projection matrix

Parallel Views Basic Theory Comp 767: Advanced Topics in Graphics: Image-Based Rendering image point p = (x,y) scene point P = (X,Y,Z) p = Π P where Π is a homogenous projection matrix Π = [ H | - HC] H: position and orientation of image plane C: euclidean position of the camera Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer

Parallel Views Mathematics for View Interpolation Comp 767: Advanced Topics in Graphics: Image-Based Rendering Πs: Linear interpolation of Π 0 and Π 1 Πs = (1 – s) Π 0 + s Π 1 s = [0,1] Cs = (sCx, sCy, 0) fs = (1 – f) f 0 + s f 1 f : focal lengh C : center of a camera

View Interpolation Recap View Interpolation vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer View Interpolation Source: 3D Games by Alan Watt and Fabio Policarpo

How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View morphing in 1 steps: Assume parallel views! 1.Compute a morph between the parallel images

Non-Parallel Views The General Case Comp 767: Advanced Topics in Graphics: Image-Based Rendering This is where the 3-Step Algorithm comes into play 1.Prewarping: I 0 to Î 0 and I 1 to Î 1 2.Morphing: Î 0 and Î 1 into Î s 3.Postwarping: Î s into I s Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer

Non-Parallel Views Mathematics for Image Reprojection Comp 767: Advanced Topics in Graphics: Image-Based Rendering ĤH -1 p = p H and Ĥ are 3x3 matrices that represent the position and the orientation of their image planes The resulting 3x3 matrix, ĤH -1 is a projective transformation that reprojects the image plane I into Î Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer ^

Limitations Singular Views cannot be reprojected to form parallel views Singular configurations are settings where one of the camera resides in the field of view of another camera Still works, just kind of conceptually hacky Comp 767: Advanced Topics in Graphics: Image-Based Rendering Parallel view Source: Steven M. Seitz, Charles R. Dyer Singular view Source: Steven M. Seitz, Charles R. Dyer

Traditional Problems Change in visibility creates 1.Folds –Occurs when a visible area becomes occluded 2.Holes –Occurs when an occluded area becomes visible Comp 767: Advanced Topics in Graphics: Image-Based Rendering Penumbra, umbra and hole regions Source: Chen and Williams Fold Source: Irwin Chiu Hau Area Point

Producing The Morph We have talked about theory behind algorithm 1.How to project/unproject images to parallel planes 2.How to warp between parallel planes 3.Theoretical problems Now, let’s do an actual View Dependent Morph! Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Producing The Morph We need: 1.Two images I 0 and I 1 2.Two perspective projection matrices Π 0 and Π 1 3.Correspondence between pixels Note that a sequence of projection matrices Π s is required to control the entire animation, but Π s can be computed automatically if we know H s. Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Controlling The Morph H s can be obtained indirectly by establishing constraints –Recall H s is position and orientation of image plane –One way is to specify four control points Note: Control points implicitly determine the postwarping transformation Comp 767: Advanced Topics in Graphics: Image-Based Rendering View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer Four control points form the red bounding box to determine the postwarping stage Yellow lines are set of features to determine the prewarping stage

View Morphing sans Prewarping Prewarping is not necessary for: –Objects that aren’t closely related Prewarping is less effective Computation is unstable –Images that are approx. orthographic (eg. telephoto) However, postwarping should not be left out to: –Reduce image plane distorsions Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: n.edu/homes/seitz/vmorp h/vmorph.htm

Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Facial view morphs Source: Steven M. Seitz, Charles R. Dyer

Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Facial view morphs Source:

Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source:

Results Image Morphing vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Image Morphing vs View Morphing Source: Steven M. Seitz, Charles R. Dyer

Conclusions Things to remember View Morphing Powerful extension to image morphing Produces new views of a scene 3D shape preserving Comp 767: Advanced Topics in Graphics: Image-Based Rendering

References View Morphing - Seitz and Dyer, 1996 View Interpolation - Chen and Williams, 1993 Image Morphing - Beier and Neely, D Games: Realtime rendering and Sofware Technology - Alan Watt and Fabio Policarpo Comp 767: Advanced Topics in Graphics: Image-Based Rendering

Questions? Comp 767: Advanced Topics in Graphics: Image-Based Rendering