1. CS559: Computer Graphics Lecture 7: Image Warping and Panorama Li Zhang Spring 2008 Most slides borrowed from Yungyu ChuangYungyu Chuang

2. Today Image Warping and Morphing Reading – Shirley, Ch 5, Linear Algebra – Shirley, Ch 6.1, 2D Transform – Shirley, Ch 6.3, Translation, p151-152(1 st paragraph) – (Optional) Image morphing: a survey, George Wolberg, Visual Computer 1998

3. Changing pixel values Moving pixels around Image Manipulation h f g h([x,y])=[x,y/2]

4. Parametric (global) warping translation rotation aspect affine perspective cylindrical Examples of parametric warps:

5. Application of Image Warp Creating virtual wide-angle camera Mosaics: stitching images together

6. Application of Image Warp Creating realistic surface appearance Texture mapping http://www.futuretech.blinkenlights.nl/tex.html

7. Application of Image Warp Morphing morphing image #1image #2

8. Parametric (global) warping Transformation T is a coordinate-changing machine: p ’ = T(p) What does it mean that T is global? – can be described by just a few numbers (parameters) – the parameters are the same for any point p Represent T as a matrix: p ’ = Mp T p = (x,y)p’ = (x’,y’) h([x,y])=[x,y/2]

9. Scaling Scaling a coordinate means multiplying each of its components by a scalar Uniform scaling means this scalar is the same for all components:  2 2 f g

10. Non-uniform scaling: different scalars per component: Scaling x  2, y  0.5

11. Scaling Scaling operation: Or, in matrix form: scaling matrix S What’s inverse of S?

12. 2-D Rotation  (x, y) (x’, y’) x’ = x cos(  ) - y sin(  ) y’ = x sin(  ) + y cos(  )

13. 2-D Rotation This is easy to capture in matrix form: How can I remember this? Even though sin(  ) and cos(  ) are nonlinear to , – x ’ is a linear combination of x and y – y ’ is a linear combination of x and y What is the inverse transformation? – Rotation by –  – For rotation matrices, det(R) = 1 R x’ = x cos(  ) - y sin(  ) y’ = x sin(  ) + y cos(  )

14. 2x2 Matrices What types of transformations can be represented with a 2x2 matrix? 2D Identity? 2D Scale around (0,0)?

15. 2x2 Matrices What types of transformations can be represented with a 2x2 matrix? 2D Rotate around (0,0)? 2D Shear?

16. What types of transformations can be represented with a 2x2 matrix? 2x2 Matrices 2D Shear? (1,1) 1 1 0 x y 1 1 0 x’ y’ sh y sh x

17. 2x2 Matrices What types of transformations can be represented with a 2x2 matrix? 2D Mirror about Y axis? 2D Mirror over (0,0)?

18. Linear transformations are combinations of … – Scale, – Rotation, – Shear, and – Mirror Any 2D transform can be decomposed into the product of a rotation, scale, and a rotation All 2D Linear Transformations

19. Linear transformations are combinations of … – Scale, – Rotation, – Shear, and – Mirror A symmetric 2D transform can be decomposed into the product of a rotation, scale, and the inverse rotation All 2D Linear Transformations

20. Linear transformations are combinations of … – Scale, – Rotation, – Shear, and – Mirror Properties of linear transformations: – Origin maps to origin – Lines map to lines – Parallel lines remain parallel – Ratios are preserved – Closed under composition

21. 2x2 Matrices What types of transformations can not be represented with a 2x2 matrix? 2D Translation? Only linear 2D transformations can be represented with a 2x2 matrix NO!

22. Translation Example of translation t x = 2 t y = 1 Homogeneous Coordinates

23. Affine Transformations Affine transformations are combinations of … – Linear transformations, and – Translations Properties of affine transformations: – Origin does not necessarily map to origin – Lines map to lines – Parallel lines remain parallel – Ratios are preserved – Closed under composition – Models change of basis

24. Image warping Given a coordinate transform x ’ = T(x) and a source image I(x), how do we compute a transformed image I ’ (x ’ ) = I(T(x))? I(x)I(x)I’(x’) xx’ T(x)T(x)

25. Forward warping Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’) I(x)I(x)I’(x’) xx’ T(x)T(x)

26. Forward warping fwarp(I, I’, T) { for (y=0; y<I.height; y++) for (x=0; x<I.width; x++) { (x’,y’)=T(x,y); I’(x’,y’)=I(x,y); } II’ x x’ T

27. Forward warping Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’) f(x)f(x)g(x’) xx’ h(x)h(x) What if pixel lands “between” two pixels? Will be there holes? Answer: add “contribution” to several pixels, normalize later (splatting)

28. Forward warping fwarp(I, I’, T) { for (y=0; y<I.height; y++) for (x=0; x<I.width; x++) { (x’,y’)=T(x,y); Splatting(I’,x’,y’,I(x,y),kernel); } II’ x x’ T

29. Splatting Computed weighted sum of contributed colors using a kernel function, where weights are normalized values of filter kernel k, such as Gauss radius d for all q q.color = 0; q.weight = 0; for all p from source image for all q’s dist < radius d = dist(p, q); w = kernel(d); q.color += w*p; q.weight += w; for all q q.Color /= q.weight; p q Destination Image May get a blurry image!

30. Inverse warping Get each pixel I’(x’) from its corresponding location x = T -1 (x’) in I(x) I(x)I(x)I’(x’) xx’ T -1 (x’)

31. Inverse warping iwarp(I, I’, T) { for (y=0; y<I’.height; y++) for (x=0; x<I’.width; x++) { (x,y)=T -1 (x’,y’); I’(x’,y’)=I(x,y); } II’ x x’ T -1

32. Inverse warping Get each pixel I’(x’) from its corresponding location x = T -1 (x’) in I(x) What if pixel comes from “between” two pixels? Answer: resample color value from interpolated source image f(x)f(x)g(x’) xx’

33. Inverse warping iwarp(I, I’, T) { for (y=0; y<I’.height; y++) for (x=0; x<I’.width; x++) { (x,y)=T -1 (x’,y’); I’(x’,y’)=Reconstruct(I,x,y,kernel); } II’ x x’ T -1

34. Reconstruction (interpolation) Possible reconstruction filters (kernels): – nearest neighbor – bilinear – bicubic – sinc

35. Bilinear interpolation (triangle filter) A simple method for resampling images

36. Non-parametric image warping

37. Specify a more detailed warp function T [x,y] [x’,y’]

38. Non-parametric image warping Specify a more detailed warp function Tabulate pixel motion (lookup table)

39. Non-parametric image warping Mappings implied by correspondences Inverse warping P’ ?

40. Non-parametric image warping

41. Warping between two triangles Idea: find an affine that transforms ABC to A’B’C’ A C B A’ C’ B’

42. Warping between two triangles Idea: find an affine that transforms ABC to A’B’C’ 6 unknowns, 6 equations A C B A’ C’ B’

43. Barycentric coordinates Idea: represent P using A1,A2,A3

44. Barycentric coordinates Idea: represent P using A1,A2,A3

45. Barycentric coordinates Idea: represent P using A1,A2,A3

46. Non-parametric image warping P’ Barycentric coordinate P Turns out to be equivalent to affine transform

47. Non-parametric image warping Gaussian

48. Demo http://www.colonize.com/warp/warp04-2.php Warping is a useful operation for mosaics, video matching, view interpolation and so on.

49. Image morphing

50. The goal is to synthesize a fluid transformation from one image to another. image #1 image #2 dissolving Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects. (1-t) ∙ Image1 + t ∙ Image2

51. Image morphing Why ghosting? Morphing = warping + cross-dissolving shape (geometric) color (photometric)

52. morphing cross-dissolving Image morphing image #1image #2 warp

53. Morphing sequence

54. Image morphing create a morphing sequence: for each time t 1.Create an intermediate warping field (by interpolation) t=0 t=1t=0.33 A(0) A(1) A(0.33) B(0) B(1) B(0.33) C(0) C(1) C(0.33)

55. Image morphing create a morphing sequence: for each time t 1.Create an intermediate warping field (by interpolation) 2.Warp both images towards it t=0 t=1t=0.33 A(0) A(1) A(0.33) B(0) B(1) B(0.33) C(0) C(1) C(0.33)

56. Image morphing create a morphing sequence: for each time t 1.Create an intermediate warping field (by interpolation) 2.Warp both images towards it t=0 t=1t=0.33 A(0) A(1) A(0.33) B(0) B(1) B(0.33) C(0) C(1) C(0.33)

57. Image morphing create a morphing sequence: for each time t 1.Create an intermediate warping field (by interpolation) 2.Warp both images towards it 3.Cross-dissolve the colors in the newly warped images t=0 t=1t=0.33 A(0) A(1) A(0.33) B(0) B(1) B(0.33) C(0) C(1) C(0.33)

58. More complex morph Triangular Mesh

59. Results Michael Jackson’s MTV “Black or White” http://www.michaeljackson.com/quicktime_blackorwhite.html

60. Multi-source morphing

62. The average face http://www.uni- regensburg.de/Fakultaeten/phil_Fak_II/Psycholog ie/Psy_II/beautycheck/english/index.htm http://www.uni- regensburg.de/Fakultaeten/phil_Fak_II/Psycholog ie/Psy_II/beautycheck/english/index.htm

63. 3D Face morphing http://www.youtube.com/watch?v=nice6NYb_WA Blanz and Vetter, SIGGRAPH 1998