1. Computer Graphics: Image Warping/Morphing
Jinxiang Chai

2. Outline
Image warping
Image morphing

3. Required Readings
Section 3.6 (Szeliski book)
 Image morphing paper

4. Image Warping Image filtering: change range of image
g(x) = h(f(x)) f x f x h
Image warping: change domain of image
g(x) = f(h(x)) f x f x h

5. Image Warping Image filtering: change range of image
g(x) = h(f(x)) f g h
Image warping: change domain of image
g(x) = f(h(x)) f g h

6. Image Warping Why? - texture mapping
- image processing (rotation, zoom in/out, etc)
- image morphing/blending
- image editing
- image based-modeling & rendering

7. Parametric (global) Warping
Examples of image warps:
aspect
translation
rotation
affine
perspective
cylindrical

8. Transformation Function
f, g
Transform the geometry of an image to a desired geometry

9. Definition: Image Warping
Source Image: Image to be used as the reference. The geometry of this image is no changed
Target Image: this image is obtained by transforming the reference image.
(x,y): coordinates of points in the reference image
(u,v): coordinates of points in the target image
f,g or F,G: x and y components of a transformation function

10. Definition: Image Warping
Control points: Unique points in the reference and target images. The coordinates of corresponding control points in images are used to determine a transformation function.
Source Image
Target Image

11. A Transformation Function
Used to compute the corresponding points
u = f(x,y)
v = g(x.y)
x = F(u,v)
y = G(x.v)
Source Image S(x,y)
Target Image T(u,v)

12. Warping Types Simple mappings:
- Similarity
- Affine mapping
- Projective mapping
These can be applied globally over a subdivision of the plane:
- Piecewise affine over triangulation
- Piecewise projective over quadrilaterization
- Piecewise bilinear over a rectangular grid
Or other, arbitrary functions can be used, e.g.
- Bieer-neely warp (popular for morphs)
- Store u(x,y) and v(x,y) in large arrays

13. Similarity Transform Have the form: In matrix notation:
A combination of 2-D scale, rotation, and translation transformations.
Allows a square to be transformed into any rotated rectangle.
Angle between lines is preserved
5 degrees of freedom (sx,sy,θ,tx,ty)
Inverse is of same form (is also similarity). Given by inverse of 3X3 matrix above

14. Affine Transform
Have the form:
In matrix notation:
A combination of 2-D scale, rotation, shear, and translation transformations.
Allows a square to be distorted into any parallelogram.
6 degrees of freedom (a,b,c,d,e,f)
Inverse is of same form (is also affine). Given by inverse of 3X3 matrix above
Good when controlling a warp with triangles, since 3 points in 2D determined the 6 degrees of freedom

15. Projective Transform (a.k.a “perspective”)
Have the form:
In matrix notation:
Linear numerator & denominator
If g=h=0, then you get affine as a special case
Allow a square to be distorted into any quadrilateral
8 degrees of freedom (a-h). We can choose i=1, arbitrarily
Inverse is of same form (is also projective).
Good when controlling a warp with quadrilaterals, since 4 points in 2D determine the 8 degrees of freedom

16. Image Warping x u y v T(u,v) S(x,y)
Given a coordinate transform function f,g or F,G and source image S(x,y), how do we compute a transformed image T(u,v)?

17. Forward Warping x u v y T(u,v) S(x,y) Forward warping algorithm:
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)

18. Forward Warping x u v y T(u,v) S(x,y) Forward warping algorithm:
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)
Any problems for forward warping?

19. Forward Warping x u y v T(u,v) S(x,y)
Q: What if the transformed pixel located between pixels?

20. Forward Warping x u y v T(u,v) S(x,y)
Q: What if the transformed pixel located between pixels?
A: Distribute color among neighboring pixels
- known as “splatting” 20

21. Forward Warping Iterate over source, sending pixels to destination
Some source pixels maps to the same dest. pixel
Some dest. pixels may have no corresponding source
Holes in reconstruction
Must splat etc. x u y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)

22. Forward Warping Iterate over source, sending pixels to destination
Some source pixels map to the same dest. pixel
Some dest. pixels may have no corresponding source
Holes in reconstruction
Must splat etc.
- How to remove the holes? x u y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)

23. Forward Warping Iterate over source, sending pixels to destination
Some source pixels map to the same dest. pixel
Some dest. pixels may have no corresponding source
Holes in reconstruction
Must splat etc.
- How to remove the holes? x u y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v) 23

24. Inverse Warping x u y v T(u,v) S(x,y) Inverse warping algorithm:
for v = vmin to vmax
for u = umin to umax
x = F(u,v); y = G(u,v)
copy pixel at source S(x,y) to T(u,v)

25. Inverse Warping x u y v S(x,y) T(u,v)
Q: What if pixel comes from “between” two pixels?
A: Interpolate color values from neighboring pixels

26. Inverse Warping Iterate over dest., finding pixels from source
Non-integer evaluation source image, resample
May oversample source
But no holes
Simpler, better than forward mapping x u
for v = vmin to vmax
for u = umin to umax
x = F(u,v); y = G(u,v)
copy pixel at source S(x,y) to T(u,v) y v

27. Resampling Filter

28. Resampling x u y v
This is a 2D signal reconstruction problem!

29. Resampling Compute weighted sum of pixel neighborhood
- Weights are normalized values of kernel function
- Equivalent to convolution at samples with kernel
- Find good filters using ideas of previous lectures x u y v

30. Point Sampling Nearest neighbor
Copies the color of the pixel with the closest integer coordinate
A fast and efficient way to process textures if the size of the
target is similar to the size of the reference
Otherwise, the result can be a chunky, aliased, or blurred image. x u y v

31. Bilinear Filter
Weighted sum of four neighboring pixels x u y v

32. Bilinear Filter Sampling at S(x,y): y (i,j) (i,j+1) u x v (i+1,j)
S(x,y) = (1-a)*(1-b)*S(i,j) a*(1-b)*S(i+1,j)
+ (1-a)*b*S(i,j+1) + a*b*S(i+1,j+1)

33. Bilinear Filter Sampling at S(x,y): y (i,j) (i,j+1) u x v (i+1,j)
S(x,y) = a*b*S(i,j) a*(1-b)*S(i+1,j)
+ (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1)
To optimize the above, do the following
Si = S(i,j) + a*(S(i,j+1)-S(i))
Sj = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j))
S(x,y) = Si+b*(Sj-Si)

34. Bilinear Filter y
(i,j)
(i,j+1) x
(i+1,j)
(i+1,j+1)

35. Inverse Warping and Resamplingx u y v
(x,y)
(u,v)
Inverse warping algorithm:
for v = vmin to vmax
for u = umin to umax
float x = F(u,v); float y = G(u,v);
T(u,v) = resample_souce(x,y,w);

36. Outline
Image warping
Image morphing

37. Morphing = Object Averaging
The aim is to find “an average” between two objects
Not an average of two images of objects…
…but an image of the average object!
How can we make a smooth transition in time?
Do a “weighted average” over time t
How do we know what the average object looks like?
We haven’t a clue!
But we can often fake something reasonable
Usually required user/artist input

38. Averaging Points P and Q can be anything: What’s the average
v = Q - P
What’s the average
of P and Q? P
P + 1.5v
= P + 1.5(Q – P)
= -0.5P Q
(extrapolation)
P + 0.5v
= P + 0.5(Q – P)
= 0.5P Q
Linear Interpolation
(Affine Combination):
New point aP + bQ,
defined only when a+b = 1
So aP+bQ = aP+(1-a)Q
P and Q can be anything:
points on a plane (2D) or in space (3D)
Colors in RGB or HSV (3D)
Whole images (m-by-n D)… etc.

39. Idea #1: Cross-Dissolve
Interpolate whole images:
- Imagehalfway = (1-t)*Image1 + t*image2
This is called cross-dissolve in film industry
But what if the images are not aligned?

40. Dog Averaging
What to do?
Cross-dissolve doesn’t work
Any ideas?

41. Dog Averaging What to do? Cross-dissolve doesn’t work Any ideas?
Feature matching!
Nose to nose, tail to tail, etc.
This is a local (non-parametric) warp

42. Idea #2: Local Warping Morphing procedure: for every t,
1. Find the average shape (the “mean dog”)
- local warping
2. Find the average color
- Cross-dissolve the warped images

43. Local (non-parametric) Image Warping
Need to specify a more detailed warp function
Global warps were functions of a few (2,4,8) parameters
Non-parametric warps u(x,y) and v(x,y) can be defined independently for every single location x,y!
Once we know vector field u,v we can easily warp each pixel (use backward warping with interpolation)
Will it work for these dogs?
Probably not… Need user control.

44. Warp Specification How can we specify the warp?
Specify corresponding points
interpolate to a complete warping function
How do we do it?

45. Finding Transformation Function
Generally this is a 2D scattered data interpolation problem
- radial basis function, etc

46. Finding Transformation Function
Generally this is a 2D scattered data interpolation problem
- radial basis function, etc
Piecewise affine transformation

47. Finding Transformation Function
Generally this is a 2D scattered data interpolation problem
- radial basis function, etc
Piecewise affine transformation

48. Finding Transformation Function
Generally this is a 2D scattered data interpolation problem
- radial basis function, etc
Piecewise affine transformation

49. Triangular Mesh Input correspondences at key feature points
Define a triangular mesh over the points (Delaunay triangulation)
Same mesh in both images!
Now we have triangle-to-triangle correspondences
Warp each triangle separately from source to destination
How do we warp a triangle?
3 points = affine warp!
Just like texture mapping

50. Warp Specification How can we specify the warp?
Specify corresponding vectors
interpolate to a complete warping function
The Beier & Neely Algorithm

51. Beier&Neely (SIGGRAPH 1992)
Single line-pair PQ to P’Q’:

52. Algorithm (single line-pair)
For each X in the destination image:
Find the corresponding u,v
Find X’ in the source image for that u,v
destinationImage(X) = sourceImage(X’)
Examples:

53. Multiple Lines Length: length of the line segment,
Dist: distance to line segment
a, p, b: constants specified by the user

54. Resulting Warp

55. Full Algorithm

56. Results

57. Morph Sequences

58. Image Morphing
We know how to warp one image into the other, but how do we create a morphing sequence?
Create an intermediate warping field (by interpolation)
Warp both images towards it
Cross-dissolve the colors in the newly warped images

59. Morphing Video
Click here!

60. Summary
Image warping
Image morphing