1. CSCE 641:Computer Graphics Image Warping/Registration Jinxiang Chai

2. Outline Image warping Image Registration

3. Image Warping Image filtering: change range of image g(x) = h(f(x)) f x h f x f x h f x Image warping: change domain of image g(x) = f(h(x))

4. Image Warping Image filtering: change range of image g(x) = h(f(x)) Image warping: change domain of image g(x) = f(h(x)) h fg h f g

5. Image Warping Why? - texture mapping - image processing (rotation, zoom in/out, etc) - image morphing/blending - image editing - image based-modeling & rendering

6. Parametric (global) Warping Examples of image warps: translation rotation aspect affineperspective cylindrical

7. Transformation Function f, g Transform the geometry of an image to a desired geometry

8. Definition: Image Warping Source Image: Image to be used as the reference. The geometry of this image is no changed Target Image: this image is obtained by transforming the reference image. (x,y): coordinates of points in the reference image (u,v): coordinates of points in the target image f,g or F,G: x and y components of a transformation function

9. Definition: Image Warping Control points: Unique points in the reference and target images. The coordinates of corresponding control points in images are used to determine a transformation function. Source ImageTarget Image

10. A Transformation Function Used to computer the corresponding points Source Image S(x,y)Target Image T(u,v) u = f(x,y) v = g(x.y) x = F(u,v) y = G(x.v)

11. Warping Types Simple mappings: - Similarity - Affine mapping - Projective mapping These can be applied globally over a subdivision of the plane: - Piecewise affine over triangulation - Piecewise projective over quadrilaterization - Piecewise bilinear over a rectangular grid Or other, arbitrary functions can be used, e.g. - Bieer-neely warp (popular for morphs) - Store u(x,y) and v(x,y) in large arrays

12. Similarity Transform A combination of 2-D scale, rotation, and translation transformations. Allows a square to be transformed into any rotated rectangle. Angle between lines is preserved 5 degrees of freedom (s x,s y,θ,t x,t y ) Inverse is of same form (is also similarity). Given by inverse of 3X3 matrix above Have the form: In matrix notation:

13. Affine Transform A combination of 2-D scale, rotation, shear, and translation transformations. Allows a square to be distorted into any parallelogram. 6 degrees of freedom (a,b,c,d,e,f) Inverse is of same form (is also affine). Given by inverse of 3X3 matrix above Good when controlling a warp with triangles, since 3 points in 2D determined the 6 degrees of freedom Have the form: In matrix notation:

14. Projective Transform (a.k.a “perspective”) Linear numerator & denominator If g=h=0, then you get affine as a special case Allow a square to be distorted into any quadrilateral 8 degrees of freedom (a-h). We can choose i=1, arbitrarily Inverse is of same form (is also projective). Good when controlling a warp with quadrilaterals, since 4 points in 2D determine the 8 degrees of freedom Have the form: In matrix notation:

15. Image Warping x y u v Given a coordinate transform function f,g or F,G and source image S(x,y), how do we compute a transformed image T(u,v)? S(x,y) T(u,v)

16. Forward Warping x y u v S(x,y) T(u,v) Forward warping algorithm: for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v)

17. Forward Warping x y u v S(x,y) T(u,v) Forward warping algorithm: for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) - Any problems for forward warping?

18. Forward Warping x y u v S(x,y) T(u,v) Q: What if the transformed pixel located between pixels? A: Distribute color among neighboring pixels - known as “splatting”

19. Forward Warping Iterate over source, sending pixels to destination Some source pixel map to multiple dest. pixels Some dest. pixels may have no corresponding source Holes in reconstruction Must splat etc. for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) x y u v

20. Forward Warping Iterate over source, sending pixels to destination Some source pixel map to multiple dest. pixels Some dest. pixels may have no corresponding source Holes in reconstruction Must splat etc. for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) x y u v - How to remove the holes?

21. Inverse Warping x y u v S(x,y) T(u,v) Inverse warping algorithm: for v = v min to v max for u = u min to u max x = F(u,v); y = G(u,v) copy pixel at source S(x,y) to T(u,v)

22. Inverse Warping x y u v S(x,y)T(u,v) Q: What if pixel comes from “between” two pixels? A: Interpolate color values from neighboring pixels

23. Inverse Warping Iterate over dest., finding pixels from source Non-integer evaluation source image, resample May oversample source But no holes Simpler, better than forward mapping for v = v min to v max for u = u min to u max x = F(u,v); y = G(u,v) copy pixel at source S(x,y) to T(u,v) x y u v

24. Resampling x y u v This is a 2D signal reconstruction problem!

25. Resampling Filter

26. Review: Signal Reconstruction in Freq. Domain T2T … -2T-T … 0 x f s (x) f(x) x F s (u) u -f max f max F(u) u -f max f max Inverse Fourier transform Fourier transform

27. Review: Signal Reconstruction in Spatial Domain T2T … -2T-T … 0 x x f s (x) sinc(x)

28. Resampling Compute weighted sum of pixel neighborhood - Weights are normalized values of kernel function - Equivalent to convolution at samples with kernel - Find good filters using ideas of previous lectures x y u v

29. Point Sampling Nearest neighbor x y u v - Copies the color of the pixel with the closest integer coordinate - A fast and efficient way to process textures if the size of the target is similar to the size of the reference - Otherwise, the result can be a chunky, aliased, or blurred image.

30. Bilinear Filter Weighted sum of four neighboring pixels x y u v

31. Bilinear Filter Sampling at S(x,y): (i+1,j) (i,j) (i,j+1) (i+1,j+1) S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) u v y x

32. Bilinear Filter Sampling at S(x,y): (i+1,j) (i,j) (i,j+1) (i+1,j+1) S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) S i = S(i,j) + a*(S(i,j+1)-S(i)) S j = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j)) S(x,y) = S i +b*(S j -S i) To optimize the above, do the following u v y x

33. Bilinear Filter (i+1,j) (i,j) (i,j+1) (i+1,j+1) y x

34. Anisotropic Filter Anisotropic means "non-uniform shape" - Used in texture mapping - A circle in screen space corresponds to an ellipse in texture space - An Isotropic filter in screen space means an anisotropic filter - Changing viewpoint results in different filters - Calculated at run time - Many calculations need to be done to draw a single pixel Screen space Texture space

35. Comparison of Resampling Filters

36. Inverse Warping and Resampling Inverse warping algorithm: for v = v min to v max for u = u min to u max float x = F(u,v); float y = G(u,v); T(u,v) = resample_souce(x,y,w); x y u v (u,v) (x,y)

37. Outline Image warping Image Registration

38. Image warping: given h and f, compute g g(x) = f(h(x)) h f g? Image registration: given f and g, compute h h? f g

39. Why Image Registration? Lots of uses –Correct for camera jitter (stabilization) –Align images (mosaics) –View morphing –Special effects –Image based modeling/rendering –Etc. [Seitz 96]

40. Image Registration How do we align two images automatically? Two broad approaches: –Feature-based alignment Find a few matching features in both images compute alignment –Direct (pixel-based) alignment Search for alignment where most pixels agree

41. Outline Image registration - feature-based approach - pixel-based approach

42. Outline Image registration - feature-based approach - pixel-based approach

43. Feature-based Alignment 1.Find a few important features (aka Interest Points) 2.Match them across two images 3.Compute image transformation function h

44. Feature-based Alignment 1.Find a few important features (aka Interest Points) 2.Match them across two images 3.Compute image transformation function h How to choose features –Choose only the points (“features”) that are salient, i.e. likely to be there in the other image –How to find these features?

45. Feature-based Alignment 1.Find a few important features (aka Interest Points) 2.Match them across two images 3.Compute image transformation function h How to choose features –Choose only the points (“features”) that are salient, i.e. likely to be there in the other image –How to find these features? windows where has two large eigenvalues Harris Corner detector Better features (SIFT features)

46. Feature Detection -Two images taken at the same place with different angles - Projective transformation H 3X3

47. Feature Matching ? -Two images taken at the same place with different angles - Projective transformation H 3X3

48. Feature Matching ? -Two images taken at the same place with different angles - Projective transformation H 3X3 How do we match features across images? Any criterion?

49. Feature Matching ? -Two images taken at the same place with different angles - Projective transformation H 3X3 How do we match features across images? Any criterion?

50. Feature Matching Intensity/Color similarity –The intensity of pixels around the corresponding features should have similar intensity

51. Feature Matching Intensity/Color similarity –The intensity of pixels around the corresponding features should have similar intensity –Sum of squared differences (SSD), Normalized cross-correlationSum of squared differences (SSD) Normalized cross-correlation

52. Feature Matching Feature similarity (Intensity or SIFT signature) –The intensity of pixels around the corresponding features should have similar intensity –Cross-correlation, SSDCross-correlationSSD Distance constraint –The displacement of features should be smaller than a given threshold

53. Feature-space Outlier Rejection bad Good

54. Feature-space Outlier Rejection Can we now compute H 3X3 from the blue points?

55. Feature-space Outlier Rejection Can we now compute H 3X3 from the blue points? –No! Still too many outliers…

56. Feature-space Outlier Rejection Can we now compute H 3X3 from the blue points? –No! Still too many outliers… –What can we do?

57. Feature-space Outlier Rejection Can we now compute H 3X3 from the blue points? –No! Still too many outliers… –What can we do? Robust estimation!

58. Robust EstimationRobust Estimation: A Toy Example How to fit a line based on a set of 2D points?

59. Robust EstimationRobust Estimation: A Toy Example How to fit a line based on a set of 2D points? RANSACRANSAC: an iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliersoutliers

60. RANSACRANSAC: RANdom SAmple Consensus Objective Robust fit of model to data set S which contains outliers Algorithm (i)Randomly select a sample of s data points from S and instantiate the model from this subset. (ii)Determine the set of data points S i which are within a distance threshold t of the model. The set S i is the consensus set of samples and defines the inliers of S. (iii)If the subset of S i is greater than some threshold T, re- estimate the model using all the points in S i and terminate (iv)If the size of S i is less than T, select a new subset and repeat the above. (v)After N trials the largest consensus set S i is selected, and the model is re-estimated using all the points in the subset S i

61. RANSAC Repeat M times: –Sample minimal number of matches to estimate two view relation (affine, perspective, etc). –Calculate number of inliers or posterior likelihood for relation. –Choose relation to maximize number of inliers.

62. RANSAC Line Fitting Example Task: Estimate best line

63. RANSAC Line Fitting Example Sample two points

64. RANSAC Line Fitting Example Fit Line

65. RANSAC Line Fitting Example Total number of points within a threshold of line.

66. RANSAC Line Fitting Example Repeat, until get a good result

67. RANSAC Line Fitting Example Repeat, until get a good result

68. RANSAC Line Fitting Example Repeat, until get a good result

69. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 proportion of outliers e s5%10%20%25%30%40%50% 2235671117 33479111935 435913173472 54612172657146 64716243797293 748203354163588 8592644782721177

70. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 proportion of outliers e s5%10%20%25%30%40%50% 2235671117 33479111935 435913173472 54612172657146 64716243797293 748203354163588 8592644782721177 Affine transform

71. How Many Samples? Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99 proportion of outliers e s5%10%20%25%30%40%50% 2235671117 33479111935 435913173472 54612172657146 64716243797293 748203354163588 8592644782721177 Projective transform

72. RANSAC for Estimating Projective Transformation RANSAC loop: 1.Select four feature pairs (at random) 2.Compute the transformation matrix H (exact) 3.Compute inliers where SSD(p i ’, H p i ) < ε 4.Keep largest set of inliers 5.Re-compute least-squares H estimate on all of the inliers

73. RANSAC

74. Feature-based Registration Works for small or large motion Model the motion within a patch or whole image using a parametric transformation model

75. Feature-based Registration Works for small or large motion Model the motion within a patch or whole image using a parametric transformation model How to deal with motions that cannot be described by a small number of parameters?

76. Outline Image registration - feature-based approach - pixel-based approach

77. Next Lecture More on image registration Image mosaicing