1. Fitting a transformation: feature-based alignment Thursday, September 24 th 2015 Devi Parikh Virginia Tech 1 Slide credit: Kristen Grauman Disclaimer: Many slides have been borrowed from Kristen Grauman, who may have borrowed some of them from others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits are inaccurate.

2. Announcements Project proposals –Due Tuesday –Teams of > 2 –Look at class webpage for guidelines PS2 out –Due October 5 th PS1 graded –Grades will be released soon 2 Slide credit: Adapted by Devi Parikh from Kristen Grauman

3. Given: initial contour (model) near desired object a.k.a. active contours, snakes Figure credit: Yuri Boykov Goal: evolve the contour to fit exact object boundary [Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987] Main idea: elastic band is iteratively adjusted so as to be near image positions with high gradients, and satisfy shape “preferences” or contour priors Last time: Deformable contours 3 Slide credit: Kristen Grauman

4. Last time: Deformable contours Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG Kristen Grauman 4

5. Recap: deformable contour A simple elastic snake is defined by: –A set of n points, –An internal energy term (tension, bending, plus optional shape prior) –An external energy term (gradient-based) To use to segment an object: –Initialize in the vicinity of the object –Modify the points to minimize the total energy Kristen Grauman 5

6. Main idea: determine optimal position (state) of predecessor, for each possible position of self. Then backtrack from best state for last vertex. states 1 2 … m vertices Complexity: vs. brute force search ____? Viterbi algorithm Example adapted from Y. Boykov 6 Slide credit: Kristen Grauman

7. With this form of the energy function, we can minimize using dynamic programming, with the Viterbi algorithm. Iterate until optimal position for each point is the center of the box, i.e., the snake is optimal in the local search space constrained by boxes. [Amini, Weymouth, Jain, 1990] Fig from Y. Boykov Energy minimization: dynamic programming 7 Slide credit: Kristen Grauman

8. DP can be applied to optimize an open ended snake For a closed snake, a “loop” is introduced into the total energy. Work around: 1)Fix v 1 and solve for rest. 2)Fix an intermediate node at its position found in (1), solve for rest. Energy minimization: dynamic programming 8 Slide credit: Kristen Grauman

9. Aspects we need to consider Representation of the contours Defining the energy functions –External –Internal Minimizing the energy function Extensions: –Tracking –Interactive segmentation 9 Slide credit: Kristen Grauman

10. Tracking via deformable contours 1.Use final contour/model extracted at frame t as an initial solution for frame t+1 2.Evolve initial contour to fit exact object boundary at frame t+1 3.Repeat, initializing with most recent frame. Tracking Heart Ventricles (multiple frames) Kristen Grauman 10

11. Visual Dynamics GroupVisual Dynamics Group, Dept. Engineering Science, University of Oxford. Traffic monitoring Human-computer interaction Animation Surveillance Computer assisted diagnosis in medical imaging Applications: Tracking via deformable contours Kristen Grauman 11

12. 3D active contours Jörgen Ahlberg http://www.cvl.isy.liu.se/ScOut/Masters/Papers/Ex1708.pdf Kristen Grauman 12

13. May over-smooth the boundary Cannot follow topological changes of objects Limitations 13 Slide credit: Kristen Grauman

14. Limitations External energy: snake does not really “see” object boundaries in the image unless it gets very close to it. image gradients are large only directly on the boundary 14 Slide credit: Kristen Grauman

15. Distance transform External image can instead be taken from the distance transform of the edge image. original -gradient distance transform edges Value at (x,y) tells how far that position is from the nearest edge point (or other binary mage structure) >> help bwdist Kristen Grauman 15

16. Deformable contours: pros and cons Pros: Useful to track and fit non-rigid shapes Contour remains connected Possible to fill in “subjective” contours Flexibility in how energy function is defined, weighted. Cons: Must have decent initialization near true boundary, may get stuck in local minimum Parameters of energy function must be set well based on prior information Kristen Grauman 16

17. Summary Deformable shapes and active contours are useful for –Segmentation: fit or “snap” to boundary in image –Tracking: previous frame’s estimate serves to initialize the next Fitting active contours: –Define terms to encourage certain shapes, smoothness, low curvature, push/pulls, … –Use weights to control relative influence of each component cost –Can optimize 2d snakes with Viterbi algorithm. Image structure (esp. gradients) can act as attraction force for interactive segmentation methods. Kristen Grauman 17

18. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 18 Slide credit: Kristen Grauman

19. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 19 Slide credit: Kristen Grauman

20. Interactive forces How can we implement such an interactive force with deformable contours? Kristen Grauman 20

21. Interactive forces An energy function can be altered online based on user input – use the cursor to push or pull the initial snake away from a point. Modify external energy term to include a term such that Nearby points get pushed hardest Adapted by Devi Parikh from Kristen Grauman 21

22. Intelligent scissors [Mortensen & Barrett, SIGGRAPH 1995, CVPR 1999] Another form of interactive segmentation: Compute optimal paths from every point to the seed based on edge-related costs. 22 Adapted by Devi Parikh from Kristen Grauman Demo: http://www.luberth.com/java/scissors/

23. http://rivit.cs.byu.edu/Eric/Eric.html Intelligent scissors 23 Slide credit: Kristen Grauman

24. http://rivit.cs.byu.edu/Eric/Eric.html Intelligent scissors 24 Slide credit: Kristen Grauman

25. Beyond boundary snapping… Another form of interactive guidance: specify regions Usually taken to suggest foreground/background color distributions Boykov and Jolly (2001) User Input Result How to use this information? Kristen Grauman 25

26. q Recall: Images as graphs Fully-connected graph node for every pixel link between every pair of pixels, p,q similarity w pq for each link »similarity is inversely proportional to difference in color and position p w pq w Steve Seitz 26

27. Recall: Segmentation by Graph Cuts Break graph into segments Delete links that cross between segments Easiest to break links that have low similarity –similar pixels should be in the same segments –dissimilar pixels should be in different segments w ABC Steve Seitz 27

28. Link Cut set of links whose removal makes a graph disconnected cost of a cut: A B Find minimum cut gives you a segmentation fast algorithms exist for doing this Source: Steve Seitz Recall: Segmentation by Graph Cuts 28

29. Adding hard constraints: Add two additional nodes, object and background “terminals” Link each pixel To both terminals To its neighboring pixels Graph cuts for interactive segmentation Yuri Boykov 29

30. Graph cuts for interactive segmentation Adding hard constraints: Let the edge weight to object or background terminal reflect similarity to the respective seed pixels. Yuri Boykov 30

31. Yuri Boykov Graph cuts for interactive segmentation Boykov and Jolly (2001) 31

32. Intelligent Scissors Mortensen and Barrett (1995) GrabCut Rother et al. (2004) Graph Cuts Boykov and Jolly (2001) Graph cuts for interactive segmentation Another interaction modality: specify bounding box 32 Slide credit: Kristen Grauman

33. “Grab Cut” Loosely specify foreground region Iterated graph cut Rother et al (2004) User Initialisation K-means for learning colour distributions Graph cuts to infer the segmentation ? User initialization 33

34. “Grab Cut” Loosely specify foreground region Iterated graph cut Rother et al (2004) Gaussian Mixture Model (typically 5-8 components) Foreground & Background Background Foreground Background G R G R Iterated graph cut 34

35. “Grab Cut” Rother et al (2004) 35

36. Topics overview Features & filters Grouping & fitting –Segmentation and clustering –Hough transform –Deformable contours –Alignment and 2D image transformations Multiple views and motion Recognition Video processing 36 Slide credit: Kristen Grauman

37. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 37 Slide credit: Kristen Grauman

38. Motivation: Recognition Figures from David Lowe 38 Slide credit: Kristen Grauman

39. Motivation: medical image registration 39 Slide credit: Kristen Grauman

40. Motivation: mosaics Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/ (In detail next week) 40 Slide credit: Kristen Grauman

41. Alignment problem We have previously considered how to fit a model to image evidence –e.g., a line to edge points, or a snake to a deforming contour In alignment, we will fit the parameters of some transformation according to a set of matching feature pairs (“correspondences”). T xixi xixi ' 41 Slide credit: Kristen Grauman

42. Parametric (global) warping Examples of parametric warps: translation rotation aspect affine perspective Source: Alyosha Efros 42

43. Parametric (global) warping Transformation T is a coordinate-changing machine: p’ = T(p) What does it mean that T is global? Is the same for any point p can be described by just a few numbers (parameters) Let’s represent T as a matrix: p’ = Mp T p = (x,y)p’ = (x’,y’) Source: Alyosha Efros 43

44. Scaling Scaling a coordinate means multiplying each of its components by a scalar Uniform scaling means this scalar is the same for all components:  2 2 Source: Alyosha Efros 44

45. Non-uniform scaling: different scalars per component: Scaling X  2, Y  0.5 Source: Alyosha Efros 45

46. Scaling Scaling operation: Or, in matrix form: scaling matrix S Source: Alyosha Efros 46

47. What transformations can be represented with a 2x2 matrix? 2D Rotate around (0,0)? 2D Shear? Source: Alyosha Efros 2D Scaling? 47

48. What transformations can be represented with a 2x2 matrix? Source: Alyosha Efros 2D Mirror about Y axis? 2D Mirror over (0,0)? 2D Translation? NO! 48

49. 2D Linear Transformations Only linear 2D transformations can be represented with a 2x2 matrix. Linear transformations are combinations of … Scale, Rotation, Shear, and Mirror Source: Alyosha Efros 49

50. Homogeneous coordinates Converting from homogeneous coordinates homogeneous image coordinates To convert to homogeneous coordinates: 50 Slide credit: Kristen Grauman

51. Homogeneous Coordinates Q: How can we represent 2d translation as a 3x3 matrix using homogeneous coordinates? A: Using the rightmost column: Source: Alyosha Efros 51

52. Translation t x = 2 t y = 1 Homogeneous Coordinates Source: Alyosha Efros 52

53. Basic 2D Transformations Basic 2D transformations as 3x3 matrices Translate RotateShear Scale Source: Alyosha Efros 53

54. 2D Affine Transformations Affine transformations are combinations of … Linear transformations, and Translations Parallel lines remain parallel 54 Slide credit: Kristen Grauman

55. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 55 Slide credit: Kristen Grauman

56. Alignment problem We have previously considered how to fit a model to image evidence –e.g., a line to edge points, or a snake to a deforming contour In alignment, we will fit the parameters of some transformation according to a set of matching feature pairs (“correspondences”). T xixi xixi ' Kristen Grauman 56

57. Image alignment Two broad approaches: –Direct (pixel-based) alignment Search for alignment where most pixels agree –Feature-based alignment Search for alignment where extracted features agree Can be verified using pixel-based alignmen t 57 Slide credit: Kristen Grauman

58. Fitting an affine transformation Assuming we know the correspondences, how do we get the transformation? 58 Slide credit: Kristen Grauman

59. An aside: Least Squares Example Say we have a set of data points (X1,X1’), (X2,X2’), (X3,X3’), etc. (e.g. person’s height vs. weight) We want a nice compact formula (a line) to predict X’s from Xs: Xa + b = X’ We want to find a and b How many (X,X’) pairs do we need? What if the data is noisy? Ax=B overconstrained Source: Alyosha Efros 59

60. Fitting an affine transformation Assuming we know the correspondences, how do we get the transformation? 60 Slide credit: Kristen Grauman

61. Fitting an affine transformation How many matches (correspondence pairs) do we need to solve for the transformation parameters? Once we have solved for the parameters, how do we compute the coordinates of the corresponding point for ? Where do the matches come from? Kristen Grauman 61

62. What are the correspondences? ? Compare content in local patches, find best matches. e.g., simplest approach: scan with template, and compute SSD or correlation between list of pixel intensities in the patch Later in the course: how to select regions according to the geometric changes, and more robust descriptors. Kristen Grauman 62

63. Fitting an affine transformation Figures from David Lowe, ICCV 1999 Affine model approximates perspective projection of planar objects. 63 Slide credit: Kristen Grauman

64. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 64 Slide credit: Kristen Grauman

65. Outliers Outliers can hurt the quality of our parameter estimates, e.g., –an erroneous pair of matching points from two images –an edge point that is noise, or doesn’t belong to the line we are fitting. Kristen Grauman 65

66. Outliers affect least squares fit 66 Slide credit: Kristen Grauman

67. Outliers affect least squares fit 67 Slide credit: Kristen Grauman

68. RANSAC RANdom Sample Consensus Approach: we want to avoid the impact of outliers, so let’s look for “inliers”, and use those only. Intuition: if an outlier is chosen to compute the current fit, then the resulting line won’t have much support from rest of the points. 68 Slide credit: Kristen Grauman

69. RANSAC: General form RANSAC loop: 1.Randomly select a seed group of points on which to base transformation estimate (e.g., a group of matches) 2.Compute transformation from seed group 3.Find inliers to this transformation 4.If the number of inliers is sufficiently large, re-compute estimate of transformation on all of the inliers Keep the transformation with the largest number of inliers 69 Slide credit: Kristen Grauman

70. RANSAC for line fitting example Source: R. Raguram Lana Lazebnik 70

71. RANSAC for line fitting example Least-squares fit Source: R. Raguram Lana Lazebnik 71

72. RANSAC for line fitting example 1.Randomly select minimal subset of points Source: R. Raguram Lana Lazebnik 72

73. RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model Source: R. Raguram Lana Lazebnik 73

74. RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function Source: R. Raguram Lana Lazebnik 74

75. RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function 4.Select points consistent with model Source: R. Raguram Lana Lazebnik 75

76. RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function 4.Select points consistent with model 5.Repeat hypothesize-and- verify loop Source: R. Raguram Lana Lazebnik 76

77. 77 RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function 4.Select points consistent with model 5.Repeat hypothesize-and- verify loop Source: R. Raguram Lana Lazebnik

78. 78 RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function 4.Select points consistent with model 5.Repeat hypothesize-and- verify loop Uncontaminated sample Source: R. Raguram Lana Lazebnik

79. RANSAC for line fitting example 1.Randomly select minimal subset of points 2.Hypothesize a model 3.Compute error function 4.Select points consistent with model 5.Repeat hypothesize-and- verify loop Source: R. Raguram Lana Lazebnik 79

80. RANSAC for line fitting Repeat N times: Draw s points uniformly at random Fit line to these s points Find inliers to this line among the remaining points (i.e., points whose distance from the line is less than t) If there are d or more inliers, accept the line and refit using all inliers Lana Lazebnik 80

81. RANSAC pros and cons Pros Simple and general Applicable to many different problems Often works well in practice Cons Lots of parameters to tune Doesn’t work well for low inlier ratios (too many iterations, or can fail completely) Can’t always get a good initialization of the model based on the minimum number of samples Lana Lazebnik 81

82. Today Interactive segmentation Feature-based alignment –2D transformations –Affine fit –RANSAC 82 Slide credit: Kristen Grauman

83. Coming up: alignment and image stitching 83 Slide credit: Kristen Grauman

84. Questions? See you Tuesday! 84 Slide credit: Devi Parikh