1. Geometric Active Contours Ron Kimmel www.cs.technion.ac.il/~ron Computer Science Department Technion-Israel Institute of Technology Geometric Image Processing Lab

2. Edge Detection qEdge Detection: uThe process of labeling the locations in the image where the gray level’s “rate of change” is high. nOUTPUT: “edgels” locations, direction, strength qEdge Integration: uThe process of combining “local” and perhaps sparse and non-contiguous “edgel”-data into meaningful, long edge curves (or closed contours) for segmentation nOUTPUT: edges/curves consistent with the local data

3. The Classics qEdge detection: uSobel, Prewitt, Other gradient estimators uMarr Hildreth zero crossings of uHaralick/Canny/Deriche et al. “optimal” directional local max of derivative qEdge Integration: utensor voting (Rom, Medioni, Williams, …) udynamic programming (Shashua & Ullman) ugeneralized “grouping” processes (Lindenbaum et al.)

4. The “New-Wave” qSnakes qGeodesic Active Contours qModel Driven Edge Detection Edge Curves “nice” curves that optimize a functional of g( ), i.e. nice: “regularized”, smooth, fit some prior information Image Edge Indicator Function

5. Geodesic Active Contours qSnakes Terzopoulos-Witkin-Kass 88 uLinear functional efficient implementation unon-geometric depends on parameterization qOpen geometric scaling invariant, Fua-Leclerc 90 qNon-variational geometric flow Caselles et al. 93, Malladi et al. 93 uGeometric, yet does not minimize any functional qGeodesic active contours Caselles-Kimmel-Sapiro 95 uderived from geometric functional unon-linear inefficient implementations: nExplicit Euler schemes limit numerical step for stability qLevel set method Ohta-Jansow-Karasaki 82, Osher-Sethian 88 uautomatically handles contour topology qFast geodesic active contours Goldenberg-Kimmel-Rivlin-Rudzsky 99 uno limitation on the time step uefficient computations in a narrow band

6. Laplacian Active Contours qClosed contours on vector fields uNon-variational models Xu-Prince 98, Paragios et al. 01 uA variational model Vasilevskiy-Siddiqi 01 qLaplacian active contours open/closed/robust Kimmel-Bruckstein 01 Most recent: variational measures for good old operators Kimmel-Bruckstein 03

7. Segmentation

8. qUltrasound images Caselles,Kimmel, Sapiro ICCV’95

9. Segmentation Pintos

10. Woodland Encounter Bev Doolittle 1985 qWith a good prior who needs the data…

11. Segmentation Caselles,Kimmel, Sapiro ICCV’95

12. Prior knowledge…

14. Segmentation

16. Caselles,Kimmel, Sapiro ICCV’95

17. Segmentation qWith a good prior who needs the data…

18. Wrong Prior???

21. Curves in the Plane qC(p)={x(p),y(p)}, p [0,1] y x C(0) C(0.1) C(0.2) C(0.4) C(0.7) C(0.95) C(0.9) C(0.8) p C =tangent

22. Arc-length and Curvature s(p)= | |dp C

23. Calculus of Variations Find C for which is an extremum Euler-Lagrange:

24. Calculus of Variations Important Example  Euler-Lagrange:, setting   Curvature flow

25. Potential Functions (g) x I(x,y)I(x) x g(x) x x g(x,y) Image Edges

26. Snakes & Geodesic Active Contours qSnake model Terzopoulos-Witkin-Kass 88 qEuler Lagrange as a gradient descent qGeodesic active contour model Caselles-Kimmel-Sapiro 95 qEuler Lagrange gradient descent

27. Maupertuis Principle of Least Action Snake = Geodesic active contour up to some, i.e  Snakes depend on parameterization.  Different initial parameterizations yield solutions for different geometric functionals x y p 1 0 Caselles Kimmel Sapiro, IJCV 97

28. Geodesic Active Contours in 1D Geodesic active contours are reparameterization invariant I(x) x g(x) x

29. Geodesic Active Contours in 2D g(x)= G *I s

30. Controlling -max I g Smoothness Cohen Kimmel, IJCV 97

31. Fermat’s Principle In an isotropic medium, the paths taken by light rays are extremal geodesics w.r.t. i.e., Cohen Kimmel, IJCV 97

32. Experiments - Color Segmentation Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

33. Tumor in 3D MRI Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

34. Segmentation in 4D Malladi, Kimmel, Adalsteinsson, Caselles, Sapiro, Sethian SIAM Biomedical workshop 96

35. Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

36. Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

37. Edge Gradient Estimators Xu-Prince 98, Paragios et al. 01, Vasilevskiy-Siddiqi 01, Kimmel-Bruckstein 01

38. Edge Gradient Estimators qWe want a curve with large points and small ‘s so: qConsider the functional qWhere is a scalar function, e.g..

39. The Classic Connection Suppose and we consider a closed contour for C(s). We have and by Green’s Theorem we have

40. qTherefore:  Hence curves that maximize are curves that enclose all regions where is positive! qWe have that the optimal curves in this case are The Zero Crossings of the Laplacian isn’t this familiar? The Classic Connection

41. qIt is pedagogically nice, but the MARR-HILDRETH edge detector is a bit too sensitive. qSo we do not propose a grand return to MH but a rethinking of the functionals used in active contours in view of this. qINDEED, why should we ignore the gradient directions (estimates) and have every edge integrator controlled by the local gradient intensity alone? The Classic Connection

42. Our Proposal qConsider functional of the form qThese functionals yield “regularized” curves that combine the good properties of LZC’s where precise border following is needed, with the good properties of the GAC over noisy regions!

43. Implementation Details qWe implement curve evolution that do gradient descent w.r.t. the functional Here the Euler Lagrange Equations provide the explicit formulae. qFor closed contours we compute the evolved curve via the Osher-Sethian “miracle” numeric level set formulation.

44. Closed contours EL eq. GAC LZC Kimmel-Bruckstein IVCNZ01

45. Closed contours EL eq. GAC LZC LZC+ e GAC Kimmel-Bruckstein IVCNZ01

46. Along the curve b.c. at C(0) and C(L) Open contours Kimmel-Bruckstein IVCNZ01

47. Open contours Kimmel-Bruckstein IVCNZ01

48. Geometric Measures Weighted arc-length Weighted area Alignment Robust - alignment e.g. Variational meaning for Marr-Hildreth edge detector Kimmel-Bruckstein IVCNZ01

49. Geometric Measures Minimal variance Chan-Vese, Mumford-Shah, Max-Lloyd, Threshold,…

50. Geometric Measures Robust minimal deviation

51. Haralick/Canny-like Edge Detector qHaralick suggested as edge detector Laplace Alignment Topological Homogeneity

52. Haralick/Canny Edge Detector qHaralick co-area h Thus, indicates optimal alignment + topological homogeneity

53. Closed Contours & Level Set Method implicit representation of C Then, Geodesic active contour level set formulation Including weighted (by g) area minimization y x C(t) C(t) level set x y

54. Operator Splitting Schemes qAdditive operator splitting (AOS) Lu et al. 90, Weickert, et al. 98 uunconditionally stable for non-linear diffusion qGiven the evolution write qConsider the operator qExplicit scheme u, the time step, is upper bounded for stability

55. LOD: Operator Splitting Schemes qImplicit scheme uinverting large bandwidth matrix qFirst order, semi-implicit, additive operator splitting (AOS), or locally one-dimensional (LOD) multiplicative schemes are stable and efficient given by linear tridiagonal systems of equations that can be solved for by Thomas algorithm AOS:

56. Operator Splitting Schemes qWe used the following relation (AOS) qLocally One-Dimensional scheme (LOD) qDecoupling the axes and the implicit formulation leads to computational efficiency The 1st order `splitting’ idea is based on the operator expansion

57. qThe geodesic active contour model Where I is the image and  the implicit representation of the curve  If  is a distance, then, and the short time evolution is qNote that and thus can be computed once for the whole image Example: Geodesic Active Contour y x C(t) x y Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

58. Example: Geodesic Active Contour  is restricted to be a distance map: Re-initialization by Sethian’s fast marching method every iteration in O(n). Computations are performed in a narrow band around the zero set Multi-scale approach: process a Gaussian pyramid of the image y x C(t) x y

59. Tracking Objects in Movies qMovie volume as a spatial-temporal 3D hybrid space uThe AOS scheme is uEdge function derived by the Beltrami framework Sochen Kimmel Malladi 98 qContour in frame n is the initial condition for frame n+1. x x y y t t

60. Experiments - Curvature Flow

61. Experiments - Curvature Flow CPU Time

62. Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

63. Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

64. Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

65. Information extraction Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

66. Holzman-Gazit, Goldshier, Kimmel 2003 Thin Structures

67. Segmentation in 3D Change in topology Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

68. Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001 Coupled surfaces EL equations

69. Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

70. Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

72. Futurism Recognition from periodic motion Carlo Carra, 1914 Dynamism of a Dog on a Leash Giacomo Balla, 1912 Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002 Eadweard Muybridge, Animals in Motion, 1887

73. Classification (dogs & cats) walkrungallopcat... Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

74. Classification (dogs & cats) Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

76. Classification (people) walkrunrun45 Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

77. Classification (people) Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

79. Conclusions qGeometric-Variational method for segmentation and tracking in finite dimensions based on prior knowledge (more accurately, good initial conditions). qUsing the directional information for edge integration. qGeometric-variational meaning for the Marr-Hildreth and the Haralick (Canny) edge detectors, leads to ways to design improved ones. qEfficient numerical implementation for active contours. qVarious medical and more general applications. www.cs.technion.ac.il/~ron

80. Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

81. Edge Indicator Function for Color qBeltrami framework: Color image = 2D surface in space qThe induced metric tensor for the image surface qEdge indicator = largest eigenvalue of the structure tensor metric. It represents the direction of maximal change in X I Y

82. AOS Proof: The whole low order splitting idea is based on the operator expansion