Geometric Active Contours Ron Kimmel Computer Science Department Technion-Israel Institute of Technology Geometric Image Processing Lab

Edge Detection qEdge Detection: uThe process of labeling the locations in the image where the gray level’s “rate of change” is high. nOUTPUT: “edgels” locations, direction, strength qEdge Integration: uThe process of combining “local” and perhaps sparse and non-contiguous “edgel”-data into meaningful, long edge curves (or closed contours) for segmentation nOUTPUT: edges/curves consistent with the local data

The Classics qEdge detection: uSobel, Prewitt, Other gradient estimators uMarr Hildreth zero crossings of uHaralick/Canny/Deriche et al. “optimal” directional local max of derivative qEdge Integration: utensor voting (Rom, Medioni, Williams, …) udynamic programming (Shashua & Ullman) ugeneralized “grouping” processes (Lindenbaum et al.)

The “New-Wave” qSnakes qGeodesic Active Contours qModel Driven Edge Detection Edge Curves “nice” curves that optimize a functional of g( ), i.e. nice: “regularized”, smooth, fit some prior information Image Edge Indicator Function

Geodesic Active Contours qSnakes Terzopoulos-Witkin-Kass 88 uLinear functional efficient implementation unon-geometric depends on parameterization qOpen geometric scaling invariant, Fua-Leclerc 90 qNon-variational geometric flow Caselles et al. 93, Malladi et al. 93 uGeometric, yet does not minimize any functional qGeodesic active contours Caselles-Kimmel-Sapiro 95 uderived from geometric functional unon-linear inefficient implementations: nExplicit Euler schemes limit numerical step for stability qLevel set method Ohta-Jansow-Karasaki 82, Osher-Sethian 88 uautomatically handles contour topology qFast geodesic active contours Goldenberg-Kimmel-Rivlin-Rudzsky 99 uno limitation on the time step uefficient computations in a narrow band

Laplacian Active Contours qClosed contours on vector fields uNon-variational models Xu-Prince 98, Paragios et al. 01 uA variational model Vasilevskiy-Siddiqi 01 qLaplacian active contours open/closed/robust Kimmel-Bruckstein 01 Most recent: variational measures for good old operators Kimmel-Bruckstein 03

Segmentation

qUltrasound images Caselles,Kimmel, Sapiro ICCV’95

Segmentation Pintos

Woodland Encounter Bev Doolittle 1985 qWith a good prior who needs the data…

Segmentation Caselles,Kimmel, Sapiro ICCV’95

Prior knowledge…

Segmentation

Caselles,Kimmel, Sapiro ICCV’95

Segmentation qWith a good prior who needs the data…

Wrong Prior???

Curves in the Plane qC(p)={x(p),y(p)}, p [0,1] y x C(0) C(0.1) C(0.2) C(0.4) C(0.7) C(0.95) C(0.9) C(0.8) p C =tangent

Arc-length and Curvature s(p)= | |dp C

Calculus of Variations Find C for which is an extremum Euler-Lagrange:

Calculus of Variations Important Example  Euler-Lagrange:, setting   Curvature flow

Potential Functions (g) x I(x,y)I(x) x g(x) x x g(x,y) Image Edges

Snakes & Geodesic Active Contours qSnake model Terzopoulos-Witkin-Kass 88 qEuler Lagrange as a gradient descent qGeodesic active contour model Caselles-Kimmel-Sapiro 95 qEuler Lagrange gradient descent

Maupertuis Principle of Least Action Snake = Geodesic active contour up to some, i.e  Snakes depend on parameterization.  Different initial parameterizations yield solutions for different geometric functionals x y p 1 0 Caselles Kimmel Sapiro, IJCV 97

Geodesic Active Contours in 1D Geodesic active contours are reparameterization invariant I(x) x g(x) x

Geodesic Active Contours in 2D g(x)= G *I s

Controlling -max I g Smoothness Cohen Kimmel, IJCV 97

Fermat’s Principle In an isotropic medium, the paths taken by light rays are extremal geodesics w.r.t. i.e., Cohen Kimmel, IJCV 97

Experiments - Color Segmentation Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Tumor in 3D MRI Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

Segmentation in 4D Malladi, Kimmel, Adalsteinsson, Caselles, Sapiro, Sethian SIAM Biomedical workshop 96

Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Tracking in Color Movies Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Edge Gradient Estimators Xu-Prince 98, Paragios et al. 01, Vasilevskiy-Siddiqi 01, Kimmel-Bruckstein 01

Edge Gradient Estimators qWe want a curve with large points and small ‘s so: qConsider the functional qWhere is a scalar function, e.g..

The Classic Connection Suppose and we consider a closed contour for C(s). We have and by Green’s Theorem we have

qTherefore:  Hence curves that maximize are curves that enclose all regions where is positive! qWe have that the optimal curves in this case are The Zero Crossings of the Laplacian isn’t this familiar? The Classic Connection

qIt is pedagogically nice, but the MARR-HILDRETH edge detector is a bit too sensitive. qSo we do not propose a grand return to MH but a rethinking of the functionals used in active contours in view of this. qINDEED, why should we ignore the gradient directions (estimates) and have every edge integrator controlled by the local gradient intensity alone? The Classic Connection

Our Proposal qConsider functional of the form qThese functionals yield “regularized” curves that combine the good properties of LZC’s where precise border following is needed, with the good properties of the GAC over noisy regions!

Implementation Details qWe implement curve evolution that do gradient descent w.r.t. the functional Here the Euler Lagrange Equations provide the explicit formulae. qFor closed contours we compute the evolved curve via the Osher-Sethian “miracle” numeric level set formulation.

Closed contours EL eq. GAC LZC Kimmel-Bruckstein IVCNZ01

Closed contours EL eq. GAC LZC LZC+ e GAC Kimmel-Bruckstein IVCNZ01

Along the curve b.c. at C(0) and C(L) Open contours Kimmel-Bruckstein IVCNZ01

Open contours Kimmel-Bruckstein IVCNZ01

Geometric Measures Weighted arc-length Weighted area Alignment Robust - alignment e.g. Variational meaning for Marr-Hildreth edge detector Kimmel-Bruckstein IVCNZ01

Geometric Measures Minimal variance Chan-Vese, Mumford-Shah, Max-Lloyd, Threshold,…

Geometric Measures Robust minimal deviation

Haralick/Canny-like Edge Detector qHaralick suggested as edge detector Laplace Alignment Topological Homogeneity

Haralick/Canny Edge Detector qHaralick co-area h Thus, indicates optimal alignment + topological homogeneity

Closed Contours & Level Set Method implicit representation of C Then, Geodesic active contour level set formulation Including weighted (by g) area minimization y x C(t) C(t) level set x y

Operator Splitting Schemes qAdditive operator splitting (AOS) Lu et al. 90, Weickert, et al. 98 uunconditionally stable for non-linear diffusion qGiven the evolution write qConsider the operator qExplicit scheme u, the time step, is upper bounded for stability

LOD: Operator Splitting Schemes qImplicit scheme uinverting large bandwidth matrix qFirst order, semi-implicit, additive operator splitting (AOS), or locally one-dimensional (LOD) multiplicative schemes are stable and efficient given by linear tridiagonal systems of equations that can be solved for by Thomas algorithm AOS:

Operator Splitting Schemes qWe used the following relation (AOS) qLocally One-Dimensional scheme (LOD) qDecoupling the axes and the implicit formulation leads to computational efficiency The 1st order `splitting’ idea is based on the operator expansion

qThe geodesic active contour model Where I is the image and  the implicit representation of the curve  If  is a distance, then, and the short time evolution is qNote that and thus can be computed once for the whole image Example: Geodesic Active Contour y x C(t) x y Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE T-IP 2001

Example: Geodesic Active Contour  is restricted to be a distance map: Re-initialization by Sethian’s fast marching method every iteration in O(n). Computations are performed in a narrow band around the zero set Multi-scale approach: process a Gaussian pyramid of the image y x C(t) x y

Tracking Objects in Movies qMovie volume as a spatial-temporal 3D hybrid space uThe AOS scheme is uEdge function derived by the Beltrami framework Sochen Kimmel Malladi 98 qContour in frame n is the initial condition for frame n+1. x x y y t t

Experiments - Curvature Flow

Experiments - Curvature Flow CPU Time

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Tracking Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Information extraction Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Holzman-Gazit, Goldshier, Kimmel 2003 Thin Structures

Segmentation in 3D Change in topology Caselles,Kimmel, Sapiro, Sbert, IEEE T-PAMI 97

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001 Coupled surfaces EL equations

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Futurism Recognition from periodic motion Carlo Carra, 1914 Dynamism of a Dog on a Leash Giacomo Balla, 1912 Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002 Eadweard Muybridge, Animals in Motion, 1887

Classification (dogs & cats) walkrungallopcat... Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Classification (dogs & cats) Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Classification (people) walkrunrun45 Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Classification (people) Goldenberg, Kimmel, Rivlin, Rudzsky, ECCV 2002

Conclusions qGeometric-Variational method for segmentation and tracking in finite dimensions based on prior knowledge (more accurately, good initial conditions). qUsing the directional information for edge integration. qGeometric-variational meaning for the Marr-Hildreth and the Haralick (Canny) edge detectors, leads to ways to design improved ones. qEfficient numerical implementation for active contours. qVarious medical and more general applications.

Gray Matter Segmentation Goldenberg Kimmel Rivlin Rudzsky, VLSM 2001

Edge Indicator Function for Color qBeltrami framework: Color image = 2D surface in space qThe induced metric tensor for the image surface qEdge indicator = largest eigenvalue of the structure tensor metric. It represents the direction of maximal change in X I Y

AOS Proof: The whole low order splitting idea is based on the operator expansion