Active Contour Models (Snakes) By: Sajjad Fouladvand Professor : Dr Alireza Ossareh December 16th, 2011 Shahid Chamran University Of Ahvaz

Overview Introduction Modeling Simple Demo Weakness An Active Contour Balloon Model Gradient Vector Flow (GVF) Applications of snakes References 1

Introduction What is the objective? What is segmentation? To perform the task of Image Segmentation. What is segmentation? Subdividing or partitioning an image into its constituent regions or objects. 2

Introduction First introduced in 1987 by Kass et al, and gained popularity since then. Represents an object boundary or some other salient image feature as a parametric curve. An energy functional E is associated with the curve. The problem of finding object boundary is cast as an energy minimization problem. 3

Introduction A higher level process or a user initializes any curve close to the object boundary. The snake then starts deforming and moving towards the desired object boundary. In the end it completely “shrink-wraps” around the object. 4

Simple modeling The snake deforming and moving towards the desired object boundary. 5

Simple modeling Represent the curve with a set of n points : 𝑣 𝑖 = 𝑥 𝑖 , 𝑦 𝑖 𝑖=0 , …, 𝑛−1 6

Simple modeling The total energy of the snake is defined as 𝐸 𝑡𝑜𝑡𝑎𝑙 = 𝐸 𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙 + 𝐸 𝑒𝑥𝑡𝑒𝑟𝑛𝑎𝑙 Encourages smoothness or a particular shape Encourages curve onto image structures (e.g edges) 7

Simple modeling The energy terms are defined cleverly in a way such that the final position of the contour will have a minimum energy ( 𝐸 𝑚𝑖𝑛 ) Therefore our problem of detecting objects reduces to an energy minimization problem. 8

Modeling - internal energy The snake should not break apart! Points on the snake must stay close to each other Each point on the snake pulls its neighbors The force is proportional to the distance |Pi – Pi-1| 9

Modeling- internal energy For a curve represented as a set of points a simple elastic energy term is : 𝐸 𝑖𝑛𝑡𝑒𝑟𝑛𝑎𝑙 =𝛼 𝑖=0 𝑛−1 𝐿 𝑖 2 =𝛼 𝑖=0 𝑛−1 ( 𝑥 𝑖+1 − 𝑥 𝑖 ) 2 +( 𝑦 𝑖+1 − 𝑦 𝑖 ) 2 This encourages the closed curve to shrink to a point (like a very small elastic band) 10

Modeling- external energy Edge pixels must “pull” the snake points. The stronger the edge, the stronger the pull. • Numerous forms can be used, attracting the curve toward different image features 11

Modeling - external energy Suppose we have an image 𝐼 𝑥,𝑦 Generate gradient images 𝐺 𝑥 𝑥,𝑦 , 𝐺 𝑦 𝑥,𝑦 The absolute edge strength at a point is then 𝐺 𝑥 (𝑥,𝑦) 2 + 𝐺 𝑦 (𝑥,𝑦) 2 An external energy term for a snake is thus: 𝐸 𝑒𝑥𝑡𝑒𝑟𝑛𝑎𝑙 =− 𝑖=0 𝑛−1 𝐺 𝑥 (𝑥,𝑦) 2 + 𝐺 𝑦 (𝑥,𝑦) 2 (Negative in order that minimizing it forces the curve toward large edges) 12

Modeling - Simple Elastic Snake A simple elastic snake is thus defined by a set of n points An internal elastic energy term An external edge based energy term • To use this to locate the outline of an object Initialize in the vicinity of the object Modify the points to minimize the total energy 13

Snake Energy Functional Given a snake with N points 𝑣1, 𝑣2, …, 𝑣𝑁 Define the following Energy Functional: 𝐸= 𝑖=1 𝑁 𝑎 𝑖 𝐸 𝑐 ( 𝑣 𝑖 )+ 𝑏 𝑖 𝐸 𝑠 𝑣 𝑖 + 𝑐 𝑖 𝐸 𝑔 ( 𝑣 𝑖 ) Where: “Continuity” 𝐸 𝑐 𝐸 𝑠 “Smoothness” 𝐸 𝑔 “Edgeness” ai, bi, ci are “weights” to control influence 14

Snake Forces Edge attraction Continuity Smoothness 15

Distance between points should be kept close to average Continuity Term Given a snake with 𝑁 points 𝑣1, 𝑣2, …, 𝑣𝑁 Let d be the average distance between points Define the continuity term of the Energy Functional: 𝐸 𝑐 𝑣 𝑖 = (𝑑− 𝑣 𝑖 − 𝑣 𝑖−1 ) 2 Distance between points should be kept close to average 16

Smoothness Term 𝐸 𝑠 𝑣 𝑖 = 𝑣 𝑖−1 −2 𝑣 𝑖 + 𝑣 𝑖+1 2 Given a snake with N points 𝑣1, 𝑣2, …, 𝑣𝑁 Curvature should be kept small Define the smoothness term of the Energy Functional: 𝐸 𝑠 𝑣 𝑖 = 𝑣 𝑖−1 −2 𝑣 𝑖 + 𝑣 𝑖+1 2 Second derivative This term prevent Local minimum 17

Smoothness Term- Example (2,5) (2,2) (3,1) (1,1) (3,1) (1,1) 3−2 2 +1 2+ 1−2 5 +1 2 3−2 2 +1 2+ 1−2 2 +1 2 = −8 2=64 = −2 2=4 18

Edgeness Term 𝐸 𝑔 𝑣 𝑖 =− 𝛻𝐼( 𝑣 𝑖 ) Given a snake with N points 𝑣1, 𝑣2, …, 𝑣𝑁 Define the edgeness term of the Energy Functional: 𝐸 𝑔 𝑣 𝑖 =− 𝛻𝐼( 𝑣 𝑖 ) Magnitude of the gradient should be LARGE 19

Energy minimization Several algorithms have been proposed to fit deformable contours: Greedy search Dynamic programming 20

Energy minimization- greedy For each point, search window around it and move to where energy function is minimal Typical window size, e.g., 5 x 5 pixels Stop when predefined number of points have not changed in last iteration, or after max number of iterations Note: Convergence not guaranteed Need decent initialization 21

Model The most basic way of representing a curve in two dimensions is the explicit form: 𝑦 = 𝑓 𝑥 Nearly all simple curves can be represented using the explicit form, but for more complex curves with two or more y-values for a single corresponding x-value the explicit form fails: 𝑓 𝑥; 𝑦 = 0 Unfortunately the inherent form of the implicit curve equation Unfortunately the inherent form of the implicit curve equation does not allow us to compute points on the curve directly, often requiring the solution of a non-linear equation for each point. 𝑉(𝑠)=(𝑥(𝑠),𝑦(𝑠)) 𝑢𝑠𝑢𝑎𝑙𝑙𝑦 𝑤𝑖𝑡ℎ 𝑠∈[0,1] 22

Model 𝐸= 𝑠 𝛼(𝑠) 𝑣 𝑠 2 +𝛽(𝑠) 𝑣 𝑠𝑠 2 + 𝐸 𝑒𝑥𝑡 𝑣 𝑠 𝑑𝑠 𝐸 𝑒𝑥𝑡 = 𝑠 𝐸 𝑖𝑚𝑎𝑔𝑒 𝑣 𝑠 𝑑𝑠 Total internal energy of the snake 𝐸 𝑒𝑥𝑡 =− 𝛻𝐼(𝑥,𝑦) 2 𝐸 𝑒𝑥𝑡 =− 𝛻( 𝐺 𝜎 𝑥,𝑦 ∗𝐼(𝑥,𝑦)) 2 23

Model (a) The original image 𝐼(𝑥; 𝑦), showing a black square on a white background. (b) The image energy derived from the function 𝐸 𝑒𝑥𝑡 =− 𝛻 𝐺 𝜎 𝑥,𝑦 ∗𝐼 𝑥,𝑦 2 with σ= 9, shown as a 3D surface. (c) The same image energy as seen in. 24

Model This can be viewed as a force balance equation A snake that minimizes E must satisfy the Euler equation: 𝜎 𝑣 𝑠𝑠 −𝛽 𝑣 𝑠𝑠𝑠 −𝛻 𝐸 𝑖𝑚𝑎𝑔𝑒 =0 This can be viewed as a force balance equation 𝐹 𝑖𝑛𝑡 + 𝐹 𝑒𝑥𝑡 1 =0 Each term corresponds to a force produced by the respective energy terms. The contour deforms under the action of these forces. 25

Simple Demo There is a simple demo for explain snake energy http://users.ecs.soton.ac.uk/msn/book/new_demo/Snakes/ http://server.cs.ucf.edu/~vision/snakes/snakes.html#how 26

Weakness Fails to detect concave boundaries. External force cant pull control points into boundary concavity. the initial contour must, in general, be close to the true boundary or else it will likely converge to the wrong result. 27

An Active Contour Balloon Model The active contour model that Kass et al. introduced was further developed by Laurent D. Cohen in the paper “On Active Contour Models and Balloons“ [Cohen-91]. Works on the same principles as the Kass et al. snake. where the Kass et al. snake would shrink when not under the influence of image forces the Cohen snake expands. 28

An Active Contour Balloon Model Introduces makes it possible to find the contours of an object by placing the initial snake inside the object instead of outside, while also providing more stable results. 29

Gradient Vector Flow (GVF) A new external force for snakes A snake with GVF external forces moves into the concave boundary region. 30

Gradient Vector Flow (GVF) The contour can also be initialized across the boundary of object!!Something not possible with traditional snakes. also extended GVF to three dimensions 31

Model for GVF snake The GVF field is defined to be a vector field 𝑉(𝑥,𝑦) = (𝑢(𝑥,𝑦),𝑣(𝑥,𝑦)) 𝑉(𝑥,𝑦) is defined such that it minimizes the energy functional Force equation of GVF snake 𝛼 𝑣 𝑠𝑠 −𝛽 𝑣 𝑠𝑠𝑠 +𝑣=0 f(x,y) is the edge map of the image. 32

Model for GVF snake GVF field can be obtained by solving following equations 2 Is the Laplacian operator The above equations are solved iteratively using time derivative of u and v. 33

Traditional external force field v/s GVF field Traditional force GVF force Three general properties of edge maps are important in the present context. First, the gradient of an edge map has vectors pointing toward the edges, which are normal to the edges at the edges. Second, these vectors generally have large magnitudes only in the immediate vicinity of the edges. Third, in homogeneous regions, where is nearly constant, is nearly zero. Because of the first property, a snake initialized close to the edge will converge to a stable configuration near the edge. This is a highly desirable property. Because of the second property, however, the capture range will be very small, in general. Because of the third property, homogeneous regions will have no external forces whatsoever. These last two properties are undesirable 34

Traditional external force field v/s GVF field Although the external forces correctly point toward the object boundary, within the boundary concavity the forces point horizontally in opposite directions. We see that the magnitude of the external forces die out quite rapidly away from the object boundary. 35

Results Traditional snake GVF snake 36

Results Notice that the image is poor quality with sampling artifacts Magnetic resonance image of the left ventricle of human heart Notice that the image is poor quality with sampling artifacts 37

Problem with GVF snake Very sensitive to parameters. Slow. Finding GVF field is computationally expensive. 38

Applications of snakes Image segmentation particularly medical imaging community (tremendous help). Motion tracking. Stereo matching . Shape recognition. 39

References M.KASS, A.WITKIN, D.TERZOPOULOS, “Snakes: Active Contour Models”, International Journal of Computer Vision, 321-331 (1988) Ch.Xu, J.L. Prince, “Gradient Vector Flow: A New External Force for Snakes”, The Johns Hopkins University Ch.Xu, J.L. Prince, “Snakes, Shapes, and Gradient Vector Flow”, IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 3, MARCH 1998 N.P Tiilikainen, “A Comparative Study of Active Contour Snakes”, computer science department copenhagen University, 2007 40

With special thanks

Any question? Any idea