Level set method and image segmentation Lecture 4 Level set method and image segmentation

Overview What is image segmentation? Separation of image domain based on contents. General image segmentation:

Overview What is image segmentation? Separation of image domain based on contents. Object recognition (computer vision)

Overview What is image segmentation? Separation of image domain based on contents. Object recognition (computer vision)

Overview What is image segmentation? Separation of image domain based on contents. Medical image segmentation

Overview What is image segmentation? Separation of image domain based on contents. Medical image segmentation

Overview What is image segmentation? Separation of image domain based on contents. Medical image segmentation Computer-aided diagnosis

Overview What is image segmentation? Separation of image domain based on contents. Image segmentation in biology

Overview What is important for image segmentation? Main approaches: Edges which can be extracted by differentiation (lower level segmentation) Content classification (higher level segmentation) Main approaches: Energy method (Mumford-Shah) Curve evolution (snake, level set method)

Energy Method Mumford-Shah (MS) Model

Mumford-Shah Functional Mumford-Shah (MS) functional: Open questions: Observed Image Set of Discontinuities

Finite Difference Approximation of MS Model: Ambrosio and Tortorelli, 1990 Setting α=β=1, approximate MS functional by Interpreting the approximation (formal) Let τ(x) be the distance function to the jump set Su and Construct (uε,vε) -> (u,1) as ε->0, so that More rigorous analysis: by Γ-convergence

Finite Difference Approximation of MS Model: A. Chambolle, 1995 Finite difference approximation (1D) Finite difference approximation (2D) where and Γ-convergence Set of discontinuities of u

Curve Evolution Snake, Geodesic Active-Contour, Chan-Vese Model, Level Set Method

Overview Objective: automatically detect contours of objects. Questions: How contours (curves in 2D or surfaces in 3D) are represented? Explicit representation (parametric) Implicit representation (level set) How the locations of contours are determined?

Snakes: Active contour models (Kass, Witkin and Terzopoulos, 1987) where Edge Indicator Function Makes curves act membrane like. Makes curves act thin plate like. Makes curves be stuck at edges.

Snakes: Active contour models Gradient flow: Drawbacks of “snakes”: Curves’ representation is not intrinsic. We could obtain different solutions by changing the parametrization while preserving the same initial curve. Because of the regularity constraint, the model does not handle changes of topology. We can reach only a local minimum, we have to choose initial curve close enough to the object to be detected. The choice of a set of marker points for discretizing the parametrized evolving curve may need to be constantly updated. where

The Geodesic Active Contours Model Dropping the second order term of “snakes” Geodesic active contours model Intrinsic! Let Idea: weight defines a new Riemannian metric for which we search for geodesics. What’s their relations?

Curvature: Elements of Differential Geometry Parametric curves: Note: Curvature: and

Curvature: Elements of Differential Geometry Parametric curves: Note: Curvature (if the curve is parameterized by arc length):

Curvature: Elements of Differential Geometry Curves as level set of a function : Differentiating the equation Suppose , then Differentiating the equation one more time

Link Between Snakes and Geodesic Active Contours Let and Calculus of : We have Assuming : Integration by parts

Link Between Snakes and Geodesic Active Contours Denote arc length by , then Thus Decompose in tangential and normal directions where T and N are normalized tangent and normal vectors.

Link Between Snakes and Geodesic Active Contours Then By Cauchy-Schwartz inequality, the flow leads to most rapid decrease of the energy functional is

Link Between Snakes and Geodesic Active Contours Calculus of . We have Then, Integration by part w.r.t. q

Link Between Snakes and Geodesic Active Contours Then ( by ) Flow of most rapidly decrease

Link Between Snakes and Geodesic Active Contours Under suitable conditions, the models of snakes and geodesic active contours are equivalent in the following sense: Why is this true? Plug in and respectively

Note: Mean Curvature Flow If g is constant, we obtain the mean curvature flow:

Caselles et al.’s Modification Drawbacks of original geodesic active contours: hard to detect nonconvex objects. Improved evolution equation Note: the above flow does not correspond to any energy functional unless g is a constant. Implement the flow by level set method

Level Set Method Level set method is an efficient and effective method in solving curve evolution equations: Observation: A curve can be seen as the zero-level of a function in higher dimension. Consider such that Differentiating w.r.t. t

Level Set Method If the level set function is negative inward and positive outward of the curve, then the unit inward normal vector is

Level Set Method Level set equation Level set formulation of the improved geodesic active contours where

Level Set Method Discretization Mean curvature motion Reinitialization every n steps (Section 4.3.4) Central differencing

Level Set Method What does reinitialization do? How can this happen? Solution of mean curvature flow with initial curve a unite circle 0-level set Becomes flat as t approaches 1/2

Level Set Method Discretization Mean curvature motion t increases

Level Set Method Discretization Scalar speed evolution with c>0 where Change + to – if c<0 Non-oscillotary upwind discretization

Level Set Method Discretization Scalar speed evolution c=1 c=-1

Level Set Method Discretization Pure advection equation Motion vector field One-side upwind discretization

Level Set Method Discretization where

Level Set Method Numerical results

Brain Aneurysm Segmentation B. Dong et al., Level set based brain aneurysm capturing in 3D, Inverse Problems and Imaging, 4(2), 241-255, 2010.

Brain Aneurysm Segmentation Raw CT data

Brain Aneurysm Segmentation Vascular tree segmentation (e.g. using CV model)

Brain Aneurysm Segmentation Objective: separate aneurysm from vascular tree

Brain Aneurysm Segmentation What was commonly done by doctors Problems: Robustness: hard to perform for complicated vessels. Consistency: hard to unify the cut cross subjects.

Brain Aneurysm Segmentation Understanding of the problem: illusory contours

Brain Aneurysm Segmentation Understanding of the problem: illusory contours

Brain Aneurysm Segmentation Zhu and Chan, 2003:

Brain Aneurysm Segmentation Zhu and Chan, 2003:

Brain Aneurysm Segmentation Zhu and Chan, 2003: Problem with Zhu-Chan’s PDE

Brain Aneurysm Segmentation From 2D to 3D: choice of curvature

Brain Aneurysm Segmentation

Brain Aneurysm Segmentation Zhu&Chan Ours Initial Final Result

Energy Method - revisited Approximations of Mumford-Shah Model

Binary Approximation of Mumford-Shah Model Active Contours without Edges, T.F. Chan and L. A. Vese, 2001. (CV model)

Binary Approximation of Mumford-Shah Model Level set representation of curves Level set formulation of curve length and area inside C

Binary Approximation of Mumford-Shah Model Level set formulation of CV model Solution of piecewise constant MS model

Binary Approximation of Mumford-Shah Model Solving CV model Solving for the constants c

Binary Approximation of Mumford-Shah Model Solving CV model Solving for the level set function

Binary Approximation of Mumford-Shah Model Solving CV model Solving for the level set function 𝐻 2,𝜖 has global support while 𝐻 1,𝜖 is local.

Binary Approximation of Mumford-Shah Model Solving CV model Solving for the level set function

Binary Approximation of Mumford-Shah Model Solving CV model Solving for the level set function Reinitialize after every n iterations

Binary Approximation of Mumford-Shah Model Advantages of CV model Do not rely on image gradient, robust to noise

Binary Approximation of Mumford-Shah Model Advantages of CV model Do not rely on image gradient, robust to noise Find boundaries without edge information

Convexified Binary MS Model Binary Mumford-Shah CV model Gradient flow of CV model

Convexified Binary MS Model When a non-compactly supported smooth Heaviside function is chosen, the PDE has the same stationary solution as Corresponding energy functional Convexified segmentation model

Convexified Binary MS Model The non-convex MS model can be solved by a convex relaxation model! Further development: Full convexification and generalization (UCLA CAM 10-43, 10-44) Piecewise polynomial (UCLA CAM 13-50) Generalization on graphs (UCLA CAM 12-03, 14-79) Key to the proof: Co-area formula

Convexified Binary MS Model Solving the convexified piecewise constant MS model Here, 𝑔 𝑥 =1 𝑜𝑟 𝑔 𝑥 = 1 1+𝛼 𝛻𝑓(𝑥) 2 Let 𝑑 =𝛻𝑢, we have the augmented Lagrangian iteration Final algorithm is to optimize 𝑢 and 𝑑 alternatively (ADMM) Goldstein, Bresson and Osher. Journal of Scientific Computing, 2010, 45(1-3): 272-293. (Original algorithm) Liu, Zhang, Dong, Shen and Gu. SIAM Journal on Imaging Sciences, 2016, 9(2): 495-536. (Algorithm for a more complicated model and convergence analysis)

Homework (Due Nov 25th 24:00) Implement the geodesic active contours model using level set formulation (ppt page 38). Implement the convexified CV model (ppt page 68). Use images of your own selection Observe: Segmentation results for different types of images Effects of noise and blur on the results Comparison between the two models Note: codes of reinitialization will be provided.