1. Level set method and image segmentation
Lecture 4
Level set method and image segmentation

2. Overview
What is image segmentation? Separation of image domain based on contents.
General image segmentation:

3. Overview
What is image segmentation? Separation of image domain based on contents.
Object recognition (computer vision)

4. Overview
What is image segmentation? Separation of image domain based on contents.
Object recognition (computer vision)

5. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation

6. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation

7. Overview
What is image segmentation? Separation of image domain based on contents.
Medical image segmentation
Computer-aided diagnosis

8. Overview
What is image segmentation? Separation of image domain based on contents.
Image segmentation in biology

9. Overview What is important for image segmentation? Main approaches:
Edges which can be extracted by differentiation (lower level segmentation)
Content classification (higher level segmentation)
Main approaches:
Energy method (Mumford-Shah)
Curve evolution (snake, level set method)

10. Energy Method
Mumford-Shah (MS) Model

11. Mumford-Shah Functional
Mumford-Shah (MS) functional:
Open questions:
Observed Image
Set of Discontinuities

12. Finite Difference Approximation of MS Model: Ambrosio and Tortorelli, 1990
Setting α=β=1, approximate MS functional by
Interpreting the approximation (formal)
Let τ(x) be the distance function to the jump set Su and
Construct (uε,vε) -> (u,1) as ε->0, so that
More rigorous analysis: by Γ-convergence

13. Finite Difference Approximation of MS Model: A. Chambolle, 1995
Finite difference approximation (1D)
Finite difference approximation (2D)
where and
Γ-convergence
Set of discontinuities of u

14. Curve Evolution
Snake, Geodesic Active-Contour, Chan-Vese Model, Level Set Method

15. Overview Objective: automatically detect contours of objects.
Questions:
How contours (curves in 2D or surfaces in 3D) are represented?
Explicit representation (parametric)
Implicit representation (level set)
How the locations of contours are determined?

16. Snakes: Active contour models
(Kass, Witkin and Terzopoulos, 1987)
where
Edge Indicator Function
Makes curves act membrane like.
Makes curves act thin plate like.
Makes curves be stuck at edges.

17. Snakes: Active contour models
Gradient flow:
Drawbacks of “snakes”:
Curves’ representation is not intrinsic. We could obtain different solutions by changing the parametrization while preserving the same initial curve.
Because of the regularity constraint, the model does not handle changes of topology.
We can reach only a local minimum, we have to choose initial curve close enough to the object to be detected.
The choice of a set of marker points for discretizing the parametrized evolving curve may need to be constantly updated.
where

18. The Geodesic Active Contours Model
Dropping the second order term of “snakes”
Geodesic active contours model
Intrinsic! Let
Idea: weight defines a new Riemannian metric for which we search for geodesics.
What’s their relations?

19. Curvature: Elements of Differential Geometry
Parametric curves:
Note:
Curvature:
and

20. Curvature: Elements of Differential Geometry
Parametric curves:
Note:
Curvature (if the curve is parameterized by arc length):

21. Curvature: Elements of Differential Geometry
Curves as level set of a function :
Differentiating the equation
Suppose , then
Differentiating the equation one more time

22. Link Between Snakes and Geodesic Active Contours
Let and
Calculus of : We have
Assuming :
Integration by parts

23. Link Between Snakes and Geodesic Active Contours
Denote arc length by , then
Thus
Decompose in tangential and normal directions
where T and N are normalized tangent and normal vectors.

24. Link Between Snakes and Geodesic Active Contours
Then
By Cauchy-Schwartz inequality, the flow leads to most rapid decrease of the energy functional is

25. Link Between Snakes and Geodesic Active Contours
Calculus of We have
Then,
Integration by part w.r.t. q

26. Link Between Snakes and Geodesic Active Contours
Then
( by )
Flow of most rapidly decrease

27. Link Between Snakes and Geodesic Active Contours
Under suitable conditions, the models of snakes and geodesic active contours are equivalent in the following sense:
Why is this true?
Plug
in and respectively

28. Note: Mean Curvature Flow
If g is constant, we obtain the mean curvature flow:

29. Caselles et al.’s Modification
Drawbacks of original geodesic active contours: hard to detect nonconvex objects.
Improved evolution equation
Note: the above flow does not correspond to any energy functional unless g is a constant.
Implement the flow by level set method

30. Level Set Method
Level set method is an efficient and effective method in solving curve evolution equations:
Observation: A curve can be seen as the zero-level of a function in higher dimension.
Consider such that
Differentiating w.r.t. t

31. Level Set Method
If the level set function is negative inward and positive outward of the curve, then the unit inward normal vector is

32. Level Set Method Level set equation
Level set formulation of the improved geodesic active contours
where

33. Level Set Method Discretization Mean curvature motion
Reinitialization every n steps (Section 4.3.4)
Central differencing

34. Level Set Method What does reinitialization do? How can this happen?
Solution of mean curvature flow with initial curve a unite circle
0-level set
Becomes flat as t approaches 1/2

35. Level Set Method
Discretization
Mean curvature motion
t increases

36. Level Set Method Discretization Scalar speed evolution with c>0
where
Change + to – if c<0
Non-oscillotary upwind discretization

37. Level Set Method
Discretization
Scalar speed evolution
c=1
c=-1

38. Level Set Method Discretization Pure advection equation
Motion vector field
One-side upwind discretization

39. Level Set Method
Discretization
where

40. Level Set Method
Numerical results

41. Brain Aneurysm Segmentation
B. Dong et al., Level set based brain aneurysm capturing in 3D, Inverse Problems and Imaging, 4(2), ,

42. Brain Aneurysm Segmentation
Raw CT data

43. Brain Aneurysm Segmentation
Vascular tree segmentation (e.g. using CV model)

44. Brain Aneurysm Segmentation
Objective: separate aneurysm from vascular tree

45. Brain Aneurysm Segmentation
What was commonly done by doctors
Problems:
Robustness: hard to perform for complicated vessels.
Consistency: hard to unify the cut cross subjects.

46. Brain Aneurysm Segmentation
Understanding of the problem: illusory contours

47. Brain Aneurysm Segmentation
Understanding of the problem: illusory contours

48. Brain Aneurysm Segmentation
Zhu and Chan, 2003:

49. Brain Aneurysm Segmentation
Zhu and Chan, 2003:

50. Brain Aneurysm Segmentation
Zhu and Chan, 2003:
Problem with Zhu-Chan’s PDE

51. Brain Aneurysm Segmentation
From 2D to 3D: choice of curvature

52. Brain Aneurysm Segmentation

53. Brain Aneurysm Segmentation
Zhu&Chan
Ours
Initial
Final Result

54. Energy Method - revisited
Approximations of Mumford-Shah Model

55. Binary Approximation of Mumford-Shah Model
Active Contours without Edges, T.F. Chan and L. A. Vese, (CV model)

56. Binary Approximation of Mumford-Shah Model
Level set representation of curves
Level set formulation of curve length and area inside C

57. Binary Approximation of Mumford-Shah Model
Level set formulation of CV model
Solution of piecewise constant MS model

58. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the constants c

59. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function

60. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function
𝐻 2,𝜖 has global support while 𝐻 1,𝜖 is local.

61. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function

62. Binary Approximation of Mumford-Shah Model
Solving CV model
Solving for the level set function
Reinitialize after every n iterations

63. Binary Approximation of Mumford-Shah Model
Advantages of CV model
Do not rely on image gradient, robust to noise

64. Binary Approximation of Mumford-Shah Model
Advantages of CV model
Do not rely on image gradient, robust to noise
Find boundaries without edge information

65. Convexified Binary MS Model
Binary Mumford-Shah
CV model
Gradient flow of CV model

66. Convexified Binary MS Model
When a non-compactly supported smooth Heaviside function is chosen, the PDE has the same stationary solution as
Corresponding energy functional
Convexified segmentation model

67. Convexified Binary MS Model
The non-convex MS model can be solved by a convex relaxation model!
Further development:
Full convexification and generalization (UCLA CAM 10-43, )
Piecewise polynomial (UCLA CAM 13-50)
Generalization on graphs (UCLA CAM 12-03, 14-79)
Key to the proof: Co-area formula

68. Convexified Binary MS Model
Solving the convexified piecewise constant MS model
Here,
𝑔 𝑥 = 𝑜𝑟 𝑔 𝑥 = 1 1+𝛼 𝛻𝑓(𝑥) 2
Let 𝑑 =𝛻𝑢, we have the augmented Lagrangian iteration
Final algorithm is to optimize 𝑢 and 𝑑 alternatively (ADMM)
Goldstein, Bresson and Osher. Journal of Scientific Computing, 2010, 45(1-3): (Original algorithm)
Liu, Zhang, Dong, Shen and Gu. SIAM Journal on Imaging Sciences, 2016, 9(2): (Algorithm for a more complicated model and convergence analysis)

69. Homework (Due Nov 25th 24:00)
Implement the geodesic active contours model using level set formulation (ppt page 38).
Implement the convexified CV model (ppt page 68).
Use images of your own selection
Observe:
Segmentation results for different types of images
Effects of noise and blur on the results
Comparison between the two models
Note: codes of reinitialization will be provided.