1. Kapitel 13 “Interactive Segmentation” – p. 1 Interactive Segmentation  Live-wire approach  Random walker segmentation TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA Kapitel 13

2. Kapitel 13 “Interactive Segmentation” – p. 2 Live-wire approach Live-wire approach (intelligent scissors):  The user interactively picks a seed point on the boundary.  Then, a live-wire is displayed in real time from the initial point to any subsequent position taken by the cursor.  The entire 2D boundary is specified by means of a set of live-wire segments in this manner. The detection of segments is formulated as a graph search problem, which finds the globally optimal (minimum-weight) path between an initial start pixel and an end pixel.

3. Kapitel 13 “Interactive Segmentation” – p. 3 Live-wire approach E.N. Mortensen and W.A. Barrett: Intelligent scissors for image composition. SIGGRAPH, 191-198, 1995.

4. Kapitel 13 “Interactive Segmentation” – p. 4 Exkurs: Dijkstra‘s algorithm Problem: finding shortest paths from a source vertex to all other vertices in a graph. Dijkstra's algorithm: Greedy solution to the single-source shortest path problem. It works on both directed and undirected graphs. All edges must have nonnegative weights. Input: Weighted graph G = {E, V} and start vertex s ∈ V, such that all edge weights are nonnegative. Output: Lengths of shortest paths (or the shortest paths themselves) from the given start vertex s ∈ V to an end node t ∈ V (or all other vertices).

5. Kapitel 13 “Interactive Segmentation” – p. Step 1: Assign the start node with distance zero and mark it as visited. Assign all other vertices with infinity and mark them unvisited. Step 2: Consider the most recently visited node X with distance D x. For each of its unvisited neighbors Y with distance D y, replace D y by min(D x + weight(X,Y), D y ) Step 3: Choose the unvisited vertex with the smallest distance and mark it visited. Step 4: Repeat Steps 2 and 3 until the end node is marked visited. Step 5: Go backwards through the graph, retracing the minimum-weight path from the end node to the start node. 5 Exkurs: Dijkstra‘s algorithm

6. Kapitel 13 “Interactive Segmentation” – p. 6 Exkurs: Dijkstra‘s algorithm Example: Find the minimum-weight path from node s to node t. 1 1 2 3 2 3 5 2 4 s t

7. Kapitel 13 “Interactive Segmentation” – p. 7 Exkurs: Dijkstra‘s algorithm 1 3 5 1 1 2 3 2 3 5 2 4 2 3 1 3 4 3 1 1 2 3 2 3 5 2 4 2 1 3 4 7 3 1 1 2 3 2 3 5 2 4 3 2 1 2 4 5 6 1 1 2 3 2 3 5 2 4 3 3 1 2 4 5 6 1 1 2 3 2 3 5 2 4 3 3 ∞ ∞ ∞ 1 2 4 5 3 1 1 2 3 2 3 5 2 4 3 3

8. Kapitel 13 “Interactive Segmentation” – p. 8 Exkurs: Search algorithm from AI Search is a classical AI problem and many solutions exists (e.g. A*)

9. Kapitel 13 “Interactive Segmentation” – p. 9 Live-wire approach Example:

10. Kapitel 13 “Interactive Segmentation” – p. 10 Live-wire approach Example: Composition E.N. Mortensen and W.A. Barrett: Intelligent scissors for image composition. SIGGRAPH, 191-198, 1995.

11. Kapitel 13 “Interactive Segmentation” – p. 11 Live-wire approach Example: A cursor snap mechanism forces the mouse point to the pixel of maximum gradient magnitude within a user-specified neighborhood. Live-wire segment snaps to a boundary as the free point moves (via cursor movement). The path of the free point is shown in white. Live- wire segments from previous free point positions are shown in green.

12. Kapitel 13 “Interactive Segmentation” – p. 12 Live-wire approach Example: Continuous snap–drag of a live-wire to a coronary edge (the boundary is completed in about 2s) E.N. Mortensen and W.A. Barrett: Interactive live-wire boundary extraction. Medical Image Analysis, 1(4): 331-341.

13. Kapitel 13 “Interactive Segmentation” – p. 13 Live-wire approach Reproducibility for live-wire and manual tracing tool. Eight users extracted five object boundaries five times with the live- wire tool and the same object boundaries three times with the manually tracing tool. Manual tracing takes 2-3 times longer. The boundaries are virtually identical regardless of which user is performing the task!

14. Kapitel 13 “Interactive Segmentation” – p. 14 Live-wire approach Extension to 3D: segment 3D volume data or time sequences of 2D images. A.X. Falcao and J.K. Udupa: A 3D generalization of user-steered live-wire segmentation. Medical Image Analysis, 4(4), 389-402, 2000. The user specifies contours via live-wiring on a few slices that are orthogonal to the natural slices of the original data. If these slices are selected strategically, then one obtains a sufficient number of seed points in each natural slice which enable a subsequent automatic optimal boundary detection therein.

15. Kapitel 13 “Interactive Segmentation” – p. 15 Random walker segmentation Given labeled pixels, for each pixel: What is the probability that a random walker starting from this pixel first reaches each set of labels? L.Grady: Random walks for image segmentation. IEEE-TPAMI, 28: 1768–1783, 2006. (a) A two-region image(b) Use-defined seeds for each region (c) A 4-connected lattice topology seeded (labeled) pixelsunseeded (unlabeled) pixel edge weight: similarity between two nodes, based on e.g., intensity gradient, color changes (d) An undirected weighted graph low-weight edge (sharp color gradient)

16. Kapitel 13 “Interactive Segmentation” – p. 16 Random walker segmentation The algorithm labels an unseeded pixel in following steps: Step 1. Calculate the probability that a random walker starting at an unseeded pixel x first reaches a seed with label s Probability that a random walker starting from each unseeded node first reaches red seed Probability that a random walker starting from each unseeded node first reaches blue seed 0.970.900.85 0.97 0.90 0.85 0.150.100.03 0.15 0.10 0.03 0.100.15 0.03 0.10 0.15 0.850.900.97 0.85 0.90 0.97

17. Kapitel 13 “Interactive Segmentation” – p. 17 Random walker segmentation Step 2. Label each pixel with the most probable seed destination (0.97,0.03)(0.90,0.10)(0.85,0.15)(0.15,0.85) (0.10,0.90)(0.03,0.97) (0.97,0.03)(0.85,0.15)(0.15,0.85) (0.03,0.97) (0.97,0.03)(0.90,0.10)(0.85,0.15)(0.15,0.85) (0.10,0.90)(0.03,0.97) A segmentation corresponding to region boundary is obtained by biasing the random walker to avoid crossing sharp color gradients

18. Kapitel 13 “Interactive Segmentation” – p. 18 Random walker segmentation greenredyellowblue partially labeled imagesegmented image Probabilities

19. Kapitel 13 “Interactive Segmentation” – p. 19 Random walker segmentation Algorithm summary: 1.Generate weights based on image intensities. 2.Build Laplacian matrix. 3.Solve system of linear equations for each label. 4.Assign pixel (voxel) to label for which it has the highest probability.

20. Kapitel 13 “Interactive Segmentation” – p. 20 Random walker segmentation Weight generation: The random walker is governed by edge weights (e.g., probabilities of reaching a pixel p j from its neighboring pixel p i ). f(): intensity function β: influences how quickly the probability decreases The probability is higher the more likely the two pixels belong to the same region.

21. Kapitel 13 “Interactive Segmentation” – p. 21 Random walker segmentation Laplacian matrix L: d i : degree of pixel p i It is shown that finding the probabilities for a walk starting at some pixel to arrive at some seed pixel is related to minmizing:

22. Kapitel 13 “Interactive Segmentation” – p. 22 Random walker segmentation Solution of system of linear equations: Partition the pixels into two sets, V M (all marked/seed pixels, regardless of their label) and V U (unseeded). The pixels in L and x are ordered such that seed pixels are first and unseeded pixels are second. Then, the optimization problem becomes: L M : edge weighted among the marked pixels L U : edge weighted among the unmarked pixels

23. Kapitel 13 “Interactive Segmentation” – p. To minimize it, the equation is differentiated with respect to unknown X U: This is set to zero for finding the minumum of D(X U ), resulting a system of |V U | linear equations: X M : 1 for the label under consideration, and 0 for all other labels (remember: we need to solve X U for each label) 23 Random walker segmentation

24. Kapitel 13 “Interactive Segmentation” – p. 24 Random walker segmentation Cardiac segmentation across modalities

25. Kapitel 13 “Interactive Segmentation” – p. 25 Random walker segmentation Segmentation of objects with varying size, shape and texture

26. Kapitel 13 “Interactive Segmentation” – p. 26 Random walker segmentation Segmentation of natural images

27. Kapitel 13 “Interactive Segmentation” – p. 27 Random walker segmentation  Author webpage: http://cns.bu.edu/~lgrady  Random walker paper: http://cns.bu.edu/~lgrady/grady2006random.pdf  Random walker MATLAB code: http://cns.bu.edu/~lgrady/random_walker_matlab_code.zip  Random walker demo page: http://cns.bu.edu/~lgrady/Random_Walker_Image_Segmentation.html