1. Segmentation Lucia Ballerini Digital Image Processing Lecture 8 Course book reading: GW 10

2. Image "What are the objects to be analyzed?" Pre-processing, image enhancement Segmentation Binary operations Morphological operations and feature extraction Classification and matching Image analysis Data

3. Segmentation ► ► Full segmentation: Individual objects are separated from the background and given individual ID numbers (labels). ► ► Partial segmentation: The amount of data is reduced (usually by separating objects from background) to speed up the further processing. ► ► Segmentation is often the most difficult problem to solve in the process; there is no universal solution! ► ► The problem can be made much easier if solved in cooperation with the constructor of the imaging system (choice of sensors, illumination, background etc).

4. Three types of segmentation ► ► Classification – Based on some similarity measure between pixel values. The simplest form is thresholding. ► ► Edge-based – Search for edges in the image. They are then used as borders between regions ► ► Region-based – Region growing, merge & split Common idea: search for discontinuities or/and similitudes in the image

5. Thresholding global or local ► ► global: based on some kind of histogram: grey-level, edge, feature etc.   Lighting conditions are extremely important, and it will only work under very controlled circumstances. ► ► Fixed thresholds: the same value is used in the whole image ► ► local (or dynamic thresholding): depends on the position in the image. The image is divided into overlapping sections which are thresholded one by one.

6. Classical automatic thresholding algorithm 1. Select an initial estimate for T 2. Segment the image using T. This produces 2 groups: G 1, pixels with value >T and G 2, with value T and G 2, with value <T 3. Compute µ 1 and µ 2, average pixel value of G 1 and G 2 4. New threshold: T=1/2(µ 1 +µ 2 ) 5. Repeat steps 2 to 4 until T stabilizes. ► Very easy + very fast ► Assumptions: normal dist. + low noise

7. Optimal Thresholding ► ► based on the shape of the current image histogram. Search for valleys, Gaussian distributions etc. Background Real histogram Optimal threshold ? Both Foreground

8. Histograms To love… …and to hate

9. Thresholding and illumination ► Solutions:  Calibration of the imaging system  percentile filter with very large mask  Morphological operators

10. MR non-uniformity median filteringthresholding -

11. More thresholding ► ► Can also be used on other kinds of histogram: grey-level, edge, feature etc. Multivariate data (  see next lectures) ► ► Problems:   Only considers the graylevel pixel value, so it can leave “holes” in segmented objects. ► ► Solution: post-processing with morphological operators   Requires strong assumptions to be efficient   Local thresholding is better  see region growing techniques

12. Edge-based Segmentation Based on finding discontinuities (local variations of image intensity) 1. 1. Apply an edge detector ex gradient operator (Sobel) second derivative (Laplace) 2. 2. Threshold the edge image to get a binary image 3. 3. Depending on the type of edge detector:   Link edges together to close shapes (using edge direction for ex)   Remove spurrious edges

13. Gradient based procedure Sobel

14. Zero-crossing based procedure LoG

15. Laplacian of Gaussian

16. Edge-based Segmentation: examples Prewitt: needs edge linkingCanny: needs “cleaning”

17. Region based segmentation ► ► Work by extending some region based on local similarities between pixels   region growing (bottom-up method)   region splitting and merging (top-down method)   Bottom-up: from data to representation   Top-down: from model to data

18. Region growing (bottom-up method) 1. 1. Find starting points 2. 2. Include neighbouring pixels with similar features (grey-level, texture, color). 3. 3. Continue untill all pixels have been included with one of the starting points. ► ► Problems:   Not trivial to find good strating points, difficult to automate   Need good criteria for similarity.

19. Watershed (a kind of region growing) ► ► Think of the grey-level image as a landscape. Let water rise from the bottom of each valley (the water from each valley is given its own label). As soon as the water from two valleys meet, build a dam, or watershed. These watersheds will then define the borders between different regions.

20. Example of watershed directly on a gray- level image

21. Example of Watershed on a binary image

22. Watershed: problems and solutions ► Oversegmentation  watershed from markers ► Computation  new algorithm for fast watershed ► Graylevel might not be the optimal choice as the local similarity measure  bigger neighborhood when growing  other local features (statistical,  other local features (statistical, edge enhanced image, distance transformed image…)

23. Split & Merge (top-down metod) 1. 1. Set up som criteria for what is a uniform area (ex mean, variance, bimodality of histogram, texture, etc…) 2. 2. Start with the full image and split it in to 4 sub-images. 3. 3. Check each sub-image. If not uniform, divide into 4 new sub- images. 4. 4. After each iteration, compare touching regions with neighboring regions end merge if uniform. The method is also called "quadtree" division (and is also used for compression)

24. Split & Merge

25. The Hough transform ► ► A method for finding global relationships between pixels. Example: We want to find straight lines in an image   1. Apply edge enhancing filter (ex Laplace)   2. Set a threshold for what filter response is considered a true ”edge pixel”   3. Extract the pixels that are on a straight line using the Hough transform original imageedge enhanced image thresholded edge image

26. The Hough transform Finding straight lines: ► ► 1. consider a pixel in position (xi,yi) ► ► 2. equation of a straight line yi=axi+b ► ► 3. set b=-axi+ yi and draw this (single) line in ”ab-space” ► ► 4. consider the next pixel with position (xj,yj) and draw the line b=-axj+ yj ”ab-space” (also called parameter space). The poins (a’,b’) where the two lines intersect represent the line y=a’x+b’ in ”xy-space” which will go through both (xi,yi) and (xj,yj). ► ► 5. draw the line in ab-space corresponding to each pixel in xy- space. ► ► 6. divide ab-space into accumulator cells and find most common (a’, b’) which will give the line connecting the largest number of pixels

27. y The Hough transform xy-space x ab- or parameter space b a

28. The Hough transform ► ► In reality we have a problem with y=ax+b because a reaches infinity for vertical lines. Use instead. ► ► It is common to use ”filters” for finding the intersection: ”butterfly filters” ► ► Different variations of the Hough transform can also be used for finding other shapes of the form g(v,c)=0, v is a vector of coordinates, c is a vector of coefficients. ► ► Possible to find any kind of simple shape ex. circle:(3D parameter space)

29. The Hough transform

30. Conclusions ► The segmentation procedure 1.Pre-processing 2.Segmentation 3.Post-processing  Like any IP procedure ► There exists NO universal segmentation method ► Evaluation of segmentation performance is important

31. Snakes ► Example: segmentation of the brain in MRI Snake after initializationSnake at equilibrium User interaction

32. Snakes (active contours) ► A snake is an active contour parametrically represented by its position v(s)=(x(s), y(s)) ► Each position is associated to an energy: ► The final position corresponds to the minimum of the energy

33. Internal Energy The internal energy of the snake is due to bending and it is associated with a priori constraints: The internal energy of the snake is due to bending and it is associated with a priori constraints: ►  (s) controls the tension of the contour ►  (s) controls its rigidity

34. External Energy ► The external energy depends on the image and accounts for a posteriori information ► Several energy forms have been proposed based on features of interest in the image ► An energy commonly used to attract snakes towards edges is:

35. Applications

36. Applications (by Terzopoulos)

37. Considerations ► The number of nodes is an important factor for the behavior of the snake. Ability to resample the contour may be necessary. ► If we want a closed contour, we set the first and the last point equal. ► Anchor points are necessary to keep the snake in position if the image forces are not enough. ► It may be necessary to allow a snake contour to divide into two contours, or two contours to merge into one contour. ► Different applications may need different potential functions and different settings of the control parameters (damping, tension and rigidity).

38. Applications ► Tracking of a moving object  An initial estimate for the contour (e.g. interactively defined) is used in the first frame.  The contour at equilibrium is used as the starting contour for the next frame. The snake locks on to the object. ► Reconstruction from serial sections  The user draws an approximate contour in the first slice.  The contour at equilibrium is used as the starting contour in the next slice.  The 3D object is reconstructed from the contours using triangulation. ► …..

39. More segmentation Important in Image Processing in general: “If you can use expert knowledge (user interaction, modelling,…) at relatively low cost (development, computational,…)” JUST DO IT!!