Segmentation Lucia Ballerini Digital Image Processing Lecture 8 Course book reading: GW 10

Image "What are the objects to be analyzed?" Pre-processing, image enhancement Segmentation Binary operations Morphological operations and feature extraction Classification and matching Image analysis Data

Segmentation ► ► Full segmentation: Individual objects are separated from the background and given individual ID numbers (labels). ► ► Partial segmentation: The amount of data is reduced (usually by separating objects from background) to speed up the further processing. ► ► Segmentation is often the most difficult problem to solve in the process; there is no universal solution! ► ► The problem can be made much easier if solved in cooperation with the constructor of the imaging system (choice of sensors, illumination, background etc).

Three types of segmentation ► ► Classification – Based on some similarity measure between pixel values. The simplest form is thresholding. ► ► Edge-based – Search for edges in the image. They are then used as borders between regions ► ► Region-based – Region growing, merge & split Common idea: search for discontinuities or/and similitudes in the image

Thresholding global or local ► ► global: based on some kind of histogram: grey-level, edge, feature etc.   Lighting conditions are extremely important, and it will only work under very controlled circumstances. ► ► Fixed thresholds: the same value is used in the whole image ► ► local (or dynamic thresholding): depends on the position in the image. The image is divided into overlapping sections which are thresholded one by one.

Classical automatic thresholding algorithm 1. Select an initial estimate for T 2. Segment the image using T. This produces 2 groups: G 1, pixels with value >T and G 2, with value T and G 2, with value <T 3. Compute µ 1 and µ 2, average pixel value of G 1 and G 2 4. New threshold: T=1/2(µ 1 +µ 2 ) 5. Repeat steps 2 to 4 until T stabilizes. ► Very easy + very fast ► Assumptions: normal dist. + low noise

Optimal Thresholding ► ► based on the shape of the current image histogram. Search for valleys, Gaussian distributions etc. Background Real histogram Optimal threshold ? Both Foreground

Histograms To love… …and to hate

Thresholding and illumination ► Solutions:  Calibration of the imaging system  percentile filter with very large mask  Morphological operators

MR non-uniformity median filteringthresholding -

More thresholding ► ► Can also be used on other kinds of histogram: grey-level, edge, feature etc. Multivariate data (  see next lectures) ► ► Problems:   Only considers the graylevel pixel value, so it can leave “holes” in segmented objects. ► ► Solution: post-processing with morphological operators   Requires strong assumptions to be efficient   Local thresholding is better  see region growing techniques

Edge-based Segmentation Based on finding discontinuities (local variations of image intensity) Apply an edge detector ex gradient operator (Sobel) second derivative (Laplace) Threshold the edge image to get a binary image Depending on the type of edge detector:   Link edges together to close shapes (using edge direction for ex)   Remove spurrious edges

Gradient based procedure Sobel

Zero-crossing based procedure LoG

Laplacian of Gaussian

Edge-based Segmentation: examples Prewitt: needs edge linkingCanny: needs “cleaning”

Region based segmentation ► ► Work by extending some region based on local similarities between pixels   region growing (bottom-up method)   region splitting and merging (top-down method)   Bottom-up: from data to representation   Top-down: from model to data

Region growing (bottom-up method) Find starting points Include neighbouring pixels with similar features (grey-level, texture, color) Continue untill all pixels have been included with one of the starting points. ► ► Problems:   Not trivial to find good strating points, difficult to automate   Need good criteria for similarity.

Watershed (a kind of region growing) ► ► Think of the grey-level image as a landscape. Let water rise from the bottom of each valley (the water from each valley is given its own label). As soon as the water from two valleys meet, build a dam, or watershed. These watersheds will then define the borders between different regions.

Example of watershed directly on a gray- level image

Example of Watershed on a binary image

Watershed: problems and solutions ► Oversegmentation  watershed from markers ► Computation  new algorithm for fast watershed ► Graylevel might not be the optimal choice as the local similarity measure  bigger neighborhood when growing  other local features (statistical,  other local features (statistical, edge enhanced image, distance transformed image…)

Split & Merge (top-down metod) Set up som criteria for what is a uniform area (ex mean, variance, bimodality of histogram, texture, etc…) Start with the full image and split it in to 4 sub-images Check each sub-image. If not uniform, divide into 4 new sub- images After each iteration, compare touching regions with neighboring regions end merge if uniform. The method is also called "quadtree" division (and is also used for compression)

Split & Merge

The Hough transform ► ► A method for finding global relationships between pixels. Example: We want to find straight lines in an image   1. Apply edge enhancing filter (ex Laplace)   2. Set a threshold for what filter response is considered a true ”edge pixel”   3. Extract the pixels that are on a straight line using the Hough transform original imageedge enhanced image thresholded edge image

The Hough transform Finding straight lines: ► ► 1. consider a pixel in position (xi,yi) ► ► 2. equation of a straight line yi=axi+b ► ► 3. set b=-axi+ yi and draw this (single) line in ”ab-space” ► ► 4. consider the next pixel with position (xj,yj) and draw the line b=-axj+ yj ”ab-space” (also called parameter space). The poins (a’,b’) where the two lines intersect represent the line y=a’x+b’ in ”xy-space” which will go through both (xi,yi) and (xj,yj). ► ► 5. draw the line in ab-space corresponding to each pixel in xy- space. ► ► 6. divide ab-space into accumulator cells and find most common (a’, b’) which will give the line connecting the largest number of pixels

y The Hough transform xy-space x ab- or parameter space b a

The Hough transform ► ► In reality we have a problem with y=ax+b because a reaches infinity for vertical lines. Use instead. ► ► It is common to use ”filters” for finding the intersection: ”butterfly filters” ► ► Different variations of the Hough transform can also be used for finding other shapes of the form g(v,c)=0, v is a vector of coordinates, c is a vector of coefficients. ► ► Possible to find any kind of simple shape ex. circle:(3D parameter space)

The Hough transform

Conclusions ► The segmentation procedure 1.Pre-processing 2.Segmentation 3.Post-processing  Like any IP procedure ► There exists NO universal segmentation method ► Evaluation of segmentation performance is important

Snakes ► Example: segmentation of the brain in MRI Snake after initializationSnake at equilibrium User interaction

Snakes (active contours) ► A snake is an active contour parametrically represented by its position v(s)=(x(s), y(s)) ► Each position is associated to an energy: ► The final position corresponds to the minimum of the energy

Internal Energy The internal energy of the snake is due to bending and it is associated with a priori constraints: The internal energy of the snake is due to bending and it is associated with a priori constraints: ►  (s) controls the tension of the contour ►  (s) controls its rigidity

External Energy ► The external energy depends on the image and accounts for a posteriori information ► Several energy forms have been proposed based on features of interest in the image ► An energy commonly used to attract snakes towards edges is:

Applications

Applications (by Terzopoulos)

Considerations ► The number of nodes is an important factor for the behavior of the snake. Ability to resample the contour may be necessary. ► If we want a closed contour, we set the first and the last point equal. ► Anchor points are necessary to keep the snake in position if the image forces are not enough. ► It may be necessary to allow a snake contour to divide into two contours, or two contours to merge into one contour. ► Different applications may need different potential functions and different settings of the control parameters (damping, tension and rigidity).

Applications ► Tracking of a moving object  An initial estimate for the contour (e.g. interactively defined) is used in the first frame.  The contour at equilibrium is used as the starting contour for the next frame. The snake locks on to the object. ► Reconstruction from serial sections  The user draws an approximate contour in the first slice.  The contour at equilibrium is used as the starting contour in the next slice.  The 3D object is reconstructed from the contours using triangulation. ► …..

More segmentation Important in Image Processing in general: “If you can use expert knowledge (user interaction, modelling,…) at relatively low cost (development, computational,…)” JUST DO IT!!