Segmentation Divide the image into segments. Each segment: Looks uniform Belongs to a single object. Have some uniform attributes. All the pixel related to it are connected. …

Main approaches Histogram-based segmentation Region-based segmentation Edge detection Region growing Region splitting and merging. Clustering K-means Mean shift Motion segmentation

Text & Background Problem: we are given an image of a paper, and we would like to extract the text from the image. Thresholding: define a threshold T such that each pixel x,y where I(x,y)<T is “text”. How do we determine the threshold ? Just choose 128 as a threshold (problematic for dark images) Use the median/mean (both are not good, as most of the paper is white)

Histogram-based threshold Compute the gray level histogram of the image. Find two “clusters”: black and white. Minimizing the L2 error: Select initial estimate T Segment the image using T. Compute the average gray level of each segment mb,mw Compute a new threshold value: T = ½ (mb+mw) Continue until convergence. We are already familiar with this algorithm !

Problems with this approach Noise Many holes and discontinuities in the segmentation. Changes in the illumination we do not use spatial information. Some of the problems can be solved using image processing techniques. For example, we can enhance the result using morphological operations. Yet – How can we overcome the changes in the illumination ?

Adaptive Thresholding Divide the image into sub-images. Assume that the illumination in each sub-images is constant. Use a different threshold for each sub-image. Alternatively – use a running window (and use the threshold of the window only for the central pixel ) Problems: Rapid illumination changes. Regions without text: we can try to recognize that these regions is unimodal.

Region-based segmentation We would like to use spatial information. We assume that neighboring pixels tend to belong to the same segment (not always true) Edge detection: looking for the boundaries of the segments. Problem: edges usually do not determine close contours. We can try to do it with edge linking (as in Canny’s edge detector)

Region-based segmentation Basic Formulation Let R represent the entire image region. Segmentation: Partitioning R into n subgroups Ri s.t: a) b) is a connected region c) d) e) P is the partition predicate

Region growing Choose a group of points as initial regions. Expand the regions to neighboring pixels using a heuristic: Color distance from the neighbors. The total error in the region (till a certain threshold): Variance Sum of the differences between neighbors. Maximal difference from a central pixel. In some cases, we can also use structural information: the region size and shape. In this way we can handle regions with a smoothly varying gray level or color. Question: How do we choose the starting points ? It is less important if we also can merge regions.

Region merging and splitting In region merging we start with small regions (it can be pixels), and iteratively merge regions which are similar. In region splitting, we start with the whole image, and split regions which are not uniform. These methods can be combined. Formally: Choose a predicate P. Split into disjoint regions any region Ri for which Merge any adjacent regions Ri and Rj for which Stop when no further merging and splitting is possible.

QuadTree R R21 R22 R1 R23 R24 R1 R2 R3 R4 R3 R4 R21 R22 R23 R24 With quadtree, one can use a variation of the split & merge scheme: Start with splitting regions. Only at the final stage: merge regions.

Segmentation as clustering Address the image as a set of points in the n-dimensional space: Gray level images: p=(x,y,I(x,y)) in R3 Color images: p =(x,y,R(x,y),G(x,y),B(x,y)) in R5 Texture: p= (x,y,vector_of_fetures) Color Histograms: p=(R(x,y),G(x,y),B(x,y)) in R3. we ignore the spatial information. From this stage, we forget the meaning of each coordinate. We deal with arbitrary set of points. Therefore, we first need to “normalize” the features (for example - convert a color image to the appropriate linear space representation)

Again, we can use splitting & merging Here, we merge each time the closest neighbors.

K-means Idea: Determine the number of clusters •Find the cluster centers and point-cluster correspondences to minimize error Problem: Exhaustive search is too expensive. Solution: We will use instead an iterative search. [Recall the ideal quantization procedure.] Algorithm – fix cluster centers; allocate points to closest cluster – fix allocation; compute best cluster centers Error function =

Example – clustering with K-means using gray-level and color histograms (from slides by D.A. forsyth)

Mean Shift K-means is a powerful and popular method for clustering. However: It assumes a pre-determined number of clusters It “likes” compact clusters. Sometimes, we are looking for long but continues clusters. Mean Shift: Determine a window size (usually small). For each point p: Compute a weighted mean of the shift in the window: Set p := p + m Continue until convergence. At the end, use a more standard clustering method.

Mean Shift (cont’) This method is based on the assumption that points are more and more dense as we are getting near the cluster “central mass”.

Motion segmentation Background subtraction: Assumes the existence of a dominant background. Optical flow (use the motion vectors as features) Multi model motion: Divide the image to layers such that in each layer, there exist a parametric motion model.