1. Segmentácia farebného obrazu

2. Image segmentation

4. Segmentation Segmentation means to divide up the image into a patchwork of regions, each of which is “homogeneous”, that is, the “same” in some sense - intensity, texture, colour, … The segmentation operation only subdivides an image; it does not attempt to recognize the segmented image parts!

5. Complete segmentation - divides an image into nonoverlapping regions that match to the real world objects. Cooperation with higher processing levels which use specific knowledge of the problem domain is necessary. Complete vs. partial segmentation

6. Partial segmentation- in which regions do not correspond directly with image objects. Image is divided into separate regions that are homogeneous with respect to a chosen property such as brightness, color, texture, etc.

7. Gestalt (celostné) laws of perceptual organization The emphasis in the Gestalt approach was on the configuration of the elements. Proximity: Objects that are closer to one another tend to be grouped together. Closure: Humans tend to enclose a space by completing a contour and ignoring gaps.

8. Gestalt laws of perceptual organization Similarity: Elements that look similar will be perceived as part of the same form. (color, shape, texture, and motion). Continuation: Humans tend to continue contours whenever the elements of the pattern establish an implied direction.

9. Gestalt laws A series of factors affect whether elements should be grouped together. A series of factors affect whether elements should be grouped together. Proximity: tokens that are nearby tend to be grouped. Proximity: tokens that are nearby tend to be grouped. Similarity: similar tokens tend to be grouped together. Similarity: similar tokens tend to be grouped together. Common fate: tokens that have coherent motion tend to be grouped together. Common fate: tokens that have coherent motion tend to be grouped together. Common region: tokens that lie inside the same closed region tend to be grouped together. Common region: tokens that lie inside the same closed region tend to be grouped together. Parallelism: parallel curves or tokens tend to be grouped together. Parallelism: parallel curves or tokens tend to be grouped together.

10. Closure: tokens or curves that tend to lead to closed curves tend to be grouped together. Closure: tokens or curves that tend to lead to closed curves tend to be grouped together. Symmetry: curves that lead to symmetric groups are grouped together. Symmetry: curves that lead to symmetric groups are grouped together. Continuity: tokens that lead to “continuous” curves tend to be grouped. Continuity: tokens that lead to “continuous” curves tend to be grouped. Familiar configuration: tokens that, when grouped, lead to a familiar object, tend to be grouped together. Familiar configuration: tokens that, when grouped, lead to a familiar object, tend to be grouped together. Gestalt laws

13. Consequence: Groupings by Invisible Completions * Images from Steve Lehar’s Gestalt papers: http://cns-alumni.bu.edu/pub/slehar/Lehar.html Stressing the invisible groupings:

14. Image segmentation Segmentation criteria: a segmentation is a partition of an image I into a set of regions S satisfying: Segmentation criteria: a segmentation is a partition of an image I into a set of regions S satisfying: 1.  S i = SPartition covers the whole image. 2. S i  S j = , i  jNo regions intersect. 3.  S i, P(S i ) = trueHomogeneity predicate is satisfied by each region. 4. P(S i  S j ) = false,Union of adjacent regions i  j, S i adjacent S j does not satisfy it.

15. Image segmentation So, all we have to do is to define and implement the similarity predicate. But, what do we want to be similar in each region? But, what do we want to be similar in each region? Is there any property that will cause the regions to be meaningful objects? Is there any property that will cause the regions to be meaningful objects?

16. Segmetnation methods Pixel-based Histogram Clustering Region-based Region growing Split and merge Edge-based Model-based Physics-based Graph-based

17. Histogram-based segmentation How many “orange” pixels are in this image? How many “orange” pixels are in this image? This type of question can be answered by looking at the histogram. This type of question can be answered by looking at the histogram.

18. Histogram-based segmentation How many modes are there? How many modes are there? Solve this by reducing the number of colors to K and mapping each pixel to the closest color. Solve this by reducing the number of colors to K and mapping each pixel to the closest color. Here’s what it looks like if we use two colors. Here’s what it looks like if we use two colors.

19. Clustering-based segmentation How to choose the representative colors? How to choose the representative colors? This is a clustering problem! This is a clustering problem! K-means algorithm can be used for clustering. K-means algorithm can be used for clustering.

20. Clustering-based segmentation K-means clustering of color.

21. Clustering-based segmentation K-means clustering of color.

22. 22 K-means clustering using intensity alone and color alone Image Clusters on intensity Clusters on color * From Marc Pollefeys COMP 256 2003 Results of K-Means Clustering:

23. Clustering-based segmentation Clustering can also be used with other features (e.g., texture) in addition to color. Clustering can also be used with other features (e.g., texture) in addition to color. Original ImagesColor Regions Texture Regions

24. Clustering-based segmentation K-means variants: K-means variants: Different ways to initialize the means. Different ways to initialize the means. Different stopping criteria. Different stopping criteria. Dynamic methods for determining the right number of clusters (K) for a given image. Dynamic methods for determining the right number of clusters (K) for a given image. Problem: histogram-based and clustering- based segmentation can produce messy regions. Problem: histogram-based and clustering- based segmentation can produce messy regions. How can these be fixed? How can these be fixed?

25. Clustering-based segmentation Expectation-Maximization (EM) algorithm can be used as a probabilistic clustering method where each cluster is modeled using a Gaussian. Expectation-Maximization (EM) algorithm can be used as a probabilistic clustering method where each cluster is modeled using a Gaussian. The clusters are updated iteratively by computing the parameters of the Gaussians. The clusters are updated iteratively by computing the parameters of the Gaussians. Example from the UC Berkeley’s Blobworld system.

26. Clustering-based segmentation Examples from the UC Berkeley’s Blobworld system.

27. Region growing Region growing techniques start with one pixel of a potential region and try to grow it by adding adjacent pixels till the pixels being compared are too dissimilar. Region growing techniques start with one pixel of a potential region and try to grow it by adding adjacent pixels till the pixels being compared are too dissimilar. The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen from the image. The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen from the image. Usually a statistical test is used to decide which pixels can be added to a region. Usually a statistical test is used to decide which pixels can be added to a region. Region is a population with similar statistics. Region is a population with similar statistics. Use statistical test to see if neighbor on border fits into the region population. Use statistical test to see if neighbor on border fits into the region population.

28. Region growing image segmentation

29. Split-and-merge 1. Start with the whole image. 2. If the variance is too high, break into quadrants. 3. Merge any adjacent regions that are similar enough. 4. Repeat steps 2 and 3, iteratively until no more splitting or merging occur.  Idea: good Results: blocky

30. Split-and-merge

31. Split-and-merge

32. Split-and-merge A large connected region formed by merging pixels labeled as residential after classification. A satellite image.More compact sub-regions after the split-and-merge procedure.

33. Mean Shift Segmentation http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

34. 34 Mean Shift Algorithm 1.Choose a search window size. 2.Choose the initial location of the search window. 3.Compute the mean location (centroid of the data) in the search window. 4.Center the search window at the mean location computed in Step 3. 5.Repeat Steps 3 and 4 until convergence. The mean shift algorithm seeks the “mode” or point of highest density of a data distribution:

35. Mean Shift Setmentation Algorithm 1.Convert the image into tokens (via color, gradients, texture measures etc). 2.Choose initial search window locations uniformly in the data. 3.Compute the mean shift window location for each initial position. 4.Merge windows that end up on the same “peak” or mode. 5.The data these merged windows traversed are clustered together. *Image From: Dorin Comaniciu and Peter Meer, Distribution Free Decomposition of Multivariate Data, Pattern Analysis & Applications (1999)2:22–30 Mean Shift Segmentation

36. 36 Mean Shift Segmentation Extension Gary Bradski’s internally published agglomerative clustering extension: Mean shift dendrograms 1.Place a tiny mean shift window over each data point 2.Grow the window and mean shift it 3.Track windows that merge along with the data they transversed 4.Until everything is merged into one cluster Is scale (search window size) sensitive. Solution, use all scales: Best 4 clusters:Best 2 clusters: Advantage over agglomerative clustering: Highly parallelizable

37. Mean Shift Segmentation Results: http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

38. Graph Cut

39. First: select a region of interest Graph Cut

40. How to select the object automatically? ? Graph Cut

41. What are graphs? Nodes usually pixels sometimes samples Edges weights associated (W(i,j)) E.g. RGB value difference

42. Graph Cut What are cuts? Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)|

43. Graph Cut Go back to our selected region Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)|

44. Graph Cut Go back to our selected region Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)|

45. Graph Cut We want highest sum of weights Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)|

46. Graph Cut We want highest sum of weights Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)| These cuts give low points W(i,j) = |RGB(i) – RGB(j)|is low

47. Graph Cut We want highest sum of weights Each “cut” -> points, W(I,j) Optimization problem W(i,j) = |RGB(i) – RGB(j)| These cuts give high points W(i,j) = |RGB(i) – RGB(j)|is high

48. Graph-based segmentation An image is represented by a graph whose nodes are pixels or small groups of pixels. An image is represented by a graph whose nodes are pixels or small groups of pixels. The goal is to partition the nodes into disjoint sets so that the similarity within each set is high and across different sets is low. The goal is to partition the nodes into disjoint sets so that the similarity within each set is high and across different sets is low. http://www.cs.berkeley.edu/~malik/papers/SM-ncut.pdf

49. Graph-based segmentation Let G = (V,E) be a graph. Each edge (u,v) has a weight w(u,v) that represents the similarity between u and v. Let G = (V,E) be a graph. Each edge (u,v) has a weight w(u,v) that represents the similarity between u and v. Graph G can be broken into 2 disjoint graphs with node sets A and B by removing edges that connect these sets. Graph G can be broken into 2 disjoint graphs with node sets A and B by removing edges that connect these sets. Let cut(A,B) =  w(u,v). Let cut(A,B) =  w(u,v). One way to segment G is to find the minimal cut. One way to segment G is to find the minimal cut. u  A, v  B

50. Graph-based segmentation

51. Minimal cut favors cutting off small node groups, so Shi and Malik proposed the normalized cut. Minimal cut favors cutting off small node groups, so Shi and Malik proposed the normalized cut. cut(A,B) cut(A,B) Ncut(A,B) = --------------- + --------------- assoc(A,V) assoc(B,V) assoc(A,V) =  w(u,t) u  A, t  V How much is A connected to the graph as a whole Normalized cut

52. Solve for Minimum Penalty Partition A Partition B cut

53. Graph-based segmentation 2 22 22 4 1 3 2 22 3 2 2 2 1 3 3 Ncut(A,B) = ------- + ------ 21 16 A B

54. Graph Cut Optimization solver Solver Example Recursion: 1.Grow 2.If W(i,j) low 1.Stop 2.Continue

55. Graph Cut Result : Isolation

56. Image Segmentation and Graph Cuts Image Segmentation Graph Cuts

57. The Pipeline Assign W(i,j) Solve for minimum penalty Cut into 2 Subdivide? Yes No Input: Image Output: Segments Each iteration cuts into 2 pieces

58. Assign W(i,j) W(i,j) = |RGB(i) – RGB(j)| is noisy! Could use brightness and locality X(i) is the spatial location of node i F(i) is the feature vector for node i which can be intensity, color, texture, motion… The formula is set up so that w(i,j) is 0 for nodes that are too far apart.

59. Graph-based segmentation Shi and Malik turned graph cuts into an eigenvector/eigenvalue problem. Shi and Malik turned graph cuts into an eigenvector/eigenvalue problem. Set up a weighted graph G=(V,E). Set up a weighted graph G=(V,E). V is the set of (N) pixels. V is the set of (N) pixels. E is a set of weighted edges (weight w ij gives the similarity between nodes i and j). E is a set of weighted edges (weight w ij gives the similarity between nodes i and j). Length N vector d: d i is the sum of the weights from node i to all other nodes. Length N vector d: d i is the sum of the weights from node i to all other nodes. N x N matrix D: D is a diagonal matrix with d on its diagonal. N x N matrix D: D is a diagonal matrix with d on its diagonal. N x N symmetric matrix W: W ij = w ij. N x N symmetric matrix W: W ij = w ij.

60. Graph-based segmentation Let x be a characteristic vector of a set A of nodes. Let x be a characteristic vector of a set A of nodes. x i = 1 if node i is in a set A x i = 1 if node i is in a set A x i = -1 otherwise x i = -1 otherwise Let y be a continuous approximation to x Let y be a continuous approximation to x Solve the system of equations (D – W) y = D y for the eigenvectors y and eigenvalues. Solve the system of equations (D – W) y = D y for the eigenvectors y and eigenvalues. Use the eigenvector y with second smallest eigenvalue to bipartition the graph (y  x  A). Use the eigenvector y with second smallest eigenvalue to bipartition the graph (y  x  A). If further subdivision is merited, repeat recursively. If further subdivision is merited, repeat recursively.

61. Extensions: Edge Weights How to calculate the edge weights? Intensity Color (HSV) Texture

62. Continued Work: Semantic Segmentation Incorporating top-down information into low-level segmentation Interactive Graph Cuts: Yuri Boykov, et al

63. Graph-based segmentation

65. GrabCut GrabCut User Input Result Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut Rother et al 2004 Regions Boundary Regions & Boundary Slides C Rother et al., Microsoft Research, Cambridge

66. Data Term Gaussian Mixture Model (typically 5-8 components) Foreground & Background Background G R Slides C Rother et al., Microsoft Research, Cambridge

67. Smoothness term An object is a coherent set of pixels Iterate until convergence: 1. Compute a configuration given the mixture model. (E-Step) 2. Compute the model parameters given the configuration. (M-Step) Slides C Rother et al., Microsoft Research, Cambridge

69. Moderately simple examples Moderately simple examples … GrabCut completes automatically Slides C Rother et al., Microsoft Research, Cambridge

70. Difficult Examples Difficult Examples Camouflage & Low Contrast No telepathy Fine structure Initial Rectangle Result Slides C Rother et al., Microsoft Research, Cambridge

71. CS 534 – Stereo Imaging - 71 Markov Random Fields (MRF) A graphical model for describing spatial consistency in images A graphical model for describing spatial consistency in images Suppose you want to label image pixels with some labels {l 1,…,l k }, e.g., segmentation, stereo disparity, foreground-background, etc. Suppose you want to label image pixels with some labels {l 1,…,l k }, e.g., segmentation, stereo disparity, foreground-background, etc. Ref: 1. S. Z. Li. Markov Random Field Modeling in Image Analysis. Springer-Verlag, 1991 2. S. Geman and D. Geman. Stochastic relaxation, gibbs distribution and bayesian restoration of images. PAMI, 6(6):721–741, 1984. From Slides by S. Seitz - University of Washington

72. CS 534 – Stereo Imaging - 72 Definition MRF Components: A set of sites: P={1,…,m} : each pixel is a site. A set of sites: P={1,…,m} : each pixel is a site. Neighborhood for each pixel N={N p | p  P} Neighborhood for each pixel N={N p | p  P} A set of random variables (random field), one for each site F={F p | p  P} Denotes the label at each pixel. A set of random variables (random field), one for each site F={F p | p  P} Denotes the label at each pixel. Each random variable takes a value f p from the set of labels L={l 1,…,l k } Each random variable takes a value f p from the set of labels L={l 1,…,l k } We have a joint event {F 1 =f 1,…, F m =f m }, or a configuration, abbreviated as F=f We have a joint event {F 1 =f 1,…, F m =f m }, or a configuration, abbreviated as F=f The joint prob. Of such configuration: Pr(F=f) or Pr(f) The joint prob. Of such configuration: Pr(F=f) or Pr(f) From Slides by S. Seitz - University of Washington

73. CS 534 – Stereo Imaging - 73 Definition MRF Components: Pr(f i ) > 0 for all variables f i. Pr(f i ) > 0 for all variables f i. Markov Property: Each Random variable depends on other RVs only through its neighbors. Pr(f p | f S-{p} )=Pr (f p |f Np ),  p Markov Property: Each Random variable depends on other RVs only through its neighbors. Pr(f p | f S-{p} )=Pr (f p |f Np ),  p So, we need to define a neighborhood system: N p (neighbors for site p). So, we need to define a neighborhood system: N p (neighbors for site p). No strict rules for neighborhood definition. No strict rules for neighborhood definition. Cliques for this neighborhood From Slides by S. Seitz - University of Washington

74. CS 534 – Stereo Imaging - 74 Definition MRF Components: The joint prob. of such configuration: The joint prob. of such configuration: Pr(F=f) or Pr(f) Markov Property: Each Random variable depends on other RVs only through its neighbors. Pr(f p | f S-{p} )=Pr (f p |f Np ),  p Markov Property: Each Random variable depends on other RVs only through its neighbors. Pr(f p | f S-{p} )=Pr (f p |f Np ),  p So, we need to define a neighborhood system: N p (neighbors for site p) So, we need to define a neighborhood system: N p (neighbors for site p) Hammersley-Clifford Theorem:Pr(f)  exp(-  C V C (f)) Sum over all cliques in the neighborhood system V C is clique potential We may decide 1. NOT to include all cliques in a neighborhood; or 2. Use different V c for different cliques in the same neighborhood From Slides by S. Seitz - University of Washington

75. CS 534 – Stereo Imaging - 75 Optimal Configuration MRF Components: Hammersley-Clifford Theorem: Hammersley-Clifford Theorem: Pr(f)  exp(-  C V C (f)) Consider MRF’s with arbitrary cliques among neighboring pixels Consider MRF’s with arbitrary cliques among neighboring pixels Sum over all cliques in the neighborhood system V C is clique potential: prior probability that elements of the clique C have certain values Typical potential: Potts model: From Slides by S. Seitz - University of Washington

76. CS 534 – Stereo Imaging - 76 Optimal Configuration MRF Components: Hammersley-Clifford Theorem: Hammersley-Clifford Theorem: Pr(f)  exp(-  C V C (f)) Consider MRF’s with clique potentials of pairs of neighboring pixels Consider MRF’s with clique potentials of pairs of neighboring pixels Most commonly used….very popular in vision. Energy function: There are two constraints to satisfy: 1.Data Constraint: Labeling should reflect the observation. 2.Smoothness constraint: Labeling should reflect spatial consistency (pixels close to each other are most likely to have similar labels).

77. CS 534 – Stereo Imaging - 77 Probabilistic interpretation The problem is we are not observing the labels but we observe something else that depends on these labels with some noise (eg intensity or disparity) The problem is we are not observing the labels but we observe something else that depends on these labels with some noise (eg intensity or disparity) At each site we have an observation i p At each site we have an observation i p The observed value at each site depends on its label: the prob. of certain observed value given certain label at site p : g(i p,f p )=Pr(i p |F p =f p ) The observed value at each site depends on its label: the prob. of certain observed value given certain label at site p : g(i p,f p )=Pr(i p |F p =f p ) The overall observation prob. given the labels: Pr(O|f) The overall observation prob. given the labels: Pr(O|f) We need to infer about the labels We need to infer about the labels given the observation Pr(f|O)  Pr(O|f) Pr(f)

78. Using MRFs How to model different problems? How to model different problems? Given observations y, and the parameters of the MRF, how to infer the hidden variables, x? Given observations y, and the parameters of the MRF, how to infer the hidden variables, x? How to learn the parameters of the MRF? How to learn the parameters of the MRF?

79. Modeling image pixel labels as MRF MRF-based segmentation 1 real image label image Slides by R. Huang – Rutgers University

80. MRF-based segmentation Classifying image pixels into different regions under the constraint of both local observations and spatial relationships Classifying image pixels into different regions under the constraint of both local observations and spatial relationships Probabilistic interpretation: Probabilistic interpretation: region labels image pixels model param. Slides by R. Huang – Rutgers University

81. Model joint probability label image label-label compatibility Function enforcing Smoothness constraint neighboring label nodes local Observations image-label compatibility Function enforcing Data Constraint region labels image pixels model param. How did we factorize? Slides by R. Huang – Rutgers University

82. CS 534 – Stereo Imaging - 82 Probabilistic interpretation We need to infer about the labels given the observation We need to infer about the labels given the observation Pr( f | O )  Pr(O|f ) Pr(f) Pr( f | O )  Pr(O|f ) Pr(f) MAP estimate of f should minimize the posterior energy Data (observation) term: Data Constraint Neighborhood term: Smoothness Constraint

83. MRF-based segmentation EM algorithm E-Step: (Inference) E-Step: (Inference) M-Step: (learning) M-Step: (learning) Applying and learning MRF Pseduo-likelihood method. Slides by R. Huang – Rutgers University

84. Applying and learning MRF: Example Slides by R. Huang – Rutgers University