Segmentation from Examples By: A’laa Kryeem Lecturer: Hagit Hel-Or

What is Segmentation from Examples ?  Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain  Effective when making a semantic segmentation

Why to use Examples  The example defines the granularity of the desired output  Give us the ability to characterize meaningful parts in the image  Using example allow us to use non-parametric model

The example defines the granularity of the desired output: Training image Test Image Desired Segmentation Induced Segmentation

Why to use Examples  The example defines the granularity of the desired output  Give us the ability to characterize meaningful parts in the image

Give us the ability to characterize meaningful parts in the image

Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications

Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications  Problems:  Meaningful parts are often too complex  Semantic interpretation is highly subjective, depending on both the application, and the user

Meaningful parts are often too complex

Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications  Problems:  Meaningful parts are often too complex  Semantic interpretation is highly subjective, depending on both the application, and the user

Example of image different segmentation

Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input

Training set

Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation

non-parametric model Each semantic part is represented by a set of square patches

Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments

Over segmented image

Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments  Compute costs for fragment-label pairs

(fragment,label) cost example ?

Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments  Compute costs for fragment-label pairs  Graph-cuts multi-label optimization

Why do we need graph-cuts  Graph-cuts optimization is used to label each fragment in a globally optimal manner

Training set Test image Fragmentation Fragments Patch sets Classification Classification scores Graph-Cuts optimization Result

Over segmenting  Fragment: small arbitrarily-shaped and simply-connected pixel clusters  We assume that small homogeneous regions always belong to the same semantic part

Over segmenting  Fragment: small arbitrarily-shaped and simply connected pixel clusters  We assume that small homogeneous regions always belong to the same semantic part  Advantages:  Enforces a locally coherent labeling  Reduces the computational complexity

Graph-cuts multi-label optimization  For each fragment we have k cost values, we need to determine which one is the optimal  Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background, object)

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Pixel labeling costs  Given I train and L train  Representing each label(segment) in L train by a set of square patches  We get k sets {S l } l=1,…,k one for each label

Pixel labeling costs (cont.)  Next, we define φ (p, l) for each (pixel,label) pair  The cost of assigning label l to pixel p I test P’S l p:pixel at I test l : label in L train ssd(P,P’) is the sum of squared distances between P,P’ M:mxmx3 P:mXm neighborhood centered at p P’:mxm patch

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation

Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation  Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions)

Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation  Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions)  Fragment boundaries align with edges in the image

Fragmentation (cont.) Random colorizationDetailed close-up

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Fragment labeling costs  Voting scheme in order to compute labeling costs of each fragment  For each fragment f I test we pick a few representative pixels: Rep(f)={p i f } i=I,…,R f R f is proportional to |f|

Fragment labeling costs (cont.)

Fragment labeling vs. pixel labeling

 Enforces a locally coherent labeling Training image Training seg. Input image Fragment labeling Pixel labeling

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Graph-cuts optimization After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i fragment labeling costs. Costs range in the interval [0,1]

Graph-cuts optimization  Now for each pixel p I test we have a labeling cost  We need to find L test the globally optimal labeling  Requirements:  Minimizes the total labeling cost  Consistent with presence (or absence) of edges

Graph-cuts optimization (cont.)

Intuition:

Graph-cuts optimization (cont.)  Finally L test is determined by solving L test =min L E(L) Fragment labeling Labeling after Graph- Cuts Optimization

Multi-label graph-cut Colored nodes:labels Squares : fragments For each (fragment,label) pair we have an edge. Edge weigh according to φ. Edges between two squares weighed according to Ψ.

Multi-label graph-cut Induced graph Each fragment connected to a single label.

Multi-label graph-cut is NP-complete problem!

Training image Training segmentation Input image

Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

Algorithm results Training set a b c

Bear results  invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.  We can’t separate multiple objects belonging to the same label (c).

Algorithm results Training set a b c d

Summary

Thank You For Listening

References  Inducing Semantic Segmentation from an Example, Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski.  "Segmentation by Example“, Sameer Agarwal and Serge Belongie.  Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.  Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239