1. Segmentation from Examples By: A’laa Kryeem Lecturer: Hagit Hel-Or

2. What is Segmentation from Examples ?  Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain  Effective when making a semantic segmentation

3. Why to use Examples  The example defines the granularity of the desired output  Give us the ability to characterize meaningful parts in the image  Using example allow us to use non-parametric model

4. The example defines the granularity of the desired output: Training image Test Image Desired Segmentation Induced Segmentation

5. Why to use Examples  The example defines the granularity of the desired output  Give us the ability to characterize meaningful parts in the image

6. Give us the ability to characterize meaningful parts in the image

7. Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications

8. Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications  Problems:  Meaningful parts are often too complex  Semantic interpretation is highly subjective, depending on both the application, and the user

9. Meaningful parts are often too complex

10. Semantic Segmentation from an example  We want to segment an image into semantically meaningful parts  Required in various applications  Problems:  Meaningful parts are often too complex  Semantic interpretation is highly subjective, depending on both the application, and the user

11. Example of image different segmentation

12. Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input

13. Training set

14. Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation

15. non-parametric model Each semantic part is represented by a set of square patches

16. Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments

17. Over segmented image

18. Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments  Compute costs for fragment-label pairs

19. (fragment,label) cost example ?

20. Semantic Segmentation from an example  So, How to achieve semantic segmentation  Getting segmented training image(s) as input  Using non-parametric representation  Over-segmenting the Test image into small fragments  Compute costs for fragment-label pairs  Graph-cuts multi-label optimization

21. Why do we need graph-cuts  Graph-cuts optimization is used to label each fragment in a globally optimal manner

22. Training set Test image Fragmentation Fragments Patch sets Classification Classification scores Graph-Cuts optimization Result

23. Over segmenting  Fragment: small arbitrarily-shaped and simply-connected pixel clusters  We assume that small homogeneous regions always belong to the same semantic part

24. Over segmenting  Fragment: small arbitrarily-shaped and simply connected pixel clusters  We assume that small homogeneous regions always belong to the same semantic part  Advantages:  Enforces a locally coherent labeling  Reduces the computational complexity

25. Graph-cuts multi-label optimization  For each fragment we have k cost values, we need to determine which one is the optimal  Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background, object)

26. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

27. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

28. Pixel labeling costs  Given I train and L train  Representing each label(segment) in L train by a set of square patches  We get k sets {S l } l=1,…,k one for each label

29. Pixel labeling costs (cont.)  Next, we define φ (p, l) for each (pixel,label) pair  The cost of assigning label l to pixel p I test P’S l p:pixel at I test l : label in L train ssd(P,P’) is the sum of squared distances between P,P’ M:mxmx3 P:mXm neighborhood centered at p P’:mxm patch

31. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

32. Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation

33. Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation  Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions)

34. Fragmentation  We partition I test into small,color- homogeneous regions using mean shift segmentation  Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions)  Fragment boundaries align with edges in the image

35. Fragmentation (cont.) Random colorizationDetailed close-up

36. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

37. Fragment labeling costs  Voting scheme in order to compute labeling costs of each fragment  For each fragment f I test we pick a few representative pixels: Rep(f)={p i f } i=I,…,R f R f is proportional to |f|

38. Fragment labeling costs (cont.)

39. Fragment labeling vs. pixel labeling

40.  Enforces a locally coherent labeling Training image Training seg. Input image Fragment labeling Pixel labeling

41. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

42. Graph-cuts optimization After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i fragment labeling costs. Costs range in the interval [0,1]

43. Graph-cuts optimization  Now for each pixel p I test we have a labeling cost  We need to find L test the globally optimal labeling  Requirements:  Minimizes the total labeling cost  Consistent with presence (or absence) of edges

44. Graph-cuts optimization (cont.)

46. Intuition:

48. Graph-cuts optimization (cont.)  Finally L test is determined by solving L test =min L E(L) Fragment labeling Labeling after Graph- Cuts Optimization

49. Multi-label graph-cut Colored nodes:labels Squares : fragments For each (fragment,label) pair we have an edge. Edge weigh according to φ. Edges between two squares weighed according to Ψ.

50. Multi-label graph-cut Induced graph Each fragment connected to a single label.

51. Multi-label graph-cut is NP-complete problem!

52. Training image Training segmentation Input image

53. Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

54. Algorithm results Training set a b c

55. Bear results  invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.  We can’t separate multiple objects belonging to the same label (c).

56. Algorithm results Training set a b c d

57. Summary

58. Thank You For Listening

59. References  Inducing Semantic Segmentation from an Example, Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski.  "Segmentation by Example“, Sameer Agarwal and Serge Belongie.  Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.  Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239