1. The University of Ontario CS 4487/9687 Algorithms for Image Analysis Combining Color and Boundary and other extensions (graph cuts)

2. The University of Ontario CS 4487/9687 Algorithms for Image Analysis Combining Color and Boundary n Color clustering/segmentation: thresholding mean-shift K-means n Boundary regularization: live-wire deformable contours, snakes where are we? + putting together

3. The University of Ontario CS 4487/9687 Algorithms for Image Analysis Combining Color and Boundary n Boundary regularization in N-dimensions (graph cuts): implicit vs explicit representation of boundaries image and volume segmentation multi-view reconstruction, video textures, … n Color + Boundary fixed appearance models model re-estimation, clustering with regularization n Graph cuts as energy minimization submodularity, geometric functionals, posterior energy (MRF) Extra Reading: Szeliski Sec 5.5 plan for this topic

4. The University of Ontario Shortest paths approach Graph cuts Shortest paths for segmentation in 2D Example: find the shortest closed contour in a given domain of a graph Compute the shortest path p ->p for a point p. p Graph Cuts approach Compute the minimum cut that separates red region from blue region Repeat for all points on the gray line. Then choose the optimal contour.

5. The University of Ontario Graph cuts for optimal boundary detection simple example [Boykov&Jolly, ICCV’01] n-links s t a cut hard constraint hard constraint Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms)

6. The University of Ontario Optimal boundary in 2D “max-flow = min-cut”

7. The University of Ontario Optimal boundary in 3D 3D bone segmentation (real time screen capture)

8. The University of Ontario Optimal boundary in 3D

9. The University of Ontario Cuts on directed graphs n Cost of a cut includes only edges from the source to the sink components n Cut’s cost (on a directed graph) changes if terminals are swapped Swapping terminals s and t is similar to switching surface orientation s Cut on a directed graph t

10. The University of Ontario Cuts on directed graphs

11. The University of Ontario Graph Cuts Basics ( see Cormen’s book) Simple 2D example s-t graph cut A graph with two terminals S and T “source” S T “sink” Cut cost is a sum of severed edge weights Minimum cost s-t cut can be found in polynomial time Goal: divide the graph into two parts separating red and blue nodes

12. The University of Ontario s/t min cut algorithms are widely studied (combinatorial optimization) n Augmenting paths [Ford & Fulkerson, 1962] n Push-relabel [Goldberg-Tarjan, 1986]

13. The University of Ontario “Augmenting Paths” n Find a path from S to T along non-saturated edges “source” A graph with two terminals S T “sink” n Increase flow along this path until some edge saturates

14. The University of Ontario “Augmenting Paths” n Find a path from S to T along non-saturated edges “source” A graph with two terminals S T “sink” n Increase flow along this path until some edge saturates n Find next path… n Increase flow…

15. The University of Ontario “Augmenting Paths” n Find a path from S to T along non-saturated edges “source” A graph with two terminals S T “sink” n Increase flow along this path until some edge saturates Iterate until … all paths from S to T have at least one saturated edge MAX FLOW  MIN CUT

16. The University of Ontario 16 Graph cut is an old standard problem with tons of applications outside vision From Harris & Ross [1955]

17. The University of Ontario 17 From Harris & Ross [1955] Graph cut is an old standard problem with tons of applications outside vision

18. The University of Ontario Applications of max-flow (min cut) algorithms n Matrix rounding n Perfect matching n Vertex cover n Routing (airline scheduling) n Baseball elimination n Economics (circulation–demand problem) n Computer vision

19. The University of Ontario Multi-view volumetric photo-reconstruction Calibrated images of Lambertian scene 3D model of scene CVPR’05 slides from Vogiatzis, Torr, Cippola

20. The University of Ontario first pass at multiview reconstruction: use silhouettes => Visual Hull n Cameras are calibrated Each point P in 3D space projects to a known pixel in i-th camera. n Assume that each camera knows object’s 2D silhouette How can one be obtained? n Project each camera’s silhouette into space to obtain a 3D cone. n Intersection of the cones generated by each image gives the visual hull of the object visual hull is the smallest 3D shape consistent with all silhouettes.

21. The University of Ontario Visual hull of a human face CVPR’05 slides from Vogiatzis, Torr, Cippola visual hull (from silhouettes)

22. The University of Ontario only over cameras i that “see” voxel P Can refine visual hull using photoconsistency CVPR’05 slides from Vogiatzis, Torr, Cippola photo-inconsistency for voxel P surface of a photo hull Find surface of low photo-inconsistency

23. The University of Ontario Estimating visibility CVPR’05 slides from Vogiatzis, Torr, Cippola 1. Get nearest point on outer surface 2. Use outer surface for occlusions 2. Discard occluded views only over cameras i that “see” voxel P

24. The University of Ontario Graph cuts applied to multi-view reconstruction CVPR’05 slides from Vogiatzis, Torr, Cippola Source Sink The cost of the cut integrates photoconsistency over the whole space

25. The University of Ontario Graph cuts applied to multi-view reconstruction CVPR’05 slides from Vogiatzis, Torr, Cippola visual hull (silhouettes) surface of good photoconsistency

26. The University of Ontario texture/features in 3D reconstruction camera B.. camera A unknown true surface photoconsistent voxels photoconsistent voxels photoconsistent voxels photoconsistent voxels surface reconstruction based on optimizing photoconsistency 3D volume where surface is being reconstructed (search space)

27. The University of Ontario texture/features in 3D reconstruction camera B.. camera A unknown true surface photoconsistent voxels photoconsistent voxels photoconsistent voxels photoconsistent voxels surface reconstruction based on optimizing photoconsistency 3D volume where surface is being reconstructed (search space) MORE REALISTIC RESULT BASED ON DATA NOISE, BLUR, ERRORS IN CAMERA CALIBRATION, AND “SHRINKING BIAS” (more later)

28. The University of Ontario Graph cuts for video textures Graph-cuts video textures (Kwatra, Schodl, Essa, Bobick 2003) Short video clip 2 1 Long video clip a cut 3D generalization of “image-quilting” (Efros & Freeman, 2001)

29. The University of Ontario Graph cuts for video textures original short clip synthetic infinite texture Graph-cuts video textures (Kwatra, Schodl, Essa, Bobick 2003)

30. The University of Ontario Graph Cut Segmentation: n implicit boundary representation n boundary smoothness = minimum “length”

31. The University of Ontario Explicit surface (boundary) representation via live-wire or snakes Any contour (or surface mesh in 3D) is explicitly represented… by a path (a set of connected edges)or by control points

32. The University of Ontario Implicit surface (boundary) representation via graph-cuts Any contour (or surface in 3D) satisfying labeling of exterior/interior points (pixel centers) is acceptable if some explicit surface has to be output. 0 1 1 1 1 1 0 0 0 0 0 0 segment region is represented explicitly boundary is implicit cost of discontinuities (weighted boundary length)

33. The University of Ontario Explicit vs implicit boundaries Live-wire Graph cuts approximates continuous contour functional contrast-weighted length the same geometric energy can be shown for graph cuts

34. The University of Ontario Indeed, the cost of severed n-links can be seen as a “metric” C n Cost of a cut can be interpreted as some geometric functional over implicitly represented segment (contour) n Properly selected edges/weights approximate boundary “length”

35. The University of Ontario Integral geometry C a set of all lines L a subset of lines L intersecting contour C Euclidean length of C : the number of times line L intersects C Cauchy-Crofton formula probability that a “randomly drown” line intersects C

36. The University of Ontario Graph cuts and integral geometry Boykov&Kolmogorov’03 C Euclidean length graph cut cost for edge weights: the number of edges of family k intersecting C Edges of any regular neighborhood system generate families of lines {,,, } Graph nodes are imbedded in R2 in a grid-like fashion

37. The University of Ontario Metrication errors 4-neighborhoods (Manhattan metric) equal perimeter shapes 1 1 larger-neighborhoods (Euclidean metric) ekek 8-neighborhoods

38. The University of Ontario Metrication errors equal perimeter shapes 1 1 4-neighborhoods (Manhattan metric) 8-neighborhoods larger-neighborhoods (Euclidean metric) ekek

39. The University of Ontario Metrication errors equal perimeter shapes 1 1 4-neighborhoods (Manhattan metric) 8-neighborhoods larger-neighborhoods (Euclidean metric) ekek

40. The University of Ontario Metrication artifacts (contrast-sensitive metric) 4-neighborhood 8-neighborhood

41. The University of Ontario Graph Cut Segmentation: combining (fixed) color appearance and boundary regularization

42. The University of Ontario Adding regional properties another segmentation example [Boykov&Jolly’01] n-links s t a cut t-link hard constrains are not required regional bias example suppose are given “expected” intensities of object and background

43. The University of Ontario Adding regional properties another segmentation example [Boykov&Jolly’01] a cut given object and background intensity histograms s t More generally, regional bias can be based on any intensity models of object and background

44. The University of Ontario Adding regional properties another segmentation example [Boykov&Jolly’01] a cut given object and background intensity histograms More generally, regional bias can be based on any intensity models of object and background

45. The University of Ontario Adding regional properties another segmentation example

46. The University of Ontario Background Subtraction ? - = I= I obj - I bkg Threshold intensities S={ p :  I p > T } T Adding regional properties another segmentation example

47. The University of Ontario - = I= I obj - I bkg Threshold intensities S={ p :  I p > T } optimal cut thresholding graph cuts Adding regional properties another segmentation example

48. The University of Ontario Adding regional properties another segmentation example [Boykov&Jolly’01] Appearance models and can also be obtained from user seeds

49. The University of Ontario Graph cuts as energy optimization for segmentation S n-links s t a cut t-link segmentation  cut S cost of severed t-links cost(cut) = + E(S)E(S) unary termspair-wise terms cost of severed n-links regional properties of S boundary smoothness for S [Boykov&Jolly 2001]

50. The University of Ontario Unary potentials as linear term wrt. unary terms linear term

51. The University of Ontario In general,… for binary variables pair-wise terms quadratic polynomial wrt. k -arity potentials are k -th order polynomial For example:

52. The University of Ontario That is… n Graph cut algorithm minimize certain 2nd-order energy of segmentation S=(S 1,…, S n ) for binary variables S p {0,1} Unlike DP, neighborhood N can be arbitrary, but, as discussed later in this topic, coefficients f pq must be negative (submodularity constraint)

53. The University of Ontario Graph cuts vs Thresholding S thresholding graph cut [BJ, 2001]

54. The University of Ontario Graph cuts vs Region Growing like “region growing”

55. The University of Ontario Graph cuts vs Region Growing like “region growing”

56. The University of Ontario Graph cuts vs Region Growing like “region growing”

57. The University of Ontario Graph cuts vs Region Growing like “region growing”

58. The University of Ontario Graph cuts iteration 2

59. The University of Ontario Graph cuts 2

60. The University of Ontario Graph cuts 2 Any paths would work, but shorter paths give faster algorithms (in theory and practice)

61. The University of Ontario Graph cuts 3

62. The University of Ontario Graph cuts 3 Finds optimal boundary (least number of holes) due to Energy minimization

63. The University of Ontario Image Segmentation: appearance models estimation

64. The University of Ontario Simple example (one color appearance) n-links s t a cut t-link regional bias example suppose are given “expected” intensities of object and background unary potentials pairwise potentials

65. The University of Ontario with regularization K-means energy “expected” intensities of object and background can be re-estimated Simple example (one color appearance) n-links s t a cut t-link extra variables

66. The University of Ontario Block-coordinate descent for n Minimize over labeling S for fixed I 0, I 1 n Minimize over I 0, I 1 for fixed labeling S fixed for S=const optimal S can be computed using graph cuts optimal I 1, I 0 can be computed by minimizing squared errors inside object and background segments mean colors in two segments

67. The University of Ontario Chan-Vese segmentation (binary case )

68. The University of Ontario Chan-Vese segmentation (could be used for more than 2 labels ) can be used for segmentation, to reduce color-depth, or to create a cartoon

69. The University of Ontario without the smoothing term, this is like “K-means” clustering in the color space Chan-Vese segmentation (could be used for more than 2 labels ) can be used for segmentation, to reduce color-depth, or to create a cartoon

70. The University of Ontario More general example (distribution-based appearance) Spatial smoothness (boundary) assuming known [Boykov&Jolly, ICCV2001] image segmentation, graph cut Log-Likelihoods (region) parametric models – e.g. Gaussian or GMM non-parametric models - histograms

71. The University of Ontario Beyond fixed appearance models Spatial smoothness (boundary) [Rother, et al. SIGGRAPH’2004] iterative image segmentation, Grabcut (block coordinate descent S ) Log-Likelihoods (region) Models  0,  1 can be iteratively re-estimated extra variables parametric models – e.g. Gaussian or GMM non-parametric models - histograms

72. The University of Ontario Minimize over segmentation S for fixed  0,  1 Minimize over  0,  1 for fixed labeling S Block-coordinate descent for fixed for S=const optimal S can be computed using graph cuts optimal  0,  1 can be computed by minimizing the sums of log-likelihoods distribution of intensities in current bkg. Segment ={p:S p =0} distribution of intensities in current obj. segment S={p:S p =1} (proof: see next page)

73. The University of Ontario Optimal intensity distribution  for fixed segment S S S1S1 SiSi S2S2 S3S3 S4S4 S5S5 pixiels of color bin i in S probability of color i in S  = { p 1, p 2,..., p n } easy to check minimize Lagrangian - histogram

74. The University of Ontario Iterative learning of color models (binary case ) n GrabCut: iterated graph cuts [Rother et al., SIGGRAPH 04] start from models  0,  1 based on colors inside and outside some given box iterate graph cut segmentation and model re-estimation until convergence

75. The University of Ontario Iterative learning of color models (could be used for more than 2 labels ) n Unsupervised segmentation [Zhu&Yuille, 1996] initialize models  0,  1 ,  2,  from randomly sampled boxes iterate segmentation and model re-estimation until convergence

76. The University of Ontario BCD minimization of converges to a local minimum E=2.37×10 6 E=2.41×10 6 E=1.39×10 6 E=1.410×10 6

77. The University of Ontario Q: why does this energy make sense with θ i as variables? Statistical answer: it gives maximum likelihood (ML) estimation of parameters θ i Information theoretic answer: this segmentation energy can be rewritten using entropies of intensities in object and background (…see next slides….)

78. The University of Ontario From model fitting (estimation) to entropy-based clustering

79. The University of Ontario Interpretation of log-likelihoods: entropy of segment intensities S S1S1 SiSi S2S2 S3S3 S4S4 S5S5 pixiels of color i in S probability of color i in S cross entropy of distribution p S (intensities in S ) w.r.t. distribution 

80. The University of Ontario Interpretation of log-likelihoods: entropy of segment intensities S S1S1 SiSi S2S2 S3S3 S4S4 S5S5 pixiels of color i in S probability of color i in S entropy of distribution p S (intensities in S )

81. The University of Ontario Interpretation of log-likelihoods: entropy of segment intensities information theoretic energy: Minimum Description Length (MDL) can be interpreted as the number of bits unsupervised image segmentation (like in Chan-Vese) high entropy segmentation break image into 2 coherent segments with low entropy of intensities S S low entropy segmentation S S S S S S

82. The University of Ontario Interpretation of log-likelihoods: entropy of segment intensities more general than Chan-Vese (colors can vary within each segment) break image into 2 coherent segments with low entropy of intensities S S S S

83. The University of Ontario Model Fitting and Color Clustering: Gaussian models vs Histograms normal distributions arbitrary histograms θ i variance clustering criteria (K-means) one color segments entropy clustering criteria “simpler” appearance segments

84. The University of Ontario Different interpretations of energy minimized by graph cuts

85. The University of Ontario Different interpretations of energy E minimized by graph cuts 1: posterior energy (see MAP-MRF in a later topic) 2: continuous surface functional (already discussed) 3: submodular set function (will discuss now) log-likelihoods log of prior regional term boundary length for factors

86. The University of Ontario Submodular functions n Edmonds 1970 Lattice - set of elements with inf and sup operations Function s called submodular if for any

87. The University of Ontario Lattice of sets and set functions Assume set Ω, then is a lattice of subsets S T Ω NOTE: if Ω is a set of pixels then any (binary) segmentation energy E(S) is a set function 2 Ω → R S Ω set function E(S): 2 Ω → R

88. The University of Ontario Submodular set functions Set function is submodular if for any Significance: any submodular set function can be globally optimized in polynomial time [Grotschel et al.1981,88, Schrijver 2000] S T Ω

89. The University of Ontario Submodular set functions Set function is submodular if for any S T Ω an alternative equivalent definition providing intuitive interpretation: “diminishing returns” v Easily follows from the previous definition: Significance: any submodular set function can be globally optimized in polynomial time [Grotschel et al.1981,88, Schrijver 2000]

90. The University of Ontario Submodular set functions Sets are conveniently represented by binary indicator variables Thus, set functions can be represented as Define, a restriction of S to any subset and consider projections of energy E onto subsets A Ω Set function E(S) is submodular iff for any pair

91. The University of Ontario Graph cuts for minimization of submodular set functions Assume set Ω and 2nd-order (quadratic) function Function E(S) is submodular if for any Significance: submodular 2 nd -order boolean (set) function can be globally optimized in polynomial time by graph cuts [Hammer 1968, Pickard&Ratliff 1973] Indicator variables [Boros&Hammer 2000, Kolmogorov&Zabih2003]

92. The University of Ontario Graph cuts for minimization of submodular set functions Assume set Ω and 2nd-order (quadratic) function Function E(S) is submodular if for any Example: quadratic function  S p S q is submodular iff Indicator variables

93. The University of Ontario Combinatorial optimization Continuous optimization Global Optimization submodularityconvexity ?

94. The University of Ontario Global Optimization f (x,y) =  ∙ xy for  0 1 1 y x A C D B EXAMPLE: 0 1 1 y x A C D B f (0,0) + f (1,1) ≤ f (0,1) + f (1,0) submodular binary energy convex continuous extension submodularityconvexity ?

95. The University of Ontario submodularityconvexity Global Optimization 0 1 1 y x A C D B f (0,0) + f (1,1) ≤ f (0,1) + f (1,0) 0 1 1 y x A C D B submodular binary energy concave continuous extension f (x,y) =  ∙ xy for  EXAMPLE:

96. The University of Ontario Graph cuts for approximating continuous surface functionals [Boykov&Kolmogorov, ICCV 2003, 2005] Geometric length any convex, symmetric metric g e.g. Riemannian Flux any vector field v Regional bias any scalar function f Submodular quadratic boolean functions on a grid can approximate continuous geometric functionals

97. The University of Ontario Discrete vs. continuous geometric functionals Geodesic active contours n both can incorporate basic segmentation cues image contrast (edge alignment) alignment (flux) regional bias (color appearance models) Graph cuts [Caselles,Kimmel,Sapiro 1995] [BJ 2001, BK 2003]

98. The University of Ontario Graph cuts for minimization of posterior energy (statistical inference) factors, terms Any set function defines Gibbs distribution over random variables Thus, submodularity implies a “field” of random variables ( Markov Random Field or MRF) with “smoothness” prior a subset of variables Theorem [Boykov, Delong, Kolmogorov, Veksler, an unpublished book 2016?] Any pair of random variables S p and S p are positively correlated iff set function is submodular

99. The University of Ontario binary image restoration spatial prior (regularization) log-likelihoods (data term) [Greig at. al., IJRSSB, 1989] unary factors pairwise factors restoration Graph cuts for minimization of posterior energy (statistical inference) observations

100. The University of Ontario spatial prior (regularization) image segmentation [BJ ICCV 2001] log-likelihoods (data term) Graph cuts for minimization of posterior energy (statistical inference) unary factors pairwise factors observations segmentation

101. The University of Ontario Beyond submodularity Deconvolution problem Binary image blurred with mean kernel QPBO TRWS Partial Enumeration+TRWS non-submodular quadratic energy for binary labels (original intensities) blurred non-binary intensities (observations) Exercise : check that this pairwise energy is non-submodular)

102. The University of Ontario Beyond submodularity (algorithms) n Many useful non-submodular set functions E(S) in binary segmentation with learned interaction potentials in the context of binary moves for multi-label (later) n QPBO (partial optimality) [survey Kolmogorov&Rother, 2007] n LP relaxations [Schlezinger, Komodakis, Kolmogorov, Savchinsky,…] n Message passing, e.g. LBP, TRWS [Kolmogorov] n Trust Region methods, e.g. FTR [Gorelick et al., 2012/13] n Partial enumeration, e.g. [Olsson et al. 2013] n very active area of research…

103. The University of Ontario Beyond pair-wise smoothness n Shape bias star-shape (one click) [Veksler 2008] shape statistics [Cremers 2003] box prior [Lempitsky 2009] n Curvature of the boundary (like bending in snakes) n Cardinality constraints n Sparsity, MDL prior, AIC, BIC, label costs n Many others…. E(S)= E 1 (S)+…+ E n (S)

104. The University of Ontario S S p p+ p- (1,0,1) configuration should cost infinity for any 3-factor (triplet) on straight intervals [Gorelick et al. ECCV 2014] Example: shape convexity Exercise : check that convexity constraint is non-submodular

105. The University of Ontario Example: shape convexity

106. The University of Ontario pair-wise smoothness (boundary length) shape convexity prior [Gorelick et al. ECCV 2014] Example: shape convexity

107. The University of Ontario Example: curvature regularization 3D shape reconstruction [Olsson et al. CVPR 2013] noisier result with common submodular pairwise smoothness

108. The University of Ontario [Marin et al. ICCV 2015] 3D reconstruction of heart vessels center lines using curvature regularization Example: curvature regularization

109. The University of Ontario Beyond linear combination of terms n Ratios are also used Normalized cuts [Shi, Malik, 2000] Minimum Ratio cycles [Jarmin Ishkawa, 2001] Ratio regions [Cox et al, 1996] Parametric max-flow applications [Kolmogorov et al 2007]

110. The University of Ontario Summary n thresholding, region growing n mean-shift, K-means n live-wire n snakes, active contours n graph cuts (binary labeling, MRF) covered algorithms: not covered: n ratio functionals n stectral methods (normalized cuts) n random walker n many others… n multi-label problems (image labeling) n applications beyond segmentation (stereo, motion, reconstruction, etc) to be covered later: