1. ICCV Tutorial 2007 Philip Torr Papers, presentations and videos on web.....

2. Overview Recycling Solutions Illustrate on 2D graph cuts, generalization to 3D easy

3. MRF for Image & Video Segmentation Energy MRF Pair-wise Terms MAP Solution Unary likelihoodData (D) Unary likelihoodContrast TermUniform Prior (Potts Model) Seminal Work of Boykov and Jolly [ICCV 2001] Maximum-a-posteriori (MAP) solution x* = arg min E(x) x =

4. Dynamic Graph Cuts PBPB SBSB cheaper operation computationally expensive operation Simpler problem P B* differences between A and B similar PAPA SASA solve

5. First segmentation problem MAP solution GaGa Our Algorithm GbGb second segmentation problem Maximum flow residual graph ( G r ) G` difference between G a and G b updated residual graph

6. Energy Minimization using Graph cuts E MRF (a 1,a 2 ) Sink (0) Source (1) a1a1 a2a2 What really happens? Building the graph

7. Energy Minimization using Graph cuts If we cut a link to source or sink this indicates label Sink (0) Source (1) a1a1 a2a2 What really happens? Building the graph

8. Energy Minimization using Graph cuts Sink (0) Source (1) a1a1 a2a2 E MRF (a 1,a 2 ) = 2a 1 2 What really happens? Building the graph

9. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 Sink (0) Source (1) a1a1 a2a2 2 5

10. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4

11. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2

12. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2 1

13. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2 1 n-edges (pair-wise term) t-edges (unary terms)

14. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 1 E MRF (1,1) = 11 Cost of st-cut = 11

15. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8

16. Energy Minimization using Graph cuts What really happens? Building the graph E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Sink (0) Source (1) a1a1 a2a2 2 5 9 4 2 1 a 1 = 1 a 2 = 0 E MRF (1,0) = 8 Cost of st-cut = 8 + CONSTANT TERM K

17. Energy Minimization using Graph cuts Posiform E MRF (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 + CONSTANT TERM K Posiform is a multilinear polynomials in binary variables with positive coefficients

18. The Max-flow Problem - Edge capacity and flow balance constraints Computing the st-mincut from Max-flow algorithms Notation - Residual capacity (edge capacity – current flow) - Augmenting path Simple Augmenting Path based Algorithms - Repeatedly find augmenting paths and push flow. - Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem] Sink (1) Source (0) a1a1 a2a2 2 5 9 4 2 1

19. 9 + α 4 + α Adding a constant to both the t-edges of a node does not change the edges constituting the st-mincut. Key Observation Sink (1) Source (0) a1a1 a2a2 2 5 2 1 E (a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 E*(a 1,a 2 ) = E(a 1,a 2 ) + α(a 2 +ā 2 ) = E(a 1,a 2 ) + α [a 2 +ā 2 =1] Reparametrization

20. 9 + α 4 All reparametrizations of the graph are sums of these two types. Other type of reparametrization Sink (1) Source (0) a1a1 a2a2 2 5 + α 2 + α 1 - α Reparametrization, second type Both maintain the solution and add a constant α to the energy.

21. Reparametrization Nice result (easy to prove) All other reparametrizations can be viewed in terms of these two basic operations. Proof in Hammer, and also in one of Vlad’s recent papers.

22. s G t original graph 0/9 0/7 0/5 0/20/4 0/1 xixi xjxj flow/residual capacity Graph Re-parameterization

23. s G t original graph 0/9 0/7 0/5 0/20/4 0/1 xixi xjxj flow/residual capacity Graph Re-parameterization t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 st-mincut Compute Maxflow GrGr Edges cut

24. Update t-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0

25. Update t-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 capacity changes from 7 to 4

26. Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 5/-1 3/2 1/0 2/04/0 capacity changes from 7 to 4 edge capacity constraint violated! (flow > capacity) = 5 – 4 = 1 excess flow (e) = flow – new capacity

27. add e to both t-edges connected to node i Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 3/2 1/0 2/04/0 capacity changes from 7 to 4 edge capacity constraint violated! (flow > capacity) = 5 – 4 = 1 excess flow (e) = flow – new capacity 5/-1

28. Update t-edge Capacities s G` t updated residual graph xixi xjxj 0/12 3/2 1/0 4/0 capacity changes from 7 to 4 excess flow (e) = flow – new capacity add e to both t-edges connected to node i = 5 – 4 = 1 5/0 2/1 edge capacity constraint violated! (flow > capacity)

29. Update n-edge Capacities s GrGr t residual graph xixi xjxj 0/12 5/2 3/2 1/0 2/04/0 Capacity changes from 5 to 2

30. Update n-edge Capacities s t Updated residual graph xixi xjxj 0/12 5/2 3/-1 1/0 2/04/0 G` Capacity changes from 5 to 2 - edge capacity constraint violated!

31. Update n-edge Capacities s t Updated residual graph xixi xjxj 0/12 5/2 3/-1 1/0 2/04/0 G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint

32. Update n-edge Capacities s t Updated residual graph xixi xjxj 0/11 5/2 2/0 1/0 2/04/0 excess deficiency G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance!

33. Update n-edge Capacities s t Updated residual graph xixi xjxj 0/11 5/2 2/0 1/0 2/04/0 deficiency excess G` Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to/from the terminals Create capacity by adding α = excess to both t-edges.

34. Update n-edge Capacities Updated residual graph Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to the terminals Create capacity by adding α = excess to both t-edges. G` xixi xjxj 0/11 5/3 2/0 3/04/1 s t

35. Update n-edge Capacities Updated residual graph Capacity changes from 5 to 2 - edge capacity constraint violated! Reduce flow to satisfy constraint - causes flow imbalance! Push excess flow to the terminals Create capacity by adding α = excess to both t-edges. G` xixi xjxj 0/11 5/3 2/0 3/04/1 s t

36. Complexity analysis of MRF Update Operations MRF Energy Operation Graph OperationComplexity modifying a unary term modifying a pair-wise term adding a latent variable delete a latent variable Updating a t-edge capacity Updating a n-edge capacity adding a node set the capacities of all edges of a node zero O(1) O(k)* *requires k edge update operations where k is degree of the node

37. Finding augmenting paths is time consuming. Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004] -Reuses search trees after each augmentation. -Empirically shown to be substantially faster. Our Idea –Reuse search trees from previous graph cut computation –Saves us search tree creation tree time [O(#edges)] –Search trees have to be modified to make them consistent with new graphs – Constrain the search of augmenting paths New paths must contain at least one updated edge Improving the Algorithm

38. Reusing Search Trees c’ = measure of change in the energy –Running time Dynamic algorithm (c’ + re-create search tree ) Improved dynamic algorithm (c’) Video Segmentation Example - Duplicate image frames (No time is needed)

39. Experimental Analysis additional segmentation cues user segmentation cues static: 175 msec dynamic : 80 msec dynamic (optimized): 15 msec static : 175 msec Interactive Image segmentation (update unary terms) Energy MRF =

40. Experimental Analysis Image resolution: 720x576 static: 220 msec dynamic (optimized): 50 msec Image segmentation in videos (unary & pairwise terms) Graph CutsDynamic Graph Cuts Energy MRF =

41. Experimental Analysis Image resolution: 720x576 static: 177 msec dynamic (optimized): 60 msec Image segmentation in videos (unary & pairwise terms) Graph CutsDynamic Graph Cuts Energy MRF =

42. Experimental Analysis MRF consisting of 2x10 5 latent variables connected in a 4-neighborhood. Running time of the dynamic algorithm

43. Some Conclusions An exact dynamic algorithm for computing st-mincut - handles arbitrary changes in the graph Rapidly solves dynamic sub-modular energy functions Running time roughly proportional to the number of changes in the energy terms.

44. ObjCut and PoseCut: Combination of Top Down and Bottom Up Cues.. Classical Vision Problem. Combine theory of MRF’s with Object Recognition

45. Objective ImageSegmentationPose Estimate?? Aim to get a clean segmentation of a human…

46. MRF for Interactive Image Segmentation, Boykov and Jolly [ICCV 2001] Energy MRF Pair-wise Terms MAP Solution Unary likelihoodData (D) Unary likelihoodContrast TermUniform Prior (Potts Model) Maximum-a-posteriori (MAP) solution x* = arg min E(x) x =

47. However… This energy formulation rarely provides realistic (target- like) results.

48. Shape-Priors and Segmentation Combine object detection with segmentation –Obj-Cut, Kumar et al., CVPR ’05 –Zhao and Davis, ICCV ’05 Obj-Cut –Shape-Prior: Layered Pictorial Structure (LPS) –Learned exemplars for parts of the LPS model –Obtained impressive results + Layer 1Layer 2 = LPS model

49. LPS for Detection Learning – Learnt automatically using a set of examples Detection Tree of chamfers to detect parts, assemble with pictorial structure and belief propogation.

50. Solve via Integer Programming SDP formulation (Torr 2001, AI stats) SOCP formulation (Kumar, Torr & Zisserman this conference) LBP (Huttenlocher, many)

51. Obj-Cut Image Likelihood Ratio (Colour) Shape Prior Distance from  Likelihood + Distance from 

52. Integrating Shape-Prior in MRFs Unary potential Pairwise potential Label s Pixel s Prior Potts model MRF for segmentation

53. Integrating Shape-Prior in MRFs  Unary potential Pairwise potential Pose parameters Label s Pixel s Prior Potts model Pose-specific MRF

54. SegmentationImage Results Using LPS Model for Cow

55. Layer 2 Layer 1 Transformations Θ 1 P(Θ 1 ) = 0.9 Cow Instance Do we really need accurate models?

56. Segmentation boundary can be extracted from edges Rough 3D Shape-prior enough for region disambiguation

57. Energy of the Pose-specific MRF Energy to be minimized Unary term Shape prior Pairwise potential Potts model But what should be the value of θ?

58. The different terms of the MRF Original image Likelihood of being foreground given a foreground histogram Grimson- Stauffer segmentation Shape prior model Shape prior (distance transform) Likelihood of being foreground given all the terms Resulting Graph-Cuts segmentation

59. Can segment multiple views simultaneously

60. Solve via gradient descent Comparable to level set methods Could use other approaches (e.g. Objcut) Need a graph cut per function evaluation

61. Formulating the Pose Inference Problem

62. But… EACH … to compute the MAP of E(x) w.r.t the pose, it means that the unary terms will be changed at EACH iteration and the maxflow recomputed! However… Kohli and Torr showed how dynamic graph cuts can be used to efficiently find MAP solutions for MRFs that change minimally from one time instant to the next: Dynamic Graph Cuts (ICCV05).

63. Dynamic Graph Cuts PBPB SBSB cheaper operation computationally expensive operation Simpler problem P B* differences between A and B similar PAPA SASA solve

64. Segmentation Comparison Grimson-Stauffer Bathia04 Our method

65. Segmentation

66. Combine Recognition and Segmentation Using all the information should help… –E.g. segmentation result can be used to eliminate false positives. Face/Head and shoulders segmentation particularly useful for applications such as Skype or Windows Messenger.

67. ObjCut Face Segmentation

68. Face Detector and ObjCut

69. Segmentation + Recognition to prune false positives

70. Initialisation Use off-the-shelf face detector to find location of face in an image

71. Initialisation Place shape prior relative to face detection Define region over which to perform segmentation

72. Adjustment Vary parameters of shape prior to find lowest segmentation energy

73. Other Applications of Dynamic Graph Cuts

74. Computing Max-marginals Graph Cuts Dynamic Graph Cuts Image (MSRC)SegmentationMax-Marginal (Fg) : 0.002 msec per MM (1.2 secs) : 300 msec per MM (51.2 days) 1 0 0.5

75. Parameter Learning in CRFs Maximum Likelihood Parameter Learning –Kumar and Herbert –NIPS 2004, EMMCVPR 2005

76. P 3 & Beyond Solving Energies with Higher Order Cliques P ushmeet Kohli P awan Kumar P hilip H. S. Torr Oxford Brookes University

77. Energy Functions Pairwise Energy Functions UnaryPairwise Markov Random Field Observed Variables Hidden Variables

78. Energy Functions Pairwise Energy Functions UnaryPairwise Markov Random Field

79. Energy Functions Pairwise Energy Functions UnaryPairwise Markov Random Field

80. Energy Functions Pairwise Energy Functions UnaryPairwise Markov Random Field

81. Energy Functions Pairwise Energy Functions UnaryPairwise Markov Random Field Efficient Algorithms for Minimization Restricted Expressive Power!

82. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Markov Random Field More expressive than pairwise FOE: Field of Experts (Roth & Black CVPR05)

83. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Original Pairwise MRFHigher order MRF Images Courtesy: Lan et al. ECCV06 MRF for Image Denoising

84. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Computationally expensive to minimize! Exponential Complexity O(L N ) –L = Number of Labels –N = Size of Clique

85. Move Making Algorithms Solution Space Energy

86. Move Making Algorithms Solution Space Energy Search Neighbourhood Current Solution Optimal Move

87. Move Making Algorithms Moves using graph cuts [Boykov, Veksler, Zabih] PAMI 2001 Algorithm –Encode move by vector t –Transformation function T(x,t) → x` –Move Energy –Optimal move t* Submodular

88. Expansion Move Characteristics Move Variables take label  or retain current label Algorithm –Make a move for all  in L

89. Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand GroundExpand HouseExpand Sky

90. Expansion Move Characteristics Neighbourhood Size 2 N where N is the number of variables Guarantee Move energy is submodular for all metric energy functions. [Boykov, Veksler, Zabih] PAMI 2001

91. Swap Move Characteristics Move Variables labeled  can swap their labels Algorithm –Make a move for all  in L

92. Swap Move Sky House Tree Ground Swap Sky, House

93. Swap Move Characteristics Neighbourhood Size 2 N where N is the number of variables Guarantee Move energy is submodular for all semi-metric energy functions. [Boykov, Veksler, Zabih] PAMI 2001

94. Form of the Higher Order Potentials Sum Form Max Form Moves for Higher Order Cliques

95. Theoretical Results: Swap Move energy is always submodular if non-decreasing concave. See paper for proofs

96. Theoretical Results: Expansion Move energy is always submodular if increasing linear See paper for proofs

97. P N Potts Model Generalization of the Potts Model:

98. Solving the P N Potts Model Computing the optimal swap move v1v1 v2v2 vnvn MsMs MtMt t i = 0 v i  Source t j = 1 v j  Sink Source Sink

99. Solving the P N Potts Model Computing the optimal swap move Case 1: all t i = 0 ( x i =  ) Cost: Source Sink v1v1 v2v2 vnvn MsMs MtMt

100. Solving the P N Potts Model Computing the optimal swap move v1v1 v2v2 vnvn MsMs MtMt Cost: Source Sink Case 2: all t i = 1 ( x i =  )

101. Solving the P N Potts Model Computing the optimal swap move Cost: v1v1 v2v2 vnvn MsMs MtMt Source Sink Case 3: t i = 0,1 ( x i =  )

102. Solving the P N Potts Model Computing the optimal expansion move v1v1 v2v2 vnvn MsMs MtMt Source Sink

103. Solving the P N Potts Model Computing the optimal expansion move Cost: Source Sink v1v1 v2v2 vnvn MsMs MtMt Case 1: all t i = 0 ( x i = x i )

104. Solving the P N Potts Model Computing the optimal expansion move Cost: v1v1 v2v2 vnvn MsMs MtMt Source Sink Case 2: all t i = 1 ( x i =  )

105. Solving the P N Potts Model Computing the optimal expansion move Cost: v1v1 v2v2 vnvn MsMs MtMt Source Sink Case 3: t i = 0,1 ( x i = x i,  )

106. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original Image

107. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Colour Histograms Unary Cost: Tree

108. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Edge Sensitive Smoothness Cost

109. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) MAP Solution

110. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Patch Dictionary Higher Order Cost: Tree

111. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Higher Order Cost: TreeUnary Cost: Tree

112. Experimental Results Exemplar based Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original PairwiseHigher order

113. Experimental Results Original Ground Truth Swap (3.2 sec) Expansion (2.5 sec) PairwiseHigher Order Swap (4.2 sec) Expansion (3.0 sec)

114. Experimental Results Original Ground Truth PairwiseHigher Order Swap (4.7 sec) Expansion (3.7sec) Swap (5.0 sec) Expansion (4.4 sec)

115. Conclusions & Future Work Efficient minimization of certain higher order energies Can handle very large cliques Useful for many Computer Vision problems Explore other interesting family of potential functions

116. Other Relaxations New work NIPS 2007 All previous work based on LP relaxation, we demonstrate can get better approximation using SOCP relaxation..

117. Preliminaries Convex Optimization Program min f(x) subject to g i (x) ≤ 0 i=1, …, N Objective FunctionConstraints Objective function f(.) should be convex Constraints g i (.) should define a convex set Feasibility region F (P) should be a convex set P

118. Preliminaries Linear Program (LP) min c T x subject to Ax ≤ b Linear objective function Linear constraints

119. Preliminaries Quadratic Program (QP) min x T Qx + c T x subject to Ax ≤ b Convex quadratic objective function Linear constraints Q 0 LP < QP

120. Preliminaries Second Order Cone Program (SOCP) min c T x subject to x T Q i x + x T a i ≤ b i Linear objective function Convex quadratic constraints Q i 0 LP < QP < SOCP

121. Conclusions Large class of SOCP/QP dominated by LP-S New SOCP relaxations dominate LP-S Experiments conform with analysis Determining “best” SOCP constraints ??

122. SOCP Relaxations on Cycles True for odd cycles with P ij ≤ 0 True for odd cycles with P ij ≤ 0 for only one edge True for odd cycles with P ij ≥ 0 for only one edge True for all combinations of above cases