P 3 & Beyond Solving Energies with Higher Order Cliques Pushmeet Kohli Pawan Kumar Philip H. S. Torr Oxford Brookes University CVPR 2007

Energy Functions Observed Variables Hidden Variables MAP Inference

Energy Functions MAP Inference Energy Minimization

Energy Functions Pairwise Energy Functions UnaryPairwise

Energy Functions Pairwise Energy Functions UnaryPairwise

Energy Functions Pairwise Energy Functions UnaryPairwise Efficient Algorithms for Minimization Message Passing (BP, TRW) Move Making (Expansion/Swap)

Energy Functions Pairwise Energy Functions UnaryPairwise Efficient Algorithms for Minimization Message Passing (BP, TRW) Move Making (Expansion/Swap) Restricted Expressive Power!

Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order More expressive than pairwise FOE: Field of Experts (Roth & Black CVPR05)

Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Computationally expensive to minimize! Exponential Complexity O(L N ) L = Number of Labels N = Size of Clique

Minimizing Higher Order Energies Efficient BP in Higher Order MRFs (Lan, Roth, Huttenlocher & Black, ECCV 06) 2x2 clique potentials for Image Denoising Searched a restricted state space 16 minutes per iteration Pairwise MRF Higher order MRF Noisy Image

Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Our Method Move making algorithm Can handle cliques of thousand of variables Extremely Efficient ( works in seconds)

Talk Outline Move making Algorithms Solvable Higher Order Potentials Moves for the P N Potts Model Application: Texture Segmentation

Move Making Algorithms Solution Space Energy

Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy

Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc E m (t) = E(T(x c, t)) Transformation function T EmEm Move Energy T(x c, t) = x n = x c + t minimize t* Optimal Move (t)(t) x

Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x Key Characteristic: Search Neighbourhood Bigger the better!

Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] Exponential Move Search Space (Good ) Move encoded by binary vector t Move Energy

Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] Exponential Move Search Space (Good ) Move encoded by binary vector t Move Energy Optimal move t* in polynomial time Submodular

Expansion Move  Expansion Transformation Variables take label  or retain current label Optimal move can be computed for pairwise potentials which are metric. [Boykov, Veksler, Zabih]

Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih]

Swap Move Optimal move can be computed for pairwise potentials which are semi-metric.  - Swap Transformation Variables labeled  can swap their labels [Boykov, Veksler, Zabih]

Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih]

Moves for Higher Order Potentials Question you should be asking: Can my higher order potential be solved using α-expansions?

Moves for Higher Order Potentials Question you should be asking: Show that move energy is submodular for all x c Can my higher order potential be solved using α-expansions?

Moves for Higher Order Potentials Question you should be asking: Show that move energy is submodular for all x c Can my higher order potential be solved using α-expansions? Not an easy thing to do!

Form of the Higher Order Potentials Moves for Higher Order Potentials Clique Inconsistency function: Pairwise potential: xixi xjxj xkxk xmxm xlxl c Sum Form Max Form

Theoretical Results: Swap Move energy is always submodular if non-decreasing concave. See paper for proofs

Theoretical Results: Expansion Move energy is always submodular if increasing linear See paper for proofs

P N Potts Model

c

c Cost :  red

P N Potts Model c Cost :  max

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Label 1  Label 2 

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Move Energy is independent of t c and can be ignored. Label 1  Label 2 

Optimal moves for P N Potts Computing the optimal swap move c Label 1  Label 2  Label 3 Label 4 Case 2 All variables assigned label 1 or 2

Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 2 All variables assigned label 1 or 2 Can be minimized by solving a st-mincut problem Label 1  Label 2 

Solving the Move Energy Add a constant This transformation does not effect the solution

Solving the Move Energy Computing the optimal swap move Source Sink v1v1 v2v2 vnvn MsMs MtMt t i = 0 v i  Source Set t j = 1 v j  Sink Set

Solving the Move Energy Computing the optimal swap move Case 1: all x i =  (v i  Source ) Cost: Source Sink v1v1 v2v2 vnvn MsMs MtMt

Solving the Move Energy Computing the optimal swap move v1v1 v2v2 vnvn MsMs MtMt Cost: Source Sink Case 2: all x i =  (v i  Sink )

Solving the Move Energy Computing the optimal swap move Cost: v1v1 v2v2 vnvn MsMs MtMt Source Sink Case 3: all x i =  (v i  Source, Sink )

Optimal moves for P N Potts The expansion move energy Similar graph construction. See paper for details

Experimental Results Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original Image Texture Segmentation

Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Colour Histogram Unary Cost: Tree

Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Edge Sensitive Smoothness Cost

Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Expansion Solution

Higher Order Texture Potentials Patch Dictionary (Tree) 5x5 patches

Higher Order Texture Potentials Patch Dictionary (Tree) G (c,p s ): L 1 distance between patch p s and pixel set

Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original PairwiseHigher order

Experimental Results OriginalSwap (3.2 sec) Expansion (2.5 sec) PairwiseHigher Order Swap (4.2 sec) Expansion (3.0 sec)

Experimental Results Original PairwiseHigher Order Swap (4.7 sec) Expansion (3.7sec) Swap (5.0 sec) Expansion (4.4 sec)

Conclusions & Future Work Efficient minimization of certain higher order energies Can handle very large cliques Allows more expressive functions Explore other interesting family of potential functions

Thanks Questions?