Improved Moves for Truncated Convex Models M. Pawan Kumar Philip Torr

Aim Efficient, accurate MAP for truncated convex models V1V1 V2V2 ……… …………… …………… …………VnVn Random Variables V = { V 1, V 2, …, V n } Edges E define neighbourhood

Aim VaVa VbVb lili lklk  ab;ik Accurate, efficient MAP for truncated convex models  ab;ik = w ab min{ d(i-k), M }  ab;ik i-k w ab is non-negative Truncated Linear i-k  ab;ik Truncated Quadratic d(.) is convex  a;i  b;k

Motivation Low-level Vision Smoothly varying regions Sharp edges between regions min{ |i-k|, M} Boykov, Veksler & Zabih 1998 Well-researched !!

Things We Know NP-hard problem - Can only get approximation Best possible integrality gap - LP relaxation Manokaran et al., 2008 Solve using TRW-S, DD, PP Slower than graph-cuts Use Range Move - Veksler, 2007 None of the guarantees of LP

Real Motivation Gaps in Move-Making Literature LP Move- Making Potts Truncated Linear Truncated Quadratic 2 Multiplicative Bounds 2 + √2 O(√M) Chekuri et al., 2001

Real Motivation Gaps in Move-Making Literature LP Move- Making Potts Truncated Linear Truncated Quadratic 2 Multiplicative Bounds √2 2M O(√M)- Boykov, Veksler and Zabih, 1999

Real Motivation Gaps in Move-Making Literature LP Move- Making Potts Truncated Linear Truncated Quadratic 2 Multiplicative Bounds √2 4 O(√M)- Gupta and Tardos, 2000

Real Motivation Gaps in Move-Making Literature LP Move- Making Potts Truncated Linear Truncated Quadratic 2 Multiplicative Bounds √2 4 O(√M)2M Komodakis and Tziritas, 2005

Real Motivation Gaps in Move-Making Literature LP Move- Making Potts Truncated Linear Truncated Quadratic 2 Multiplicative Bounds √2 O(√M) 2 + √2 O(√M)

Outline Move Space Graph Construction Sketch of the Analysis Results

Move Space VaVa VbVb Initialize the labelling Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labelling Iterate over intervals

Outline Move Space Graph Construction Sketch of the Analysis Results

Two Problems VaVa VbVb Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labelling Large L’ => Non-submodular Non-submodular

First Problem VaVa VbVb Submodular problem Ishikawa, 2003; Veksler, 2007

First Problem VaVa VbVb Non-submodular Problem

First Problem VaVa VbVb Submodular problem Veksler, 2007

First Problem VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb Model unary potentials exactly a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb Similarly for V b a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb Model convex pairwise costs a m+1 a m+2 anan t b m+1 b m+2 bnbn

First Problem VaVa VbVb Overestimated pairwise potentials Wanted to model  ab;ik = w ab min{ d(i-k), M } For all l i, l k  I Have modelled  ab;ik = w ab d(i-k) For all l i, l k  I

Second Problem VaVa VbVb Choose interval I of L’ labels Each variable can Retain old label Choose a label from I Choose best labelling Non-submodular problem !!

Second Problem - Case 1 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn s ∞∞ Both previous labels lie in interval

Second Problem - Case 1 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn s ∞∞ w ab d(i-k)

Second Problem - Case 2 VaVa VbVb Only previous label of V a lies in interval a m+1 a m+2 anan t b m+1 b m+2 bnbn s ∞ ubub

Second Problem - Case 2 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn u b : unary potential of previous label of V b M s ∞ ubub

Second Problem - Case 2 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn M w ab d(i-k) s ∞ ubub

Second Problem - Case 2 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn M w ab ( d(i-m-1) + M ) s ∞ ubub

Second Problem - Case 3 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn Only previous label of V b lies in interval

Second Problem - Case 3 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn s uaua ∞ u a : unary potential of previous label of V a M

Second Problem - Case 4 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn Both previous labels do not lie in interval

Second Problem - Case 4 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn s uaua ubub P ab : pairwise potential for previous labels ab P ab M M

Second Problem - Case 4 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn w ab d(i-k) s uaua ubub ab P ab M M

Second Problem - Case 4 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn w ab ( d(i-m-1) + M ) s uaua ubub ab P ab M M

Second Problem - Case 4 VaVa VbVb a m+1 a m+2 anan t b m+1 b m+2 bnbn P ab s uaua ubub ab P ab M M

Graph Construction VaVa VbVb Find st-MINCUT. Retain old labelling if energy increases. a m+1 a m+2 anan b m+1 b m+2 bnbn t ITERATE

Outline Move Space Graph Construction Sketch of the Analysis Results

Analysis VaVa VbVb Current labelling f(.) Q C ≤ Q’ C VaVa VbVb Global Optimum f*(.) QPQP Previous labelling f’(.) VaVa VbVb

Analysis VaVa VbVb Current labelling f(.) Q C ≤ Q’ C VaVa VbVb Partially Optimal f’’(.) Previous labelling f’(.) VaVa VbVb Q’ 0 ≤

Analysis VaVa VbVb Current labelling f(.) Q P - Q’ C VaVa VbVb Partially Optimal f’’(.) Previous labelling f’(.) VaVa VbVb Q P - Q’ 0 ≥

Analysis VaVa VbVb Current labelling f(.) Q P - Q’ C VaVa VbVb Partially Optimal f’’(.) Local Optimal f’(.) VaVa VbVb Q P - Q’ 0 ≤ 0

Analysis VaVa VbVb Current labelling f(.) VaVa VbVb Partially Optimal f’’(.) Local Optimal f’(.) VaVa VbVb Q P - Q’ 0 ≤ 0 Take expectation over all intervals

Analysis Truncated Linear Q P ≤ 2 + max 2M, L’ L’M Q* L’ = M4Gupta and Tardos, 2000 L’ = √2M 2 + √2 Truncated Quadratic Q P ≤ O(√M) Q* L’ = √M

Outline Move Space Graph Construction Sketch of the Analysis Results

Synthetic Data - Truncated Linear Faster than TRW-S Comparable to Range Moves With LP Relaxation guarantees Time (sec) Energy

Synthetic Data - Truncated Quadratic Faster than TRW-S Comparable to Range Moves With LP Relaxation guarantees Time (sec) Energy

Stereo Correspondence Disparity Map Unary Potential: Similarity of pixel colour Pairwise Potential: Truncated convex

Stereo Correspondence AlgoEnergy1Time1Energy2Time2 Swap Exp TRW-S BP Range Our Teddy

Stereo Correspondence AlgoEnergy1Time1Energy2Time2 Swap Exp TRW-S BP Range Our Teddy

Stereo Correspondence AlgoEnergy1Time1Energy2Time2 Swap Exp TRW-S BP Range Our Tsukuba

Summary Moves that give LP guarantees Similar results to TRW-S Faster than TRW-S because of graph cuts

Questions Not Yet Answered Move-making gives LP guarantees –True for all MAP estimation problems? Huber function? Parallel Imaging Problem? Primal-dual method? Solving more complex relaxations?

Questions? Improved Moves for Truncated Convex Models Kumar and Torr, NIPS