Extensions of submodularity and their application in computer vision Vladimir Kolmogorov IST Austria Heidelberg, 14 November 2014

Discrete Optimization in Computer Vision Minimize Known as MAP inference in a graphical model maximim a posteriori estimation Energy minimization

Ising model Submodular function Minimizing f : maxflow algorithm (“graph cuts”) In Microsoft Powerpoint 2010 Two more examples Interactive image segmentation

Potts model NP-hard problem (for k>2) Efficient approximation algorithms “alpha-expansion” [Boykov et al.’01]: 2-approximation Two more examples Object recognition

Partial optimality Partial labeling: 1 5 6 2 4 7 3 Two more examples Optimal if can be extended to a minimizer of f

Partial optimality Strategy: construct function g over partial labelings - k-submodular relaxation of f minimize g Two more examples 7 1 2 3 4 5 6

Classes of problems submodular functions k-submodular functions ... Potts functions Part I: Complete classification of tractable classes For finite-valued VCSP languages Part II: Application of k-submodular functions Obtaining partial optimality Efficient algorithm for the Potts model tractable NP-hard Two more examples

Valued Constraint Satisfaction Problem (VCSP) D: fixed set of labels, e.g. D = {a,b,c} Language G: a set of cost functions VCSP(G): class of functions that can be expressed as a sum of functions from G with overlapping sets of vars Goal: minimize this sum Complexity of G? G-instance: example:

Classifications for finite-valued CSPs Theorem If G admits a binary symmetric fractional polymorphism then it can be solved in polynomial time by Basic LP relaxation (BLP) [Thapper,Živný FOCS’12], [K ICALP’13] Otherwise G is NP-hard [Thapper,Živný STOC’13] For CSPs classification still open [Feder-Vardi conjecture] Every G is either tractable or NP-hard CSP: contains functions

Submodular functions 4 3 2 1

New classes of functions

Useful classes Submodular functions k-submodular functions Pairwise functions: can be solved via maxflow (“graph cuts”) Lots of applications in computer vision k-submodular functions Partial optimality for functions of k-valued variables This talk: efficient algorithm for Potts energy [Gridchyn, K ICCV’13]

Partial optimality Input: function Partial labeling is optimal if it can be extended to a full optimal labeling

Partial optimality Input: function Partial labeling is optimal if it can be extended to a full optimal labeling Can be viewed as a labeling

k-submodular relaxations Input: function

k-submodular relaxations Input: function Construct extension which is k-submodular Minimize Theorem: Minimum of partially optimal

k-submodularity Function is k-submodular if

k-submodular relaxations Case k = 2 ([K’10,12]) Bisubmodular relaxation Characterizes extensions of QPBO Case k > 2 [Gridchyn,K ICCV’13] : - efficient method for Potts energies [Wahlström SODA’14] : - used for FPT algorithms

k-submodular relaxations for Potts energy

k-submodular relaxations for Potts energy d(a,b) : tree metric

k-submodular relaxations for Potts energy 3 4 10 gi (·) : k-submodular relaxation of fi (·) 1.5

k-submodular relaxations for Potts energy Minimizing g : O(log k) maxflows Alternative approach: [Kovtun ’03,’04] Stronger than k-submodular relaxations (labels more) Can be solved by the same approach! complexity: k => O(log k) maxflows Part of “Reduce, Reuse, Recycle’’ [Alahari et al.’08,’10] Our tests for stereo: 50-93% labeled with 9x9 windows Speeds up alpha-expansion for unlabeled part

Tree Metrics [Felzenszwalb et al.’10]: O(log k) maxflows for

Tree Metrics [Felzenszwalb et al.’10]: O(log k) maxflows for [This work]: extension to more general unary terms new proof of correctness

Special case: Total Variation Convex unary terms Reduction to parametric maxflow [Hochbaum’01], [Chambolle’05], [Darbon,Sigelle’05]

New condition: T-convexity Convexity for any pair of adjacent edges:

Algorithm: divide-and-conquer Pick edge (a,b) Compute

Algorithm: divide-and-conquer Pick edge (a,b) Compute Claim: g has a minimizer as shown below Solve two subproblems recursively

Achieving balanced splits For star graphs, all splits are unbalanced Solution [Felzenszwalb et al.’10]: insert a new short edge modify unary terms gi (·) accordingly

Algorithm illustration k = 7 labels: 1 2 3 4 5 6 7 1,2,3,4,5,6,7 5 6 7 1 2 3 4

Algorithm illustration k = 7 labels: 2 3 1 4 5 7 6 5,6,7 2 3 1,2,3,4 1 4 5 7 6

Algorithm illustration k = 7 labels: 1,2 5,6,7 3,4

Algorithm illustration k = 7 labels: 5,6 1,2 3,4 7

Algorithm illustration k = 7 labels: 1 5 6 2 4 7 3 “Kovtun labeling” unlabeled part, run alpha-expansion maxflows

Stereo results

Proof of correctness (sketch)

Proof of correctness (sketch) For labeling x and edge (a,b) define

Proof of correctness (sketch) For labeling x and edge (a,b) define

Proof of correctness (sketch) Coarea formula:

Proof of correctness (sketch) Coarea formula:

Proof of correctness (sketch) Coarea formula:

Proof of correctness (sketch) Coarea formula:

Proof of correctness (sketch) Coarea formula:

Proof of correctness (sketch) Coarea formula: Equivalent problem: minimize with subject to consistency constraints Equivalent to independent minimizations of - consistency holds automatically due to convexity of

Extension to trees Coarea formula:

Summary Part I: Complete characterization of tractable finite-valued CSPs Part II: k-submodular relaxations for partial optimality For Potts model: cast Kovtun’s approach as k-submodular function minimization O(log k) algorithm generalized alg. of [Felzenswalb et al’10] for tree metrics Future work: k-submodular relaxations for other functions?

positions are available postdoc & PhD student positions are available