1. Extensions of submodularity and their application in computer vision
Vladimir Kolmogorov
IST Austria
Heidelberg, 14 November 2014

2. Discrete Optimization in Computer Vision
Minimize
Known as
MAP inference in a graphical model
maximim a posteriori estimation
Energy minimization

3. Ising model Submodular function
Minimizing f : maxflow algorithm (“graph cuts”)
In Microsoft Powerpoint 2010
Two more examples
Interactive
image segmentation

4. Potts model NP-hard problem (for k>2)
Efficient approximation algorithms
“alpha-expansion” [Boykov et al.’01]: 2-approximation
Two more examples
Object recognition

5. Partial optimality Partial labeling: 1 5 6 2 4 7 3
Two more examples
Optimal if can be extended to a minimizer of f

6. Partial optimality Strategy:
construct function g over partial labelings
- k-submodular relaxation of f
minimize g
Two more examples 7 1 2 3 4 5 6

7. Classes of problems submodular functions k-submodular functions
...
Potts functions
Part I: Complete classification of tractable classes
For finite-valued VCSP languages
Part II: Application of k-submodular functions
Obtaining partial optimality
Efficient algorithm for the Potts model
tractable
NP-hard
Two more examples

8. Valued Constraint Satisfaction Problem (VCSP)
D: fixed set of labels, e.g. D = {a,b,c}
Language G: a set of cost functions
VCSP(G): class of functions that can be expressed as a sum of functions from G with overlapping sets of vars
Goal: minimize this sum
Complexity of G?
G-instance:
example:

9. Classifications for finite-valued CSPs
Theorem
If G admits a binary symmetric fractional polymorphism then it can be solved in polynomial time by Basic LP relaxation (BLP) [Thapper,Živný FOCS’12], [K ICALP’13]
Otherwise G is NP-hard [Thapper,Živný STOC’13]
For CSPs classification still open [Feder-Vardi conjecture]
Every G is either tractable or NP-hard
CSP: contains functions

10. Submodular functions 4 3 2 1

11. New classes of functions

12. Useful classes Submodular functions k-submodular functions
Pairwise functions: can be solved via maxflow (“graph cuts”)
Lots of applications in computer vision
k-submodular functions
Partial optimality for functions of k-valued variables
This talk: efficient algorithm for Potts energy [Gridchyn, K ICCV’13]

13. Partial optimality Input: function
Partial labeling is optimal if it can be extended to a full optimal labeling

14. Partial optimality Input: function
Partial labeling is optimal if it can be extended to a full optimal labeling
Can be viewed as a labeling

15. k-submodular relaxations
Input: function

16. k-submodular relaxations
Input: function
Construct extension which is k-submodular
Minimize
Theorem:
Minimum of partially optimal

17. k-submodularity
Function is k-submodular if

18. k-submodular relaxations
Case k = 2 ([K’10,12])
Bisubmodular relaxation
Characterizes extensions of QPBO
Case k > 2
[Gridchyn,K ICCV’13] :
- efficient method for Potts energies
[Wahlström SODA’14] :
- used for FPT algorithms

19. k-submodular relaxations for Potts energy

20. k-submodular relaxations for Potts energy
d(a,b) : tree metric

21. k-submodular relaxations for Potts energy3 4 10
gi (·) : k-submodular relaxation of fi (·)
1.5

22. k-submodular relaxations for Potts energy
Minimizing g : O(log k) maxflows
Alternative approach: [Kovtun ’03,’04]
Stronger than k-submodular relaxations (labels more)
Can be solved by the same approach!
complexity: k => O(log k) maxflows
Part of “Reduce, Reuse, Recycle’’ [Alahari et al.’08,’10]
Our tests for stereo: 50-93% labeled
with 9x9 windows
Speeds up alpha-expansion for unlabeled part

23. Tree Metrics
[Felzenszwalb et al.’10]: O(log k) maxflows for

24. Tree Metrics [Felzenszwalb et al.’10]: O(log k) maxflows for
[This work]: extension to more general unary terms
new proof of correctness

25. Special case: Total Variation
Convex unary terms
Reduction to parametric maxflow [Hochbaum’01], [Chambolle’05], [Darbon,Sigelle’05]

26. New condition: T-convexity
Convexity for any pair of adjacent edges:

27. Algorithm: divide-and-conquer
Pick edge (a,b)
Compute

28. Algorithm: divide-and-conquer
Pick edge (a,b)
Compute
Claim: g has a minimizer as shown below
Solve two subproblems recursively

29. Achieving balanced splits
For star graphs, all splits are unbalanced
Solution [Felzenszwalb et al.’10]: insert a new short edge
modify unary terms gi (·) accordingly

30. Algorithm illustration
k = 7 labels: 1 2 3 4 5 6 7
1,2,3,4,5,6,7 5 6 7 1 2 3 4

31. Algorithm illustration
k = 7 labels: 2 3 1 4 5 7 6
5,6,7 2 3
1,2,3,4 1 4 5 7 6

32. Algorithm illustration
k = 7 labels:
1,2
5,6,7
3,4

33. Algorithm illustration
k = 7 labels:
5,6
1,2
3,4 7

34. Algorithm illustration
k = 7 labels: 1 5 6 2 4 7 3
“Kovtun labeling”
unlabeled part,
run alpha-expansion
maxflows

35. Stereo results

36. Proof of correctness (sketch)

37. Proof of correctness (sketch)
For labeling x and edge (a,b) define

38. Proof of correctness (sketch)
For labeling x and edge (a,b) define

39. Proof of correctness (sketch)
Coarea formula:

40. Proof of correctness (sketch)
Coarea formula:

41. Proof of correctness (sketch)
Coarea formula:

42. Proof of correctness (sketch)
Coarea formula:

43. Proof of correctness (sketch)
Coarea formula:

44. Proof of correctness (sketch)
Coarea formula:
Equivalent problem: minimize
with subject to consistency constraints
Equivalent to independent minimizations of
- consistency holds automatically due to convexity of

45. Extension to trees
Coarea formula:

46. Summary
Part I:
Complete characterization of tractable finite-valued CSPs
Part II:
k-submodular relaxations for partial optimality
For Potts model:
cast Kovtun’s approach as k-submodular function minimization
O(log k) algorithm
generalized alg. of [Felzenswalb et al’10] for tree metrics
Future work: k-submodular relaxations for other functions?

47. positions are available
postdoc & PhD student
positions are available