1. Hierarchical Graph Cuts for Semi-Metric Labeling M. Pawan Kumar Joint work with Daphne Koller

2. Aim To obtain accurate MAP estimate for Semi-Metric MRFs efficiently V1V1 V2V2 ……… …………… …………… …………VnVn Random Variables V = { V 1, V 2, …, V n }

3. Aim VaVa VbVb lili  ab (i,j)  a (i) : arbitrary  ab (i,j) = s ab d(i,j) s ab ≥ 0  a (i)  b (j) ljlj d( i, i ) = 0 for all i d( i, j ) = d( j, i ) > 0 for all i≠j Semi-metric Distance Function d( i, j ) - d( j, k ) ≤ d( i, k ) Metric Distance Function To obtain accurate MAP estimate for Semi-Metric MRFs efficiently

4. Aim VaVa VbVb lili  ab (i,j)  a (i) : arbitrary  a (i)  b (j) ljlj f* = arg min f   a (f(a)) +   ab (f(a),f(b))  ab (i,j) = s ab d(i,j) s ab ≥ 0 To obtain accurate MAP estimate for Semi-Metric MRFs efficiently

5. Visualizing Metrics l5l5 l1l1 l2l2 l4l4 l3l3 w1w1 w2w2 w3w3 w4w4 w5w5 w6w6 w7w7 w9w9 w8w8 d( i, j ) : shortest path defined by the graph

6. Overview + f1f1 f2f2 f

7. Outline Simpler Metrics Labeling for Simpler Metrics Approximating General Metrics/Semi- Metrics Combining Labelings Results

8. r-HST Metrics Edge lengths for all children are the same l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 Graph is a Tree. Labels are leaves

9. r-HST Metrics Edge lengths decrease by factor r ≥ 2 w 2 ≤ w 1 /rw 3 ≤ w 1 /r l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3

10. Outline Simpler Metrics Labeling for Simpler Metrics Approximating General Metrics/Semi- Metrics Combining Labelings Results

11. r-HST Metric Labeling r-HST Metrics admit Divide-and-Conquer Divide original problem into subproblems l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3

12. r-HST Metric Labeling Subproblem defined at vertex ‘m’ l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 f* = arg min f   a (f(a)) +   ab (f(a),f(b)) such that f(a)  m

13. r-HST Metric Labeling Trivial problem l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 f* = arg min f   a (f(a)) +   ab (f(a),f(b)) such that f(a)  { l 4 }

14. r-HST Metric Labeling Original problem l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 f* = arg min f   a (f(a)) +   ab (f(a),f(b)) such that f(a)  { l 1, …, l 6 }

15. r-HST Metric Labeling Problems get tougher as we move up Solve the simple subproblems (starting with trivial subproblems) Use their solutions to solve difficult subproblems

16. r-HST Metric Labeling w w w f1f1 f2f2 f3f3 Find new labeling using  -Expansion

17. r-HST Metric Labeling w w w f1f1 f2f2 f3f3 Continue till we reach the root

18. Analysis w w w Mathematical Induction All variables V a such that f*(a)  m m 1 bound on the unary potentials 2r/(r-1) bound on the pairwise potentials

19. Analysis w w w Mathematical Induction m Initial step of M.I. trivial (for leaf nodes) Given children, prove for parent

20. Analysis w w w   a (f(a)) +  i   ab (f i (a),f i (b)) +  i≠j   ab (f i (a),f j (b)) f(a) = f i (a) f(b) = f i (b) f(a) = f i (a) f(b) = f j (b)

21. Analysis w w w   a (f*(a)) +  i   ab (f i (a),f i (b)) +  i≠j   ab (f i (a),f j (b))

22. Analysis w w w   a (f*(a)) +  i   ab (f*(a),f*(b)) +  i≠j   ab (f i (a),f j (b)) 2r r-1

23. Analysis w w w   a (f*(a)) +  i   ab (f*(a),f*(b)) +  i≠j   ab (f*(a),f*(b)) d max d min 2 2r r-1

24. Analysis w w w d max = 2w(1+1/r+1/r 2 +….) d min = 2w

25. Analysis w w w  i≠j   ab (f*(a),f*(b)) 2r r-1   a (f*(a)) +  i   ab (f*(a),f*(b)) + 2r r-1

26. Analysis Overall approximation bound 2r/(r-1) Previous best bound 2r/(r-2) Not Tight ?

27. Overview + f1f1 f2f2 f

28. Outline Simpler Metrics Labeling for Simpler Metrics Approximating General Metrics/Semi- Metrics Combining Labelings Results

29. Approximating Metrics D = {d t (.,.), t = 1,2,… T}, d t (i,j) ≥ d(i,j) Pr(.) over the elements of D Given distance d(.,.) min D,Pr(.) max i≠j ∑ t Pr(t) d t (i,j) d(i,j)

30. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 w1w1 w1w1 w2w2 w2w2 w2w2 w3w3 w3w3 w3w3 r-HST : hierarchical clustering of labels Use a clustering algorithm

31. Approximating Metrics Fakcharoenphol, Rao and Talwar, 2003 max d(i,j) = 2 M min i≠j d(i,j) > 1 Level ‘1’ Level ‘2’ Clustering at level 2?? Sample   [1,2] Choose a permutation π of labels = { l 1,…, l h }

32. Approximating Metrics max d(i,j) = 2 M min i≠j d(i,j) > 1 Level ‘m-2’ Level ‘m-1’ Clustering at level m?? Choose a permutation π of labels Fakcharoenphol, Rao and Talwar, 2003 Sample   [1,2]

33. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l1l1 l2l2 l3l3 π d(1,4) ≤ 2 M-m  ? Fakcharoenphol, Rao and Talwar, 2003

34. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l1l1 l2l2 l3l3 π d(2,4) ≤ 2 M-m  ? Fakcharoenphol, Rao and Talwar, 2003

35. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l1l1 l2l2 l3l3 π d(2,1) ≤ 2 M-m  ? Fakcharoenphol, Rao and Talwar, 2003

36. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l1l1 l2l2 l3l3 π d(3,4) ≤ 2 M-m  ? Fakcharoenphol, Rao and Talwar, 2003

37. Approximating Metrics l1l1 l2l2 l3l3 l4l4 l5l5 l6l6 l1l1 l2l2 l3l3 π Edge length = Diameter of cluster / 2 Fakcharoenphol, Rao and Talwar, 2003

38. Approximating Metrics Choose . Choose π Initialize root node as trivial cluster (all labels) Choose a cluster at level m-1 Run procedure to get clusters at level m Repeat for all clusters at level m-1 Stop when all clusters are singletons Repeat to get a set of r-HST metrics Fakcharoenphol, Rao and Talwar, 2003

39. Analysis d(i,j) ≤ ∑Pr(t) d t (i,j) ≤ O(log h) d(i,j) How many r-HST metrics ?? O(h log h) Charikar, Chekuri, Goel, Guha and Plotkin, 1998 Fakcharoenphol, Rao and Talwar, 2003

40. Approximating Semi-Metrics d(i,j) ≤ ∑Pr(t) d t (i,j) ≤ O((  log h) 2 ) d(i,j) How many r-HST metrics ?? O(h log h) d(i,j) - d(j,k) ≤  d(i,k)

41. Overview + f1f1 f2f2 f

42. Outline Simpler Metrics Labeling for Simpler Metrics Approximating General Metrics/Semi- Metrics Combining Labelings Results

43. Combining Labelings Use  -Expansion !!

44. Analysis Bound for r-HST Labeling = O(1) Distortion for Metrics = O(log h) Bound for Metric Labeling = O(log h) Distortion for Semi-Metrics = O((  log h) 2 ) Bound for Semi-Metric Labeling = O((  log h) 2 )

45. Analysis When h < n, all known LP bounds can be obtained using move making algorithms.

46. Refining the Labeling Current energy Q(f; d)= Q(f; d t ) Q(f’; d) ≤ Q(f’; d t ), f’ ≠ f Find best f t according to d t (.,.) Fakcharoenphol, Rao and Talwar, 2003 r-HST Metric Labeling f = f t. Repeat till convergence.

47. Outline Simpler Metrics Labeling for Simpler Metrics Approximating General Metrics/Semi- Metrics Combining Labelings Results

48. Synthetic Data T. Lin.T. Quad.r-HSTMetS-Met Exp4864552094502214811247613 Swap4872151938510554848747579 TRW-S4750651318481324735546612 BP-S5094260269528414813647402 R-Swap4804551842--- R-Exp4799851641--- Our4785051587481464753846651 Our+EM4782351413481464738246638

49. Synthetic Data T. Lin.T. Quad.r-HSTMetS-Met Exp0.440.360.290.300.36 Swap0.650.860.520.510.47 TRW-S104.29178.97713.70703.82709.36 BP-S15.7845.63150.36129.68141.79 R-Swap1.9710.73--- R-Exp5.7830.73--- Our10.2212.841.8610.5812.25 Our+EM25.6664.085.0232.7557.50

50. Image Denoising

51. Exp Swap TRW-S BP-SOurOur+EM 75641,5.09 74426,25.2268226,174.33 105845,32.9472828,70.5572332,204.55

52. Image Denoising

53. Exp Swap TRW-S BP-SOurOur+EM 86163,26.13 89264,90.7473383,529.60 526969,115.8481820,294.7281820,465.57

54. Stereo Reconstruction

55. Exp Swap TRW-S BP-SOurOur+EM 78776,12.07 97999,34.5962777,263.28 126824,50.3865116,152.7465008,361.81

56. Scene Registration

57. Exp Swap TRW-S BP-SOurOur+EM 82036,1.66 83023,8.1581118,1371.11 84396,218.0481315,104.8981258,373.60

58. Scene Registration

59. Exp Swap TRW-S BP-SOurOur+EM 68572,1.27 69767,2.7867616,1058.25 70239,159.9867682,73.6167676,240.49

60. Scene Segmentation EnergyAccuracyTiming Exp30227260.623.18 Swap30238960.603.73 TRW-S30221160.68451.02 BP-S31082560.44102.14 Our30226560.64157.03

61. Future Work Tighter approximations for semi-metrics Higher-order potentials? Learning the parameters?

62. A Diffusion Algorithm for Upper Envelope Potentials M. Pawan Kumar Joint work with Pushmeet Kohli

63. Aim Efficient MAP estimation of sparse higher order potentials V1V1 V2V2 ……… …………… …………… …………VnVn

64. Aim Efficient MAP estimation of sparse higher order potentials Z In general, f(z)  L c Some special cases computationally feasible

65. Lower Envelope Potentials Z min i  z (i) + ∑ a  C  za (i,f(a)) f(z)  L’  L c

66. Lower Envelope Potentials ENERGYENERGY

67. ENERGYENERGY

68. ENERGYENERGY

69. min i  z (i) + ∑ a  C  za (i,f(a)) f(z)  {0,1} ENERGYENERGY Robust P n Model

70. Lower Envelope Potentials + ∑ z ( min i  z (i) + ∑ a  C  za (i,f(a)) ) f* = arg min f   a (f(a)) +   ab (f(a),f(b))

71. Lower Envelope Potentials f* = arg min f   a (f(a)) +   ab (f(a),f(b)) f(z)  L’ + ∑ z  z (f(z)) + ∑ a  C  za (f(z),f(a)) Use your favorite pairwise MRF algorithm

72. Upper Envelope Potentials Z max i  z (i) + ∑ a  C  za (i,f(a)) f(z)  L’  L c

73. Upper Envelope Potentials Silhouette Object Ray Camera center At least one voxel on the ray labeled ‘object’

74. Upper Envelope Potentials Silhouette Object Ray Camera center max i  z (i) + ∑ a  C  za (i,f(a)) f(z)  {0,1}

75. Upper Envelope Potentials + ∑ z ( max i  z (i) + ∑ a  C  za (i,f(a)) ) f* = arg min f   a (f(a)) +   ab (f(a),f(b))

76. Upper Envelope Potentials + ∑ z t z f* = arg min f   a (f(a)) +   ab (f(a),f(b)) t z ≥  z (i) + ∑ a  C t za (i) t za (i) ≥  za (i,f(a)) LP Relaxation

77. Dual max  a min i  a (i) +  (a,b) min i,j  ab (i,j) +  z min i  z (i) +  (z,a) min i,j  za (i,j)    ∑ i z (i) = 1∑ j za (i,j) = z (i) za (i,j)≥ 0  a (i) =  a (i)  ab (i,j) =  ab (i,j)  z (i) = z (i)  a (i)  za (i,j) = za (i,j)  ab (i,j)

78. Dual Without Z max  a min i  a (i) +  (a,b) min i,j  ab (i,j)      

79. Diffusion VaVa 3 10 2 VaVa 5 1012 3 VaVa 4 2 0 2 3

80. Diffusion VaVa 3 00 1 VaVa 0 59 0 VaVa 4 2 0 1 5 3

81. VaVa 3 00 1 VaVa 0 59 0 VaVa 3 2 3 2 3 2

82. VaVa 6 23 3 VaVa 3 811 2 VaVa 3 2 2 2 2

83. Diffusion for Auxiliary Variable z 3 10 2 z 5 1012 3 z 4 2  z (i) =  ’ z (i) + ( z (i) - ’ z (i))  a (i)  za (i,j) =  ’ za (i,j) + ( za (i,j) - ’ za (i,j))  za (i,j)

84. Diffusion for Auxiliary Variable max ( min i  z (i) + ∑ a min i,j  za (i,j) ) ∑ I z (i) = 1 z (i)≥ 0 ∑ j za (i,j) = z (i) za (i,j)≥ 0 Solve for Expensive

85. Diffusion for Auxiliary Variable max ( min i  z (i) ) max ( min i,j  za (i,j) ) ∑ I z (i) = 1 z (i)≥ 0 ∑ j za (i,j) = z (i) ? ∑ ij za (i,j) = 1 za (i,j)≥ 0

86. Diffusion for Auxiliary Variable max ( min i  z (i) ) max ( min i,j  za (i,j) ) ∑ I z (i) = 1 z (i)≥ 0 ∑ ij za (i,j) = 1 za (i,j)≥ 0 + ∑  z;i z (i) + ∑  za;i za (i,j)  z;i + ∑ a  za;i = 0 Fractional Packing Problem

87. Diffusion for Auxiliary Variable max ( min i  z (i) ) max ( min i,j  za (i,j) ) ∑ I z (i) = 1 z (i)≥ 0 ∑ ij za (i,j) = 1 za (i,j)≥ 0 + ∑  z;i z (i) + ∑  za;i za (i,j)  z;i + ∑ a  za;i = 0 Plotkin, Shmoys and Tardos, 1995

88. Diffusion for Auxiliary Variable z 3 10 2 z 5 1012 3 z 4 2 Run Standard Diffusion on 

89. The Algorithm Choose a variable (random or auxiliary) If random variable, run standard diffusion If auxiliary variable, obtain and then run standard diffusion Repeat till convergence

90. Future Work Write the code Do the experiments A better way to get ??