1. Real-time Background Cut Alon Rubin Shira Kritchman Present: 7.5.2006, Weizmann Institute of Science

2. The Challenge Real-time bilayer segmentation of video

3. The Challenge Real-time bilayer segmentation of video –Fully automatic –High quality –Robustness to changing background –And yet – efficient!

4. Application Background substitution in video conferencing Privacy Coolness

5. Example

6. Overview General method 2 binocular (stereo) algorithms 2 monocular algorithms

7. How to Segment?

8. Information –Colour –Contrast –Disparity –Motion

9. How to Segment? Information Prior assumptions –Spatial coherence –Temporal coherence

10. Information Prior assumptions How to Segment? Notation - point in a 3D colour space - segmentation label - set of neighbours F/B

11. Combining Cues Need: mathematical formulation Prior assumptions –Spatial coherence –Temporal coherence –Priors over disparity Informative cues –Colour –Contrast –Stereo –Motion ?

12. Prior Combining Cues Probabilistic framework: –Maximizing Bayes’ law: Gibbs energy –Maximizing probability  Minimizing energy –Dynamic programming / Graph cut Likelihood Constant x - labels I - data

13. General energy: Spatial coherence + Contrast Colour

14. Modeling Colour Global VS Local Initial VS Dynamic Lots of blue! This is white!

15. Modeling Colour – Globaly –Histograms Overlearning –GMM’s (Gaussian Mixture Models) Number of components Learning: EM (iterations) –Initialization parameters –Stopping condition –Time consuming

16. General energy: Spatial coherence + Contrast Colour

17. Modeling Contrast and Spatial Coherence Spatial coherence Contrast – inhibits the penalty 22 pixels, 72 penalty 22 pixels, 21 penalty 4 3 Segmentation maps Black: Foreground White: Background

18. Algorithms Review "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al., CVPR 2005. Represents two stereo algorithms: –LDP Layered Dynamic Programming –LGC Layered Graph Cut "Background Cut", J. Sun et al., ECCV 2006, to appear "Bilayer Segmentation of Live Video", A. Criminisi et al., CVPR 2006, to appear Stereo Bilayer Segmentation Background Cut Temporal Bilayer Segmentation

19. Stereo Bilayer Segmentation Information: –Colour –Contrast –Stereo Prior: –Spatial coherence –Disparity coherence –Disparity-labeling relations Stereo Bilayer Segmentation

20. Notations F – foreground B – background O – Occluded – disparity vector Stereo Bilayer Segmentation

21. Want to find Stereo Bilayer Segmentation

22. Want to find This is intractable! Stereo Bilayer Segmentation

23. Dealing with Intractability LDP – Layered Dynamic Programming –Defining a similar problem –Separating to scanlines –Solving with dynamic programming Stereo Bilayer Segmentation

24. Dealing with Intractability LGC – Layered Graph Cut –Relaxing some dependencies on disparity –Marginalizing over –Solving with graph cut Stereo Bilayer Segmentation

25. Energy Function Define a Gibbs energy Model it as Prior: Spatial Coherence + contrast Likelihood: Matching Likelihood: Colour Stereo Bilayer Segmentation

26. The Prior V Sum of binary and unary potentials: F: –Spatial coherence –Contrast dependency Stereo Bilayer Segmentation

27. Recall: F: –F,B,O –Sophisticated switch Stereo Bilayer Segmentation The Prior V

28. Recall: V*: –e=0  same equation, –e=1  dilution, –e=0  no use of contrast, Stereo Bilayer Segmentation The Prior V

29. Sum of unary and binary potentials: G: –Higher disparities in foreground –Based on a threshold –Uniform penalty Stereo Bilayer Segmentation The Prior V

30. Likelihood for Matching Distinguish Matched (F,B) from Occluded (O) Determine disparity Model as - balance between occlusion and bad matches Stereo Bilayer Segmentation

31. Likelihood for Matching N – measures quality of match between patches –Classical SSD: Additive + Multiplicative normalization  Robustness –NSSD : Stereo Bilayer Segmentation

32. Likelihood for Matching Balance between occlusion and bad matches:  Preference for occlusion Stereo Bilayer Segmentation

33. Likelihood for Colour GMM’s for Foreground and Background –20 mixture componenets Learn from previous frames Learn using EM –10 iterations Stereo Bilayer Segmentation

34. Likelihood for Colour Model as: Too strong  Balancing factor Stereo Bilayer Segmentation

35. Fusion of Cues Colour FusionStereo Stereo Bilayer Segmentation

36. LDP Layered Dynamic Programming Want: separation to scanlines Recall: V – Sum on neighbouring pixels Use only horizontal cliques  Work on scanlines Prior: Spatial Coherence + contrast Stereo Bilayer Segmentation - LDP

37. LDP Classical DP Diagonal: matched Vertical: occluded Horizontal: occluded Stereo Bilayer Segmentation - LDP

38. LDP Layered DP Alternates between matched and occluded! Stereo Bilayer Segmentation - LDP

39. LDP Layered DP The whole line is matched! Stereo Bilayer Segmentation - LDP

40. LDP Layered DP No diagonal moves Vertical: matched or occluded Horizontal: matched or occluded

41. LDP 4-State Space Many parameters: Learn parameters from labeled data = mean width of matched region = mean width of occluded region a – viewing geometry considerationsa 0,c - normalization Stereo Bilayer Segmentation - LDP

42. LDP 6-State Space Stereo Bilayer Segmentation - LDP Solve with dynamic programming!

43. LGC – Layered Graph Cut Does not solve for disparity Minimizes Marginalize over disparities: Stereo Bilayer Segmentation - LGC X

44. LGC Expansion move algorithm Stereo Bilayer Segmentation - LGC

45. LGC Expansion move algorithm with savings: –Only 3 Labels –Only 2 iterations : Initialize with B for all pixels Run F-expansion Run O-expansion on a constrained region Stereo Bilayer Segmentation - LGC

46. Results LGCLDP Stereo Bilayer Segmentation

47. Results (LGC) Stereo Bilayer Segmentation

48. Results – Errors Stereo Bilayer Segmentation

49. Quantitative Results Hand labeled ground truth (any 5 th /10 th frame) Percentage of misclassified Stereo Bilayer Segmentation

50. Quantitative Results Hand labeled ground truth (any 5 th /10 th frame) Percentage of misclassified Stereo Bilayer Segmentation

51. Quantitative Results Stereo Bilayer Segmentation

52. Quantitative Results Computation times: Around 10 fps at 320 X 240 resolution On a conventional 3GHz processor Stereo Bilayer Segmentation

53. Results Stereo Bilayer Segmentation

54. Stereo Segmentation – Summary 2 algorithms: LGC and LDP Require binocular configuration Temporal relations are implicit Stereo cues are very strong Stereo Bilayer Segmentation

55. Algorithms Review "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al., CVPR 2005. Represents two stereo algorithms: –LDP Layered Dynamic Programming –LGC Layered Graph Cut "Background Cut", J. Sun et al., ECCV 2006, to appear "Bilayer Segmentation of Live Video", A. Criminisi et al., CVPR 2006, to appear Stereo Bilayer Segmentation Background Cut Temporal Bilayer Segmentation

56. Background Cut Information: –Colour –Contrast –Initialization phase Prior: –Spatial coherence Background Cut

57. - = Most Efficient Approach: Background Subtraction Background Cut

58. Problems: Foreground-Background similarity Sensitive threshold Most Efficient Approach: Background Subtraction Background Cut

59. Spatial coherence Colour model  Background maintenance (Minimize by min-cut) Background Cut

60. Basic Model – Colour Term Background: global and local Global: GMM model Background Cut

61. Background: global and local Global: GMM model Local: single Gaussian Basic Model – Colour Term t Background Cut

62. Background: global and local Global: GMM model Local: single gaussian Combination: Basic Model – Colour Term Background Cut

63. Foreground colour model 5 components GMM Basic Model – Colour Term Background Cut

64. Basic Model – Colour Term Background Cut

65. Colour Term Adaptive mixture global-local colour model Background Cut

66. Colour Term Adaptive mixture global-local colour model How can we quantify the difference? Kullback-Liebler divergence Background Cut

67. Colour Term Kullback-Liebler divergence quantify the difference between two GMM’s Background Cut

68. Colour Term Only Global Equally Local and Global Background Cut

69. Colour Term – Summary Background Cut

70. Basic Model – Contrast Term Penalty term + Penalty inhibition Background Cut

71. Contrast Term Background contrast attenuation Background Cut

72. Contrast Term Foreground boundaries Background contrast Clues: Comparison to original background contrast Background Cut

73. Contrast Term Over attenuation of boundaries! Background Cut

74. Clues: Comparison to original background contrast Difference from original background Contrast Term Background Cut

75. Clues: Comparison to original background contrast Difference from original background Contrast Term Background Cut

76. Contrast Term – Summary +Background contrast +Background colour +Background contrast Background Cut

77. Contrast Term – Summary Background Cut

78. Background Maintenance Sudden illuminance change Auto gain control Fluorescent lamps Light switching Background Cut

79. Background Maintenance Minor change Histogram transformation function Major change Colour model rebuilding Background Cut

80. Background Maintenance Colour model rebuilding Foreground threshold increasing Background uncertainty map initialization Mixture model modification Dynamic updating of and Background Cut

81. Background Maintenance Movement in the background Sleeping and waking objects Casual camera shaking - Relying on global model - Keeping biggest connected component - Background maintenance - Appling Gaussian blurring - Using less local colour model Background Cut

82. Background Maintenance

83. Background Cut Background Maintenance

84. Background Cut Quantitative Results

85. Computation times: Around 12-15 fps at 320 X 240 resolution On a conventional 3.2 GHz processor Background Cut

86. Background Cut – Summary Adaptive mixture global-local colour model Background contrast attenuation Background Maintenance Background Cut

87. Algorithms Review "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al., CVPR 2005. Represents two stereo algorithms: –LDP Layered Dynamic Programming –LGC Layered Graph Cut "Background Cut", J. Sun et al., ECCV 2006, to appear "Bilayer Segmentation of Live Video", A. Criminisi et al., CVPR 2006, to appear Stereo Bilayer Segmentation Background Cut Temporal Bilayer Segmentation

88. Information: –Colour –Contrast –Initialization phase –Motion Prior: –Spatial coherence –Temporal coherence Temporal Bilayer Segmentation

89. Motion – Notations Basic image features (YUV) Temporal Bilayer Segmentation

90. Temporal continuity Spatial continuity Colour likelihood Motion likelihood Temporal Bilayer Segmentation

91. Temporal Coherence 4 pixel types: Not likely: BF  B, FB  F Temporal Bilayer Segmentation

92. Temporal Coherence 4 pixel types: Not likely: BF  B, FB  F Temporal Bilayer Segmentation

93. Temporal Coherence 4 pixel types: Temporal Bilayer Segmentation

94. Spatial Coherence The usual term: Temporal Bilayer Segmentation

95. Likelihood for Colour GMM –Number of mixture components –Learning: Initialization Convergence Time consuming Histograms –Nonparametric –Smoothed to avoid overlearning X Temporal Bilayer Segmentation

96. Likelihood for Colour Foreground –Learned adaptively from previous frames Background –Learned from initialization phase –Static over time –Only global (Claim: local doesn’t improve much) Temporal Bilayer Segmentation

97. Likelihood for Motion Optical flow –Inaccuracies along boundaries –The aperture problem –Expensive Motion/Non-motion classifier –Adaptive –Efficient X Temporal Bilayer Segmentation

98. Motion Classifier Basic features: ? Temporal Bilayer Segmentation

99. Motion Classifier ! X-axis: Grad( I ) Y-axis: I.

100. Testing the Motion Classifier Not sufficient: Must fill in the gaps Temporal Bilayer Segmentation

101. Minimizing the Energy Want: Instead: Allowing for changes in t-1 Temporal Bilayer Segmentation

102. Minimizing the Energy Instead: Temporal Bilayer Segmentation

103. Minimizing the Energy Now minimize using Graph Cut Instead: Temporal Bilayer Segmentation

104. Results Temporal Bilayer Segmentation

105. Quantitative Results Hand labeled ground truth (any 5 th /10 th frame) Percentage of misclassified Temporal Bilayer Segmentation

106. Quantitative Results Hand labeled ground truth (any 5 th /10 th frame) Percentage of misclassified Temporal Bilayer Segmentation

107. Limitations High illuminance changes  Failure (2/6 seq’s) Recommend: switch off Auto Gain Control Stereo V Monocular X Temporal Bilayer Segmentation

108. Summary LDPLGCBackground Cut Temporal Bilayer Seg. Colour/ Contrast VVVV Colour Model GMM’s Histograms Background maintenance VVV– Disparities ExplicitImplicit –– Background Attenuation ––V– Motion –––V Temporal Coherence –––V

109. Another Approach to Background Substituition

110. Thank You! A special thank to Dr. Vladimir Kolmogorov and to Eli Shechtman for their assitatnce 7.5.2006, Weizmann Institute of Science

111. F.A.Q. Alon Rubin Shira Kritchman 7.5.2006, Weizmann Institute of Science

112. Likelihood for Matching Empirical test – Is N discriminative? –Take labeled data –Compute and discretize N –Count matched pixels for each N –Count occluded pixels for each N –Divide Get: Likelihood ratio of matching as a function of N Stereo Bilayer Segmentation

113. Likelihood for Matching Example: Take N=0.1 –15% of matched pixels have N=0.1 –5% of occluded pixels have N=0.1 => likelihood ratio for N=0.1 Get: Likelihood ratio of matching as a function of N Stereo Bilayer Segmentation

114. Likelihood for Matching Stereo Bilayer Segmentation Empirical results –X axis: Discretized values of N –Y axis: Log-likelihood ratio

115. Quantitative Results Hand labeled ground truth (any 5 th /10 th frame) Percentage of misclassified Stereo Bilayer Segmentation

116. Stereo – Prior Parameters LDP: LGC: Working parameters: Stereo Bilayer Segmentation

117. Contrast Term Background Cut Alternative suggestion: Inter-label attenuation

118. Dynamic updating of and Background Cut

119. Kullback-Liebler Divergence Background Cut

120. VS Basic Model Background Cut

121. Background Maintenance Background Cut

122. Temporal Coherence Why 2 nd order? Temporal Bilayer Segmentation

123. Motion Classifier Why use spatial derivatives? Temporal Bilayer Segmentation

124. Minimizing the Energy Instead: Temporal Bilayer Segmentation

125. Results Temporal Bilayer Segmentation

126. Results With colour Without colour Temporal Bilayer Segmentation