1. Learning a Region-based Scene Segmentation ModelF O R D
Learning a Region-based Scene Segmentation Model
M. Pawan Kumar
Daphne Koller

2. Aim To learn an accurate scene segmentation model
Divide image into non-overlapping regions
Assign each region to a semantic class
Features extracted from each region
Spatial prior

3. Why Regions?
Black or Brown?
Shape?

4. Why Regions?
Black or Brown?
Shape?

5. Why Regions? Two concentric circles (shape)
Inner circle is metallic (texture)
Outer circle is black (color)

6. Which Regions? Bottom-up Over-segmentation Mean-Shift, N-cuts
Too small to capture useful cues
Not faithful to boundaries between scene entities

7. Which Regions? Choose regions according to a global energy function
Regions that give the best accuracy

8. Outline Region-based Segmentation Model Learning from Coarse Labels
Inference
Results

9. Region-based Segmentation Model
G = (V(fP),E(fP))
fR: Regions  Classes
fP: Pixels  Regions
Unknown
Number
Gould et al., 2009

10. Region-based Segmentation Model
E(f) = ∑w1(fR(r))T r(fP=r) + ∑w2T rs(fP=r,fP=s)
Region Features
Region Pairwise Features
Boundary Contrast
Shape, Texture, Color
Internal Pixels Contrast
Fraction of pixels above/below horizon
w : Model parameters to be learnt
Gould et al., 2009

11. Outline Region-based Segmentation Model Learning from Coarse Labels
Inference
Results

12. Ground Truth Labeling NO Amazon’s Mechanical Turk
Are they the best regions?
Car
Top-half (transparent)
Bottom-half (solid)
Wheels (circular)
Tree
Crown (leafy)
Trunk (brown)

13. Ground Truth Labeling
Desired ground truth f*
Coarser version fC

14. Learning with Coarse Labels
Refined labeling fR faithful to coarse label fC
∑ iwTi* – log Zi(w)
HUGE summation!!
Marginalize over fR??
Very difficult!!
MAP is bread-and-butter of Vision
An accurate MAP estimation algorithm !!!

15. Learning with Coarse Labels
Use max-margin learning framework
At each iteration:
Complete the ground truth
Find the most violated constraint
Approximate MAP Inference

16. Outline Region-based Segmentation Model Learning from Coarse Labels
Inference
Results

17. Region Selection
Image
Dictionary of Regions
Human Annotation

18. Region Selection Image Dictionary of Regions Each super-pixel covered
by exactly one selected region
Super-Pixels

19. Integer Program miny ∑ r(i)yr(i) + ∑ rs(i,j)yrs(i,j)
Binary yr(0) = 1 iff r is not selected
Binary yr(1) = 1 iff r is selected
miny ∑ r(i)yr(i) + ∑ rs(i,j)yrs(i,j)
Minimize the energy
s.t. yr(0) + yr(1) = 1
Assign one label to r from L
yrs(i,0) + yrs(i,1) = yr(i)
Ensure yrs(i,j) = yr(i)ys(j)
yrs(0,j) + yrs(1,j) = ys(j)
∑r “covers” u yr(1) = 1
Each super-pixel is covered
by exactly one selected region
yr(i), yrs(i,j)  {0,1}
Binary variables

20. Linear Program miny ∑ r(i)yr(i) + ∑ rs(i,j)yrs(i,j)
Binary yr(0) = 1 iff r is not selected
Binary yr(1) = 1 iff r is selected
miny ∑ r(i)yr(i) + ∑ rs(i,j)yrs(i,j)
s.t. yr(0) + yr(1) = 1
yrs(i,0) + yrs(i,1) = yr(i)
yrs(0,j) + yrs(1,j) = ys(j)
∑r “covers” u yr(1) = 1
yr(i), yrs(i,j)  [0,1]

21. Linear Program Clique of overlapping and neighboring regions
(non-submodular potentials)
Mutual exclusivity
Covering constraint
Computationally expensive?
Efficient Dual Decomposition

22. The Learning Approach At each iteration Given (i) fC (coarse labeling)
(ii) current set of parameters
Find fR (refined labeling)
Region Partitioning Problem
Most violated constraint
Region Partitioning + Label assignment

23. Outline Region-based Segmentation Model Learning from Coarse Labels
Inference
Results

24. Stanford Background Dataset
715 outdoor scenes
7 background + 1 foreground
Amazon’s Mechanical Turk
Available for download:

25. Completing the Ground Truth

26. Completing the Ground Truth

27. Completing the Ground Truth

28. Pixel-wise Accuracy Pixel-based Model: 67.71%
Human Labeled Regions: 64.85%
Our Approach: 72.49%

29. Examples

30. Examples

31. Examples

32. Summary Coarse labels are easy to obtain
Refined labels are easy to train with
Marginalization is not possible
Approximate MAP (required for testing) is used to complete the labeling

33. Future Work Convergence analysis Region features + context
Learning with Different Labelings

34.
Questions?