1. Contour-Based Joint Clustering of Multiple Segmentations Daniel Glasner * 1 Shiv N. Vitaladevuni * 2 Ronen Basri 1 * equal contribution authors 1 2

2. 1 2 3 1 2 3 1 2 3 1 2 3 Joint-Clustering of Image Segments Objective: 1. Similar shape across frames 2. Coherent regions within frames Segments / Super-pixels Our clustering result Input frame Object contours  can be matched Oversegmentation artifacts  can not

3. Large deformation histograms are similar Single object Inter image similarity: Overlap Color Related WorkThis Work Co-segmentation [Rother et al. 06, Bagon et al. 08, Hochbaum et al. 09, Joulin et al. 10] Co-clustering of image segments[Vitaladevuni and Basri 2010] Small deformation shapes are similar Full image segmentation # objects unknown Inter image similarity: Shape Intra Image similarity: Color / optical flow etc.

4. Different lighting conditions Some Applications Video segmentation EM slices [Vitaladevuni & Basri CVPR10]

5. ( ) + Coherency ( ) F(union) = Shape-sim (, ) Shape similarity across frames × Coherency within frame ✓ ✓ Searching for a Good Cluster Convex functional of unions of segments 1.Bounding contours match - across frames 2.Coherent regions - within frame Exterior bounding contours match

6. Incorporating Shape Input frames Shape-based joint clustering Intersection Shape-based similarity Shape-based similarity

7. Descriptor (segment) = bounding contour A Novel Contour Descriptor 3 dimension = # contour samples in image 1 Image 1 0 e i(θ+  0 e iα eiei eiθeiθ 3 3

8. Descriptor (union) = its bounding contour Additive Descriptor & Contour Cancellation eiei eiθeiθ 0 e i(θ+  union eiei 0 + = dimension = #contour samples

9. Comparing Shapes Across Images B2B2 B1B1 0 0eiθ0eiθ # contour elements image 2 # segments image 2 # contour elements image 1 # segments image 1 Image 2Image 1

10. Comparing Shapes Across Images B2B2 x2x2 B1B1 = = Contour descriptor in image 2 Contour descriptor in image 1 x1x1 Image 2Image 1 Binary indicator of segments in image 1 Binary indicator of segments in image 2

11. B2B2 x2x2 B1B1 = = x1x1 Shape-sim (, ) = ? Correspondence Matrix W 1,2 B1HB1H x1Tx1T ? # contour elements image1 X # contour elements image2 Comparing Shapes Across Images

12. B2B2 x2x2 W 1,2 x1Tx1T B1HB1H Contour descriptor of shape in image 1 Contour descriptor of shape in image 2 Q 1,2 = # segments image1 X # segments image2 # contour elements image1 X # contour elements image2

13. 1.Similar shape across frames 2.Coherent regions within frames = = Shape-sim (, ) xTxT x Q 1,2 Q 1,2H = For arbitrary selection of segments Q 1,1 Q 2,2 ( ) + Coherency ( )

14. Optimization Vitaladevuni and Basri CVPR 2010 NP-hard [Garey and Johnson 1979]

15. Convex Relaxation Efficient l inear programming relaxation [ Vitaladevuni & Basri 2010, Charikar et al. 2003 ] Q is hermitian  objective is real

16. Video Dataset for Occlusion/Object Boundary Detection A.Stein and M. Hebert. Occlusion boundaries from motion: low-level detection and mid-level reasoning. IJCV, 82(3), 2009. 2 http://www.cs.cmu.edu/~stein/occlusion_data/ reference frame ground truth

17. Qualitative Results Reference frame: Our result:

18. Qualitative Results Reference frame: Our result:

19. Summary Joint segmentation of closely related images Additive contour representation – Ignores internal contours of unions of segments Efficient convex optimization to find subsets of segments with a similar shape Combine inter and intra image similarity

20. Thank you!