1. Learning Jigsaws for clustering appearance and shape John Winn, Anitha Kannan and Carsten Rother NIPS 2006

2. Learning jigsaws Aim: Cluster regions in images with similar appearance and shape. Examples of clusters (jigsaw pieces) Eye Noses Cheek Eyebrows

3. Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

4. Clustering image patches Patches Clusters [Leibe & Schiele, BMVC 2003]

5. Clustering image patches Cluster? Patch wrong shape

6. Clustering image patches Cluster? Patch wrong shape

7. Clustering image patches Cluster? Part is occluded

8. Clustering image patches Cluster? Need to adapt the patch shape depending on the image.

9. Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

10. Aims of jigsaw model Learn clusters (jigsaw pieces) so that: 1. Clustered patches have similar shape and appearance 2. Patches are as large as possible 3. Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)

11. The Jigsaw model Jigsaw J Image I 1... Image I 2 I N Offset map L 2 L N L 1 Region of constant offset

12. The Jigsaw model Jigsaw J Offset map prior (Potts model) Appearance model Jigsaw Mean μ(z) and inverse variance λ(z) for each jigsaw pixel z. Image I Offset map L offset at pixel i cost of patch boundary

13. Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

14. Toy example Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps (see paper for details) Imagewith segmentationJigsaw MeanVariance

15. Comparison: Mixture of Gaussians  fixed patch shape Cluster centres

16. Comparison: Epitome [Jojic et al., ICCV 2003]  fixed patch shape  translation invariant Epitome

17. Comparison: Jigsaw  learned patch shape  translation invariant  non-overlapping patches Jigsaw

18. Comparison: all methods Original Jigsaw Epitome Error = 0.054 Error = 0.071 MoG Error = 0.103

19. Faces example Source: Olivetti face database Face imageswith segmentations Jigsaw 128  128 mean

20. Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

21. Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

22. Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

23. Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

24. Shape clustering on faces Jigsaw showing pieces Commonly used pieces

25. Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

26. Jigsaw applications  Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems.  Applications include:  Object recognition/detection  Object segmentation  Stereo matching  Texture synthesis  Super-resolution  Motion segmentation  Image/video compression

27. Future work  Allow rotation/scaling/deformation of the patches.  Incorporate shape clustering into the probabilistic model  Incorporate additional invariances e.g. to illumination  Apply to other domains: audio, biology

28. Conclusions  Jigsaw model allows learning the shape and appearance of recurring regions in images.  Jigsaw performs unsupervised discovery of object parts.

29. Thank you Jigsaw paper (compressed) http://johnwinn.org

31. Comparison: Epitome [Jojic et al., ICCV 2003]  fixed patch shape  translation invariant  overlapping patches Epitome

32. Patch averaging Error = 0.071Error = 0.054 EpitomeMoG