1. Jigsaws: joint appearance and shape clustering John Winn with Anitha Kannan and Carsten Rother Microsoft Research, Cambridge

2. Patch models Used for:  Object recognition/detection  Object segmentation But also:  Stereo matching, photo stitching  Texture synthesis  Super-resolution  Motion segmentation  Image/video compression

3. Patch models  Patch clustering/codebook (e.g. Leibe & Schiele)  Epitome (Jojic et al.) parameter sharing + translation invariant

4. Issues with fixed patch size/shape  Patch includes background patches containing the same object are not clustered together  Patch excludes part of object patch is less discriminative  Patch includes occlusion occluded and unoccluded objects are not clustered together

5. Patch size? Small (single pixel) Large (entire image) More discriminative Less sharing More sharing Less discriminative Optimal size/shape? Depends on: object size/shape object variability size of training set Size

6. Aims of jigsaw model Learn patches (jigsaw pieces) which are 1. Shared: each piece is similar in shape and appearance to many regions of the training images; 2. Discriminative: each piece is as large as possible; 3. Exhaustive: all parts of the training images can be reconstructed from the set of jigsaw pieces.

7. The Jigsaw model ImageI 1 Offset map L 1... ImageI 2 Offset map L 2 ImageI N Offset map L N Jigsaw J

8. The Jigsaw model Jigsaw J ImageI 1 Offset map L 1... ImageI 2 Offset map L 2 ImageI N Offset map L N

9. The Jigsaw model Jigsaw J ImageI 1 Offset map L 1... ImageI 2 Offset map L 2 ImageI N Offset map L N Potts model:

10. Toy example Training image Jigsaw Learned using EM + graph cuts

11. Dog example Training image 3232 Jigsaw mean

12. Dog example Reconstructed image Learned segmentation 3232 Jigsaw mean Epitome reconstruction

13. Faces example 128128 Jigsaw mean 100 6464 images Source: Olivetti face database

14. Learning the ‘pieces’ ImageI 1 Offset map L 1... ImageI 2 Offset map L 2 ImageI N Offset map L N Jigsaw J

15. Learning the ‘pieces’ Jigsaw J

16. Faces example Results of shape clustering on the face images

17. 64x64 jigsaw Object recognition (preliminary)  Trained set: 20 street images Allow patches to deform (as in LayoutCRF, CVPR 2006).

18. Object recognition (preliminary)  Trained set: 20 street images (10 labelled) 64x64 jigsaw Accuracy improves (~1%) if you include an additional 10 unlabelled images when learning the jigsaw. Allow patches to deform (as in LayoutCRF, CVPR 2006).

19. Work in progress…  Training larger jigsaws on 100s of images  Incorporating shape clustering into the probabilistic model  Learning additional invariances e.g. to illumination  Object recognition results on MSRC and other datasets

20. Conclusions  Jigsaw model allows learning the shape and appearance of objects or object parts in images. Can also handle occlusion.  Clustering shape and appearance much more powerful for recognition than appearance alone.  Can be used as a ‘plug-and-play’ replacement for fixed size patches in any existing patch- based system.

21. Thank you jwinn@microsoft.com http://johnwinn.org