Learning Jigsaws for clustering appearance and shape John Winn, Anitha Kannan and Carsten Rother NIPS 2006

Learning jigsaws Aim: Cluster regions in images with similar appearance and shape. Examples of clusters (jigsaw pieces) Eye Noses Cheek Eyebrows

Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

Clustering image patches Patches Clusters [Leibe & Schiele, BMVC 2003]

Clustering image patches Cluster? Patch wrong shape

Clustering image patches Cluster? Patch wrong shape

Clustering image patches Cluster? Part is occluded

Clustering image patches Cluster? Need to adapt the patch shape depending on the image.

Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

Aims of jigsaw model Learn clusters (jigsaw pieces) so that: 1. Clustered patches have similar shape and appearance 2. Patches are as large as possible 3. Every image pixel belongs to exactly one patch (i.e. the images are segmented into patches)

The Jigsaw model Jigsaw J Image I 1... Image I 2 I N Offset map L 2 L N L 1 Region of constant offset

The Jigsaw model Jigsaw J Offset map prior (Potts model) Appearance model Jigsaw Mean μ(z) and inverse variance λ(z) for each jigsaw pixel z. Image I Offset map L offset at pixel i cost of patch boundary

Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

Toy example Learned by iteratively maximising joint probability w.r.t. jigsaw and offset maps (see paper for details) Imagewith segmentationJigsaw MeanVariance

Comparison: Mixture of Gaussians  fixed patch shape Cluster centres

Comparison: Epitome [Jojic et al., ICCV 2003]  fixed patch shape  translation invariant Epitome

Comparison: Jigsaw  learned patch shape  translation invariant  non-overlapping patches Jigsaw

Comparison: all methods Original Jigsaw Epitome Error = Error = MoG Error = 0.103

Faces example Source: Olivetti face database Face imageswith segmentations Jigsaw 128  128 mean

Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

Learning the jigsaw pieces Jigsaw J... Image I 1 I 2 I N Offset map L 2 L N L 1

Shape clustering on faces Jigsaw showing pieces Commonly used pieces

Road map  Clustering image patches  The Jigsaw model  Results on toy and real images  Learning jigsaw pieces  Discussion and conclusions

Jigsaw applications  Can be used as ‘plug-and-play’ replacement for fixed-shape patch model in existing systems.  Applications include:  Object recognition/detection  Object segmentation  Stereo matching  Texture synthesis  Super-resolution  Motion segmentation  Image/video compression

Future work  Allow rotation/scaling/deformation of the patches.  Incorporate shape clustering into the probabilistic model  Incorporate additional invariances e.g. to illumination  Apply to other domains: audio, biology

Conclusions  Jigsaw model allows learning the shape and appearance of recurring regions in images.  Jigsaw performs unsupervised discovery of object parts.

Thank you Jigsaw paper (compressed)

Comparison: Epitome [Jojic et al., ICCV 2003]  fixed patch shape  translation invariant  overlapping patches Epitome

Patch averaging Error = 0.071Error = EpitomeMoG