Boltzmann Machines and their Extensions S. M. Ali Eslami Nicolas Heess John Winn March 2013 Heriott-Watt University

Goal Define a probabilistic distribution on images like this: 2

What can one do with an ideal shape model? 3 Segmentation

Weizmann horse dataset 4 Sample training images 327 images

What can one do with an ideal shape model? 5 Image

What can one do with an ideal shape model? 6 Computer graphics

Energy based models 7 Gibbs distribution

Shallow architectures 8 Mean

Shallow architectures 9 MRF

Existing shape models 10 Most commonly used architectures MRFMean sample from the model

What is a strong model of shape? We define a strong model of object shape as one which meets two requirements: 11 Realism Generates samples that look realistic Generalization Can generate samples that differ from training images Training images Real distribution Learned distribution

Shallow architectures 12 HOP-MRF

Shallow architectures 13 RBM

Shallow architectures 14 The effect of the latent variables can be appreciated by considering the marginal distribution over the visible units: Restricted Boltzmann Machines

In fact, the hidden units can be summed out analytically. The energy of this marginal distribution is given by: Shallow architectures 15 Restricted Boltzmann Machines where

All hidden units are conditionally independent given the visible units and vice versa. Shallow architectures 16 Restricted Boltzmann Machines

RBM inference 17 Block-Gibbs MCMC

RBM inference 18 Block-Gibbs MCMC

RBM learning Maximize with respect to 19 Stochastic gradient descent

RBM learning Getting an unbiased sample of the second term, however is very difficult. It can be done by starting at any random state of the visible units and performing Gibbs sampling for a very long time. Instead: 20 Contrastive divergence

RBM inference 21 Block-Gibbs MCMC

RBM inference 22 Block-Gibbs MCMC

RBM learning Crudely approximating the gradient of the log probability of the training data. More closely approximating the gradient of another objective function called the Contrastive Divergence, but it ignores one tricky term in this objective function so it is not even following that gradient. Sutskever and Tieleman have shown that it is not following the gradient of any function. Nevertheless, it works well enough to achieve success in many significant applications. 23 Contrastive divergence

Deep architectures 24 DBM

Deep architectures 25 Deep Boltzmann Machines

Conditional distributions remain factorised due to layering. Deep architectures 26 Deep Boltzmann Machines

Shallow and Deep architectures 27 Modeling high-order and long-range interactions MRF RBM DBM

Deep Boltzmann Machines Probabilistic Generative Powerful Typically trained with many examples. We only have datasets with few training examples. 28 DBM

From the DBM to the ShapeBM 29 Restricted connectivity and sharing of weights DBMShapeBM Limited training data, therefore reduce the number of parameters: 1.Restrict connectivity, 2.Tie parameters, 3.Restrict capacity.

Shape Boltzmann Machine 30 Architecture in 2D Top hidden units capture object pose Given the top units, middle hidden units capture local (part) variability Overlap helps prevent discontinuities at patch boundaries

ShapeBM inference 31 Block-Gibbs MCMC image reconstructionsample 1sample n Fast: ~500 samples per second

ShapeBM learning Maximize with respect to 1.Pre-training Greedy, layer-by-layer, bottom-up, ‘Persistent CD’ MCMC approximation to the gradients. 2.Joint training Variational + persistent chain approximations to the gradients, Separates learning of local and global shape properties. 32 Stochastic gradient descent ~2-6 hours on the small datasets that we consider

Results

Weizmann horses – 327 images – hidden units Sampled shapes 34 Evaluating the Realism criterion Weizmann horses – 327 images Data FA Incorrect generalization RBM Failure to learn variability ShapeBM Natural shapes Variety of poses Sharply defined details Correct number of legs (!)

Weizmann horses – 327 images – hidden units Sampled shapes 35 Evaluating the Realism criterion Weizmann horses – 327 images This is great, but has it just overfit?

Sampled shapes 36 Evaluating the Generalization criterion Weizmann horses – 327 images – hidden units Sample from the ShapeBM Closest image in training dataset Difference between the two images

Interactive GUI 37 Evaluating Realism and Generalization Weizmann horses – 327 images – hidden units

Further results 38 Sampling and completion Caltech motorbikes – 798 images – hidden units Training images ShapeBM samples Sample generalization Shape completion

Constrained shape completion 39 Evaluating Realism and Generalization Weizmann horses – 327 images – hidden units ShapeBM NN

Further results 40 Constrained completion Caltech motorbikes – 798 images – hidden units ShapeBM NN

Imputation scores 1.Collect 25 unseen horse silhouettes, 2.Divide each into 9 segments, 3.Estimate the conditional log probability of a segment under the model given the rest of the image, 4.Average over images and segments. 41 Quantitative comparison Weizmann horses – 327 images – hidden units MeanRBMFAShapeBM Score

Multiple object categories Train jointly on 4 categories without knowledge of class: 42 Simultaneous detection and completion Caltech-101 objects – 531 images – hidden units Shape completion Sampled shapes

What does h 2 do? Weizmann horses Pose information 43 Multiple categories Class label information Number of training images Accuracy

What does h 2 do? 44

What does the overlap do? 45

Summary Shape models are essential in applications such as segmentation, detection, in-painting and graphics. The ShapeBM characterizes a strong model of shape: – Samples are realistic, – Samples generalize from training data. The ShapeBM learns distributions that are qualitatively and quantitatively better than other models for this task. 46

Questions MATLAB GUI available at