1. Restricted Boltzmann Machines for Classification
Jin Mao
Postdoc, School of Information, University of Arizona
Nov. 3rd, 2016

2. Outline Restricted Boltzmann Machines RBM For Classification
Semi-supervised Learning
Experiments

3. Restricted Boltzmann Machines
General RBM
Boltzmann Machines (BMs) are a particular form of log-linear Markov Random Field (MRF), i.e., the energy function is linear in its parameters. Restrict: no visible unit is connected to any other visible unit and no hidden unit is connected to any other hidden unit.

4. Restricted Boltzmann Machines
General RBM
The energy function for x: The data negative log-likelihood gradient:
log-linear Z

5. Restricted Boltzmann Machines
Learning Weights
Set the states of the visible units according to a training example
Next, update the states of the hidden units
For each hidden unit, compute its activation energy
For each edge, compute positive weight for each edge
Positive(e_ij)

6. Restricted Boltzmann Machines
Learning Weights
Now reconstruct the visible units
Update the weight of each edge
For each visible unit, compute its activation energy
For each edge, compute negative weight for each edge
Negative(e_ij)
measures the association that the network itself generates (or “daydreams” about) when no units are fixed to training data.

7. Restricted Boltzmann Machines
Learning Weights
Test convergence by KL-Divergence

8. Restricted Boltzmann Machines
Application
The power lies in the hidden layer
In classification problems
as feature extractors (Gehler et al., 2006)
as a good initial training phase for deep neural network classifiers (Hinton,2007).

9. RBM For Classification
Generative Model
Objective:
Energy Function:
Gradient:
Activation Functions:

10. RBM For Classification
Discriminative Model
Objective:
Energy Function:
Gradient:

11. RBM For Classification
Hybrid Model
Objective:

12. Semi-supervised Learning
There are few labeled training data but many unlabeled examples of inputs.
Semi-supervised learning algorithms (Chapelle et al., 2006) address this situation by using the unlabeled data to introduce constraints on the trained model.
In the RBM framework, a natural constraint is
the model should be a good generative model of the unlabeled data
Unsupervised objectives:

13. Semi-supervised Learning
Combine unsupervised objective with supervised objective

14. Experiments
Model Selection
Experiments on two classification problems: character recognition and text classification.
In all experiments, we performed model selection on a validation set before testing.
For the different RBM models, model selection consisted in finding good values for the learning rate, the size of the hidden layer n and good weights for the different types of learning (generative and semi-supervised weights).

15. Experiments Model Selection Document Classification
Character Recognition
Document Classification

16. Reference Other materials
This presentation is from: Larochelle, H., & Bengio, Y. (2008, July). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on Machine learning (pp ). ACM.
Other materials
Hinton, G. (2010). A practical guide to training restricted Boltzmann machines. Momentum, 9(1), 926.

17. Thank you!