with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017 Deep Belief Networks with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017 11/11/2018 Deep Learning Workshop
Feature detectors
What is this unit doing?
Hidden layer units become self-organised feature detectors 1 5 10 15 20 25 … … 1 strong +ve weight low/zero weight 63
What does this unit detect? 1 5 10 15 20 25 … … 1 strong +ve weight low/zero weight 63
What does this unit detect? 1 5 10 15 20 25 … … 1 strong +ve weight low/zero weight it will send strong signal for a horizontal line in the top row, ignoring everywhere else 63
And .. What is this unit doing ?
What does this unit detect? 1 5 10 15 20 25 … … 1 strong +ve weight low/zero weight 63
What does this unit detect? 1 5 10 15 20 25 … … 1 strong +ve weight low/zero weight Strong signal for a dark area in the top left corner 63
What features might you expect a good NN to learn, when trained with data like this?
vertical lines 1 63
Horizontal lines 1 63
Small circles 1 63
Small circles But what about position, scale invariance? 1 But what about position, scale invariance? the feature detectors are tied to specific parts of the image 63
successive layers can learn higher-level features … Nodes of Layer 1: detect horizontal dark line in specific positions etc … Nodes of Layer 2: detect the presence of : -a horizontal dark line, -a vertical dark line, or -a dark circular area in any position etc … v
successive layers can learn higher-level features … Nodes of Layer 1: detect horizontal dark line in specific positions etc … Nodes of Layer 2: detect the presence of : -a horizontal dark line, -a vertical dark line, or -a dark circular area in any position etc … v Nodes of Output Layer: combine features to to indicate image is a 9
So: multiple layers make sense Many-layer neural network architectures have the ability to learn the underlying feature layers and generalise well to new test cases…
But how can we train BP NNs so that we overcome the vanishing gradient problem
The new way to train multi-layer NNs… Develop h1 features using an autoencoder
Use an Autoencoder BP autoencoder RBM autoencoder
Stacked Auto-Encoders Bengio (2007) – After Deep Belief Networks (2006) Stack many (sparse) auto-encoders in succession and train them using greedy layer-wise training Drop the decode output layer each time
Deep Belief Networks
The new way to train multi-layer NNs… Develop h1 features Develop h2 features
The new way to train multi-layer NNs… Develop h1 features Develop h2 features Develop h3 features
The new way to train multi-layer NNs… Develop h1 features Develop h2 features Develop h3 features Develop h4 features
The new way to train multi-layer NNs… Develop h1 features Final layer trained to predict class based on outputs from previous layers Develop h2 features Develop h3 features Develop h4 features Add a final supervised layer
Wide Variety of DBNs Can be used for any problem, makes no assumption about inputs Variations in architectures Different kinds of autoencoders Supervised learning of just top layer(s) Supervised learning of all layers
TUTORIAL 8 Develop and train a a DBN using Tensorflow (Python code) Rbm.py
Restricted Boltzmann Machine (RBM) Does not allow intra-layer connections (bipartite architecture) Learning both recognition model and generative model
Deep Learning Workshop RBM 11/11/2018 Deep Learning Workshop