1. with Daniel L. Silver, Ph.D. Christian Frey, BBA April 11-12, 2017
Deep Belief Networks
with
Daniel L. Silver, Ph.D.
Christian Frey, BBA
April 11-12, 2017
11/11/2018
Deep Learning Workshop

2. Feature detectors

3. What is this unit doing?

4. Hidden layer units become self-organised feature detectors… … 1
strong +ve weight
low/zero weight 63

5. What does this unit detect?… … 1
strong +ve weight
low/zero weight 63

6. What does this unit detect?… … 1
strong +ve weight
low/zero weight
it will send strong signal for a horizontal
line in the top row, ignoring everywhere else 63

7. And .. What is this unit doing ?

8. What does this unit detect?… … 1
strong +ve weight
low/zero weight 63

9. What does this unit detect?… … 1
strong +ve weight
low/zero weight
Strong signal for a dark area in the top left
corner 63

10. What features might you expect a good NN
to learn, when trained with data like this?

11. vertical lines 1 63

12. Horizontal lines 1 63

13. Small circles 1 63

14. Small circles But what about position, scale invariance?1
But what about position, scale invariance?
the feature detectors are tied to specific parts of the image 63

15. successive layers can learn higher-level features …
Nodes of Layer 1:
detect horizontal dark line in specific positions
etc …
Nodes of Layer 2:
detect the presence of :
-a horizontal dark line,
-a vertical dark line, or
-a dark circular area
in any position
etc … v

16. successive layers can learn higher-level features …
Nodes of Layer 1:
detect horizontal dark line in specific positions
etc …
Nodes of Layer 2:
detect the presence of :
-a horizontal dark line,
-a vertical dark line, or
-a dark circular area
in any position
etc … v
Nodes of Output Layer:
combine features to
to indicate image is a 9

17. So: multiple layers make sense
Many-layer neural network architectures have the ability to learn the underlying feature layers and generalise well to new test cases…

18. But how can we train BP NNs so that we overcome the vanishing gradient problem

19. The new way to train multi-layer NNs…
Develop h1 features
using an autoencoder

20. Use an Autoencoder
BP autoencoder
RBM autoencoder

21. Stacked Auto-Encoders
Bengio (2007) – After Deep Belief Networks (2006)
Stack many (sparse) auto-encoders in succession and train them using greedy layer-wise training
Drop the decode output layer each time

22. Deep Belief Networks

23. The new way to train multi-layer NNs…
Develop h1 features
Develop h2 features

24. The new way to train multi-layer NNs…
Develop h1 features
Develop h2 features
Develop h3 features

25. The new way to train multi-layer NNs…
Develop h1 features
Develop h2 features
Develop h3 features
Develop h4 features

26. The new way to train multi-layer NNs…
Develop h1 features
Final layer trained to predict class based on outputs from previous layers
Develop h2 features
Develop h3 features
Develop h4 features
Add a final supervised layer

27. Wide Variety of DBNs
Can be used for any problem, makes no assumption about inputs
Variations in architectures
Different kinds of autoencoders
Supervised learning of just top layer(s)
Supervised learning of all layers

28. TUTORIAL 8 Develop and train a a DBN using Tensorflow (Python code)
Rbm.py

29. Restricted Boltzmann Machine (RBM)
Does not allow intra-layer connections (bipartite architecture)
Learning both recognition model and generative model

30. Deep Learning Workshop
RBM
11/11/2018
Deep Learning Workshop