1. Deep belief nets experiments and some ideas.
Karol Gregor
NYU/Caltech

2. Outline
DBN Image database experiments
Temporal sequences

3. Deep belief network
Backprop
Labels H3 H2 H1
Input

4. Preprocessing – Bag of words of SIFT
With: Greg Griffin (Caltech)
Images
Features (using SIFT)
Bag of words
Image1 Image2
Word
Word
Word
… … …
Group them (e.g. K-means)

5. 13 Scenes Database – Test error

6. Train error

7. - Pre-training on larger dataset
- Comparison to svm, spm

8. Explicit representations?

9. Compatibility between databases
Pretraining: Corel database
Supervised training: 15 Scenes database

10. Conclusions Bag of words is not a good input for deep architectures
The networks can be pretrained on one database and the supervised
training can be used on other one.
Other observations:

11. Temporal Sequences

12. Simple prediction Y t W
t-1
t-2
t-3 X
Supervised learning

13. With hidden units (need them for several reasons)G H
t-1,t-2,t-3 t
t-1,t-2,t-3 t X Y
Memisevic, R. F. and Hinton, G. E., Unsupervised Learning of Image Transformations. CVPR-07

14. Example
pred_xyh_orig.m

15. G H
t-1 t
Additions
t-1 t X Y
Sparsity: When inferring the H the first time, keep only the largest n units on
Slow H change: After inferring the H the first time, take H=(G+H)/2

16. Examples
pred_xyh.m
present_line.m
present_cross.m

17. Hippocampus Cortex+Thalamus Senses Muscles
e.g. Eye (through retina, LGN)
Muscles
(through sub-cortical structures)
e.g. See: Jeff Hawkins: On Intelligence

18. Cortical patch: Complex structure (not a single layer RBM)
From Alex Thomson and Peter Bannister, (see numenta.com)

19. Desired properties

20. 1) Prediction
A B C D E F G
H J K L E F H

21. 2) Explicit representations for sequences
VISIONRESEARCH
time

22. 3) Invariance discovery
e.g. complex cell
time

23. 4) Sequences of variable length
VISIONRESEARCH
time

24. 5) Long sequences
Layer1
? ?
Layer2

25. 6) Multilayer
- Inferred only after some time
VISIONRESEARCH
time

26. 7) Smoother time steps

27. 8) Variable speed
- Can fit a knob with small speed range

28. 9) Add a clock for actual time

29. Hippocampus Cortex+Thalamus Senses Muscles
e.g. Eye (through retina, LGN)
Muscles
(through sub-cortical structures)

30. Hippocampus Cortex+Thalamus In Addition Senses Muscles
Top down attention
Bottom up attention
Imagination
Working memory
Rewards
Senses
e.g. Eye (through retina, LGN)
Muscles
(through sub-cortical structures)

31. Training data Of the real world Simplified: Cartoons (Simsons)
Videos
Of the real world
Simplified: Cartoons (Simsons)
A robot in an environment
Problem: Hard to grasp objects
Artificial environment with 3D objects that are easy to manipulate (e.g. Grand theft auto IV with objects)