Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March,

Outline Hopfield Networks Boltzman Machines Mean Field Theory Restricted Boltzman Machines (RBM) 2

Dynamic Neural Networks See handout for image of spider, beer and dog The search for a model or hypothesis can be considered the relaxation of a dynamic system into a state of equilibrium This is the nature of most physical systems – Pool of water – Air in a room Mathematics is that of thermal-dynamics – Quote from John Von Neumann 3

Hopfield Networks See hand out 4

Hopfield Networks Hopfield Network video intro – Y Y – Try these Applets: – ndex.html ndex.html – pplet.html pplet.html 5

Hopfield Networks Basics with Geoff Hinton: Introduction to Hopfield Nets – Storage capacity of Hopfield Nets – 6

Hopfield Networks Advanced concepts with Geoff Hinton: Hopfield nets with hidden units – Necker Cube – er.html er.html Adding noise to improve search – 7

Boltzman Machine -See Handout - Basics with Geoff Hinton Modeling binary data – BM Learning Algorithm – 8

Limitations of BMs BM Learning does not scale well This is due to several factors, the most important being: – The time the machine must be run in order to collect equilibrium statistics grows exponentially with the machine's size = number of nodes For each example – sample nodes, sample states – Connection strengths are more plastic when the units have activation probabilities intermediate between zero and one. Noise causes the weights to follow a random walk until the activities saturate (variance trap). 9

Potential Solutions Use a momentum term as in BP: Add a penalty term to create sparse coding (encourage shorter encodings for different inputs) Use implementation tricks to do more in memory – batches of examples Restrict number of iterations in + and – phases Restrict connectivity of network 10 w ij (t+1)=w ij (t) +ηΔw ij +αΔw ij (t-1)

Restricted Boltzman Machine 11 Source: SF/Fantasy Oscar Winner w ij j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Recall = Relaxation Σ i =w ij h j v o or h o

Restricted Boltzman Machine 12 Source: SF/Fantasy Oscar Winner w ij j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Recall = Relaxation Σ i =w ij h j v o or h o

Restricted Boltzman Machine 13 Source: SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o Oscar Winner SF/Fantasy Recall = Relaxation w ij Σ j =w ij v i

Restricted Boltzman Machine 14 Source: SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o Oscar Winner SF/Fantasy Recall = Relaxation w ij Σ j =w ij v i

Restricted Boltzman Machine 15 Source: SF/Fantasy Oscar Winner j i Σ i =w ij h j h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Update hidden units P=P+v i h j v o or h o Σ j =w ij v i

Restricted Boltzman Machine 16 Source: SF/Fantasy Oscar Winner j i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Reconstruct visible units v o or h o Σ j =w ij v i Σ i =w ij h j

Restricted Boltzman Machine 17 Source: SF/Fantasy Oscar Winner j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Learning = ~ Gradient Descent = Constrastive Divergence Reupdate hidden units v o or h o Σ i =w ij h j N=N+v i h j

Restricted Boltzman Machine 18 Source: SF/Fantasy Oscar Winner Δw ij = - j i Σ j =w ij v i h j  p j =1/(1-e -Σj ) v i  p i =1/(1-e -Σi ) Σ i =w ij h j v o or h o w ij =w ij +ηΔw ij Learning = ~ Gradient Descent = Constrastive Divergence Update weights

Restricted Boltzman Machine RBM Overview: – to-restricted-boltzmann-machines/ to-restricted-boltzmann-machines/ Wikipedia on DLA and RBM: – RBM Details and Code: –

Restricted Boltzman Machine Geoff Hinton on RBMs: RBMs and Constrastive Divergence Algorithm – An example of RBM Learning – RBMs applied to Collaborative Filtering –

Additional References Coursera course – Neural Networks fro Machine Learning: – 001/lecture 001/lecture ML: Hottest Tech Trend in next 3-5 Years –