1. Associative-Memory Networks Input: Pattern (often noisy/corrupted) Output: Corresponding pattern (complete / relatively noise-free) Process 1.Load input pattern onto core group of highly-interconnected neurons. 2.Run core neurons until they reach a steady state. 3.Read output off of the states of the core neurons. InputsOutputs Input: (1 0 1 -1 -1) Output: (1 -1 1 -1 -1)

2. Associative Network Types 1. Auto-associative: X = Y 2. Hetero-associative Bidirectional: X <> Y *Recognize noisy versions of a pattern *Iterative correction of input and output BAM = Bidirectional Associative Memory

3. Associative Network Types (2) 3. Hetero-associative Input Correcting: X <> Y *Input clique is auto-associative => repairs input patterns 4. Hetero-associative Output Correcting: X <> Y *Output clique is auto-associative => repairs output patterns

4. Hebb’s Rule Connection Weights ~ Correlations ``When one cell repeatedly assists in firing another, the axon of the first cell develops synaptic knobs (or enlarges them if they already exist) in contact with the soma of the second cell.” (Hebb, 1949) In an associative neural net, if we compare two pattern components (e.g. pixels) within many patterns and find that they are frequently in: a) the same state, then the arc weight between their NN nodes should be positive b) different states, then ” ” ” ” negative Matrix Memory: The weights must store the average correlations between all pattern components across all patterns. A net presented with a partial pattern can then use the correlations to recreate the entire pattern.

5. Correlated Field Components Each component is a small portion of the pattern field (e.g. a pixel). In the associative neural network, each node represents one field component. For every pair of components, their values are compared in each of several patterns. Set weight on arc between the NN nodes for the 2 components ~ avg correlation. ?? a b a b Avg Correlation ba w ab

6. Quantifying Hebb’s Rule Compare two nodes to calc a weight change that reflects the state correlation: Hebbian Principle: If all the input patterns are known prior to retrieval time, then init weights as: * When the two components are the same (different), increase (decrease) the weight Ideally, the weights will record the average correlations across all patterns: Weights = Average Correlations Auto-Association: Hetero-Association: Auto: Hetero: Auto:Hetero: i = input component o = output component

7. Matrix Representation Let X = matrix of input patterns, where each ROW is a pattern. So x k,i = the ith bit of the kth pattern. Let Y = matrix of output patterns, where each ROW is a pattern. So y k,j = the jth bit of the kth pattern. Then, avg correlation between input bit i and output bit j across all patterns is: 1/P (x 1,i y 1,j + x 2,i y 2,j + … + x p,i y p,j ) = w i,j To calculate all weights: Hetero Assoc:W = X T Y Auto Assoc:W = X T X In Pattern 1: x 1,1..x 1,n In Pattern 2: x 2,1..x 2,n In Pattern p: x 1,1..x 1,n : X X 1,i.. P1 P2 Pp XTXT X 2,i X p,i Out P1: y 1,1.. y 1,j ……y 1,n Out P2: y 2,1.. y 2,j ……y 2,n Out P3: y p,1.. y p,j ……y p,n : Y Dot product

8. Auto-Associative Memory 1 node per pattern unit Fully connected: clique Weights = avg correlations across all patterns of the corresponding units 1 3 4 2 1. Auto-Associative Patterns to Remember 2. Distributed Storage of All Patterns: 1 34 2 1 34 2 1 34 2 1 34 2 3. Retrieval 1 1 34 2 1 34 2 Comp/Node value legend: dark (blue) with x => +1 dark (red) w/o x => -1 light (green) => 0

9. Hetero-Associative Memory 1 3 2b a 1 node per pattern unit for X & Y Full inter-layer connection Weights = avg correlations across all patterns of the corresponding units 1 3 b 2 1. Hetero-Associative Patterns (Pairs) to Remember 2. Distributed Storage of All Patterns: 3. Retrieval 1 3 2 b a a 1

10. Hopfield Networks Auto-Association Network Fully-connected (clique) with symmetric weights State of node = f(inputs) Weight values based on Hebbian principle Performance: Must iterate a bit to converge on a pattern, but generally much less computation than in back-propagation networks. InputOutput (after many iterations) Discrete node update rule: Input value

11. Hopfield Network Example 1 34 2 1 34 2 1. Patterns to Remember p1p1 p2p2 p3p3 2. Hebbian Weight Init: Avg Correlations across 3 patterns W 12 1 1 -1 1/3 p1p1 p2p2 p3p3 Avg W 13 1 -1 -1 -1/3 W 14 -1 1 1 1/3 W 23 1 -1 1 1/3 W 24 -1 1 -1 -1/3 W 34 -1 -1 -1 -1 1 3 4 2 [-] [+] -1/3 1/3 3. Build Network 4. Enter Test Pattern 1 34 2 -1/3 1/3 -1/3 1/3 1 34 2 +10

12. Hopfield Network Example (2) 5. Synchronous Iteration (update all nodes at once) Node1234 Output 1100-1/31 21/3001/31 3-1/30011 41/300-1-1 Inputs -1/3 1/3 -1/3 1/3 Stable State p1p1 1 34 2 = Values from Input Layer From discrete output rule: sign(sum)

13. Using Matrices Goal: Set weights such that an input vector Vi, yields itself when multiplied by the weights, W. X = V1,V2..Vp, where p = # input vectors (i.e., patterns) So Y=X, and the Hebbian weight calculation is: W = X T Y = X T X 1 1 -1 1 1 1 -1 1 1 1 X = 1 1 -1 1X T = 1 -1 1 -1 1 1 -1-1 1 -1 3 1 -1 1 Common index = pattern #, so X T X = 1 3 1 -1 this is correlation sum. -1 1 3 -3 1 -1 -3 3 w 2,4 = w 4,2 = x T 2,1 x 1,4 + x T 2,2 x 2,4 + x T 2,3 x 3,4

14. Matrices (2) The upper and lower triangles of the product matrix represents the 6 weights w i,j = w j,i Scale the weights by dividing by p (i.e., averaging). Picton (ANN book) subtracts p from each. Either method is fine, as long we apply the appropriate thresholds to the output values. This produces the same weights as in the non-matrix description. Testing with input = ( 1 0 0 -1) 3 1 -1 1 (1 0 0 -1) 1 3 1 -1 = (2 2 2 -2) -1 1 3 -3 1 -1 -3 3 Scaling* by p = 3 and using 0 as a threshold gives: (2/3 2/3 2/3 -2/3) => (1 1 1 -1) * For illustrative purposes, it’s easier to scale by p at the end instead of scaling the entire weight matrix, W, prior to testing.

15. Hopfield Network Example (3) 5b. Synchronous Iteration Node1234 Output 111/3001 21/31001 3-1/31/3000 41/3-1/3000 Inputs 4b. Enter Another Test Pattern 1 34 2 -1/3 1/3 -1/3 1/3 Input pattern is stable, but not one of the original patterns. Attractors in node-state space can be whole patterns, parts of patterns, or other combinations. Spurious Outputs

16. Hopfield Network Example (4) 5c. Asynchronous Iteration (One randomly-chosen node at a time) 4c. Enter Another Test Pattern 1 34 2 Update 3 -1/3 1/3 -1/3 1/3 -1/3 1/3 -1/3 1/3 Update 4 -1/3 1/3 -1/3 1/3 -1/3 1/3 -1/3 1/3 Update 2 Stable & Spurious Asynchronous Updating is central to Hopfield’s (1982) original model.

17. Hopfield Network Example (5) 5d. Asynchronous Iteration 4d. Enter Another Test Pattern 1 34 2 Update 3 -1/3 1/3 -1/3 1/3 -1/3 1/3 -1/3 1/3 Update 4 -1/3 1/3 -1/3 1/3 -1/3 1/3 -1/3 1/3 Update 2 Stable Pattern p 3

18. Hopfield Network Example (6) 5e. Asynchronous Iteration (but in different order) 4e. Enter Same Test Pattern 1 34 2 Update 2 -1/3 1/3 -1/3 1/3 -1/3 1/3 -1/3 1/3 Update 3 or 4 (No change) -1/3 1/3 -1/3 1/3 Stable & Spurious

19. Associative Retrieval = Search Back-propagation: Search in space of weight vectors to minimize output error Associative Memory Retrieval: Search in space of node values to minimize conflicts between a) node-value pairs and average correlations (weights), and b) node values and their initial values. Input patterns are local (sometimes global) minima, but many spurious patterns are also minima. High dependence upon initial pattern and update sequence (if asynchronous) p1p1 p2p2 p3p3

20. Energy Function Basic Idea: Energy of the associative memory should be low when pairs of node values mirror the average correlations (i.e. weights) on the arcs that connect the node pair, and when current node values equal their initial values (from the test pattern). When pairs match correlations, w kj x j x k > 0 When current values match input values, I k x k > 0 Gradient Descent A little math shows that asynchronous updates using the discrete rule: yield a gradient descent search along the energy landscape for the E defined above.

21. Storage Capacity of Hopfield Networks Capacity = Relationship between # patterns that can be stored & retrieved without error to the size of the network. Capacity = # patterns / # nodes or # patterns / # weights If we use the following definition of 100% correct retrieval: When any of the stored patterns is entered completely (no noise), then that same pattern is returned by the network; i.e. The pattern is a stable attractor. A detailed proof shows that a Hopfield network of N nodes can achieve 100% correct retrieval on P patterns if: P < N/(4*ln(N)) N Max P 101 1005 1000 36 10000271 10 11 10 9 In general, as more patterns are added to a network, the avg correlations will be less likely to match the correlations in any particular pattern. Hence, the likelihood of retrieval error will increase. => The key to perfect recall is selective ignorance!!

22. Stochastic Hopfield Networks Node state is stochastically determined by sum of inputs: Node fires with probability: For these networks, effective retrieval is obtained when P < 0.138N, which is an improvement over standard Hopfield nets. Boltzmann Machines: Similar to Hopfield nets but with hidden layers. State changes occur either: a. Deterministically when b. Stochastically with probability = Where t is a decreasing temperature variable and is the expected change in energy if the change is made. The non-determinism allows the system to ”jiggle” out of local minima.

23. Hopfield Nets in the Brain?? The cerebral cortex is full of recurrent connections, and there is solid evidence for Hebbian synapse modification there. Hence, the cerebrum is believed to function as an associative memory. Flip-flop figures indicate distributed hopfield-type coding, since we cannot hold both perceptions simultaneously (binding problem)

24. The Necker Cube Excitatory Inhibitory Closer(A,B)Closer(H,G)Closer(C,D)Closer(G,H) Convex(A)Showing(G)Convex(G)Hidden(G) B G C H D F E A Which face is closer to the viewer? BCGF or ADHE? Only one side of the (neural) network can be active at a time. Steven Pinker (1997) “How the Mind Works”, pg. 107.

25. Things to Remember Auto-Associative -vs- Hetero-associative –Wide variety of net topologies –All use Hebbian Learning => weights ~ avg correlations One-shot -vs- Iterative Retrieval –Iterative gives much better error correction. Asynchronous -vs- Synchronous state updates –Synchronous updates can easily lead to oscillation –Asynchronous updates can quickly find a local optima (attractor) Update order can determine attractor that is reached. Pattern Retrieval = Search in node-state space. –Spurious patterns are hard to avoid, since many are attractors also. –Stochasticity helps jiggle out of local minima. –Memory load increase => recall error increase. Associative -vs- Feed-Forward Nets –Assoc: Many - 1 mapping Feed-Forward: many-many mapping –Backprop is resource-intensive, while Hopfield iterative update is O(n) –Gradient-Descent on an Error -vs- Energy Landscape: Backprop => arc-weight space Hopfield => node-state space