2. Robert M. French LEAD-CNRS UMR 5022 Dijon, France Why you will remember this talk and a Neural Network would not... Or The Problem (and a Solution to) Catastrophic Interference in Neural Networks

3. Organization of this talk What is catastrophic forgetting and why does it occur? A dual-network technique using pseudopattern information transfer for overcoming it for multiple pattern-learning. What is “pseudopattern information transfer”? What are “reverberated” pseudopatterns? What about sequence-learning? What about learning multiple sequences? Applications, theoretical questions and future directions

4. The problem NEURAL NETWORKS FORGET CATASTROPHICALLY Learning new information can may completely destroy previously learned information. This makes sequential learning — i.e., learning one thing after another, the way humans learn — impossible. Can sequential learning be modeled so that: i) New information does not interfere catastrophically with already-learned information? and ii) Without keeping previously learned items around?

5. Barnes-Underwood (1959) forgetting paradigm Subjects learn List A-B (non-word/word pairs): pled – table splog –book bim – car milt – bag etc. until they have all pairs learned. Then they learn List A-C (same non-words, different real word) pled –rock splog –cow bim – square milt – clock etc. different target items here first set of target items same items

6. How humans do on this task

7. How artificial neural networks (backprop) do on this task

8. WHY does this occur? Answer: Overlap of internal representations “Catastrophic forgetting is a direct consequence of the overlap of distributed representations and can be reduced by reducing this overlap.” (French, 1991)

9. How can we reduce this overlap of internal representations? Also, this seems to be the solution discovered by the brain. Hippocampus – Neocortex. (McClelland, McNaughton, & O’Reilly, 1995) Two separate networks in continual interaction: one for long-term storage, one for immediate processing of new patterns. (French, 1997; etc.) Answer:

10. Implementation We have implemented a “dual-network” system using coupled networks that completely solves this problem (French, 1997; Ans & Rousset, 1997, 2000; Ans, Rousset, French, & Musca, 2002, in press). These two separate networks exchange information by means of “reverberated pseudopatterns.”

11. Pseudopatterns? Reverberated pseudopatterns? What?

12. Pseudopatterns f(x) Assume a network-in-a-box learns a series of patterns produced by a function f(x). These original patterns are no longer available. How can you approximate f(x)?

13. 1 0 0 1 1Random Input

14. 1 0 0 1 1Random Input 1 1 0Associated output

15. 1 0 0 1 1Random Input 1 1 0Associated output This creates a pseudopattern:  1 : 1 0 0 1 1  1 1 0

16. A large enough collection of these pseudopatterns:  1 : 1 0 0 1 1  1 1 0  2 : 1 1 0 0 0  0 1 1  3 : 0 0 0 1 0  1 0 0  4 : 0 1 1 1 1  0 0 0 Etc will approximate the originally learned function.

17. Transferring information from Net 1 to Net 2 with pseudopatterns 1 0 0 1 1 1 1 0 1 0 0 1 1 1 1 0 target input Random input Associated output Net 1 Net 2

18. Learning new information in Net 1 with pseudopatterns from Net 2 0 0 1 1 1 0 1 0 1 1 1 target 1 1 0 1 1 random input New input Target Net 1 Net 2 + 1 1 0 1 1 1 1 1 New pattern to learn: 00111  010 Net 1

19. 0 0 1 1 1 0 1 0 0 0 1 target 1 1 0 0 0 random input New input Net 1 Net 2 + 1 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1 + + Etc. Target

20. This is how information is continually transferred between the two networks by means of pseudopatterns.

21. Sequential Learning using the dual-network approach Sequential learning of 20 patterns – one after the other (French, 1997)

22. On to reverberated pseudopatterns... Even though the simple dual-network system (i.e., new learning in one network; long-term storage in the other) using simple pseudopatterns does eliminate catastrophic interference, we can do better using “reverberated” pseudopatterns.

23. Building a Network that uses “reverberated” pseudopatterns. Input layer Hidden layer Output layer Start with a standard backpropagation network

24. Input layer Hidden layer Output layer Add an autoassociator

25. A new pattern to be learned, P: Input  Target, will be learned as shown below. Input Target Input

26. We start with a random input î 0, feed it through the network and collect the output on the autoassociative side of the network.. This output is fed back into the input layer (“reverberated”) and, again, the output on the autoassociative side is collected. This is done R times.

31. After R reverberations, we associate the reverberated input and the “target” output. This forms the reverberated pseudopattern:

32. This dual-network approach using reverberated pseudopattern information transfer between the two networks effectively overcomes catastrophic interference in multiple-pattern learning Net 2 Storage network Net 1 New-learning network

33. But what about multiple-sequence learning? Elman networks are designed to learn sequences of patterns. But they forget catastrophically when they attempt to learn multiple sequences. Can we generalize the dual-network, reverberated pseudopattern technique to dual Elman networks and eliminate catastrophic interference in multiple-sequence learning? Yes

34. The Problem of Multiple-Sequence Learning Real cognition requires the ability to learn sequences of patterns (or actions). (This is why SRN’s – Elman Networks – were originally developed.) But learning sequences really means being able to learn multiple sequences without the most recently learned ones erasing the previously learned ones. Catastrophic interference is a serious problem for the sequential learning of individual patterns. It is far worse when multiple sequences of patterns have to be learned consecutively.

35. Elman networks (a.k.a. Simple Recurrent Networks) Copy hidden unit activations from previous time-step Standard input S(t) Context H(t-1) Hidden H(t) S(t+1) Learning a sequence S(1), S(2), …, S(n).

36. A “Reverberated Simple Recurrent Network” (RSRN): an Elman network with an autoassociative part

37. RSRN technique for sequentially learning two sequences A(t) and B(t). Net 1 learns A(t) completely. Reverberated pseudopattern transfer to Net 2. Net 1 makes one weight-change pass through B(t). Net 2 generates a single “static” reverberated pseudopattern Net 1 does one learning epoch on this pseudopattern from Net 2. Continue until Net 1 has learned B(t). Test how well Net 1 has retained A(t).

38. Two sequences to be learned: A(0), A(1), … A(10) and B(0), B(1), … B(10) Net 1 Net 2 Net 1 learns (completely) sequence A(0), A(1), …, A(10)

39. Net 1 Net 2 010110100110010 1110010011010 : 010110100110010 1110010011010 010110100110010 Input 1110010011010 Teacher Transferring the learning to Net 2

40. Net 1 Net 2 feedforward 1110010011010 Teacher 010110100110010 Input Transferring the learning to Net 2

41. Net 1 Net 2 Backprop weight change Repeat for 10,000 pseudopatterns produced by Net 1. 010110100110010 Input 1110010011010 Teacher Transferring the learning to Net 2

42. Learning B(0), B(1), … B(10) by NET 1 Net 1 Net 2 1. Net 1 does ONE learning epoch on sequence B(0), B(1), …, B(10) 2. Net 2 generates ONE pseudopattern:  NET 2 3. Net 1 does one FF-BP pass on  NET 2

43. Net 1 Net 2 1. Net 1 does ONE learning epoch on sequence B(0), B(1), …, B(10) 2. Net 2 generates ONE pseudopattern:  NET 2 Learning B(0), B(1), … B(10) by NET 1 3. Net 1 does one FF-BP pass on  NET 2 Continue until Net 1 has learned B(0), B(1), …, B(10)

44. Sequences chosen Twenty-two distinct random binary vectors of length 100 are created. Half of these vectors are used to produce the first ordered sequence of items, A, denoted by A(0), A(1), …, A(10). The remaining 11 vectors are used to create a second sequence of items, B, denoted by B(0), B(1), …, B(10). In order to introduce a degree of ambiguity into each sequence (so that a simple BP network would not be able to learn them), we modify each sequence so that A(5) = A(8) and B(1) = B(5).

45. Test method First, sequence A is completely learned by the network. Then sequence B is learned. During the course of learning, we monitor at regular intervals how much of sequence A has been forgotten by the network.

46. Normal Elman networks: Catastrophic forgetting (height of bars equals how much forgetting has occurred). By 450 epochs sequence B has been completely learned. However, the SRN has, for all intents and purposes, completely forgotten the previously learned sequence A

47. As Sequence B is being learned, recall performance for Sequence A in the Dual-RSRN model By 400 epochs, the second sequence B has been completely learned. The previously learned sequence A shows virtually no forgetting. Forgetting – not just catastrophic forgetting – of the previously learned sequence A has been completely overcome.

48. Normal Elman Network: Massive forgetting % Error on Sequence A Dual RSRN: No forgetting of Sequence A

49. Cognitive/Neurobiological plausibility? The brain, somehow, does not forget catastrophically. Separating new learning from previously learned information seems necessary. McClelland, McNaughton, O’Reilly (1995) have suggested the hippocampal-neocortical separation may be Nature’s way of solving this problem. Pseudopattern transfer is not so far-fetched if we accept results that claim that neo-cortical memory consolidation, is due, at least in part, to REM sleep.

50. Prediction of the model : "Recall rebound" 2-network RSRN 0 20 40 60 80 100 0151020 Number of presentations of the new sequence Old sequences (% correct) Empirical data: Recall rebound confirmed Humans 0 20 40 60 80 100 0151020 Old sequences (% correct) Number of presentations of the new sequence

51. Learning a new language: initial drop in performance for the first language, followed by regaining of initial levels of performance. Learning a new piece of music Learning new motor activities, etc. Examples of the "recall rebound" in the real world

52. In case you missed it... What is so interesting about the RSRN procedure is that by means of a number of “static” input-output patterns (pseudopatterns), we can transfer sequential information into another network. In other words, a Sequence: A-B-C-D-B-E-C-F-G of actions, words, patterns, etc. can be transferred by means of a set of I/O patterns. OK, cute. But why is this so interesting?

53. Attention, all roboticists! Consider a group of robots exploring their world R4R5R3R1R2 R1 is learning the sequence of actions to open a door. R2 “ “ “ “ “ “ “ open a window. R3 “ “ “ “ “ “ “ pick up a block.

54. But R1 is continually broadcasting pseudopatterns that will be picked up by the other robots and interleaved with the sequence they are learning. Thus, all robots within transmission range of R1 will learn how to open a door, without ever having actually opened a door. Similarly, all robots (including R1) learn what the other robots have learned by picking up their pseudopatterns. Efficient Robot communication

55. Assume that there is a long sequence where each letter represents an action: A-B-C-B-A-S-T-B-S-Q-S-A-D-B Efficient Parallel Learning R1 learns this and transmits to R2 learns this and transmits to R3 learns this and transmits to R4 R4 will have then learned the entire sequence!

56. Other research issues Is this the optimal way to generate pseudopatterns? Consider the following function learned by an ANN:

57. III There are really two parts to the function.

58. III too many pseudopatterns here not enough pseudo- patterns here Uniform distribution of pseudopatterns

59. III A much better distribution that uses feedback (Holland, 1975, 1992)

60. Noise, dynamical agents and psychological theory The apparent ubiquity of 1/f noise … may inform and guide development of psychological theory (e.g., Gilden, 2001; Van Orden et al., 2003; Wagenmakers, Farrell, & Ratcliff, in press; Ward, 2002). One claim made on the basis of such findings is that 1/f noise reflects the emergent global dynamics of locally interacting agents. (Farrell, Wagenmakers, Ratcliff, submitted)

61. Conclusions For engineers: The dual-network architecture using pseudopattern information transfer solves the problem of catastrophic forgetting in artificial neural networks, both for multiple-pattern learning and for multiple-sequence learning. For cognitive psychologists and neuroscientists: Rather than being a problem, there is a good chance that evolution has found a way to allow the brain has also turned noise to its advantage as an efficient mechanism of information transfer.