General mechanisms of Neocortical memory Jeff Hawkins Director Redwood Neuroscience Institute June 12, 2003 MIT

Outline Top down analysis: nature of problem and solution representation time and prediction Bottom up example: auditory memory task - deduce necessary algorithms - unique map to anatomy

“I conclude that cytoarchitectural difference between areas of neocortex reflect differences in their patterns of extrinsic connections. The traditional or usual ‘functions’ of different areas also reflect these differences in extrinsic connections. They provide no evidence whatsoever for differences in intrinsic structure or function..” “Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.” Vernon Mountcastle, 1978

motortouchauditionvision spatially specific spatially invariant temporally specific (fast) temporally invariant

Neocortical connectivity

motortouchauditionvision spatially specific spatially invariant temporally specific (fast) temporally invariant

motortouchauditionvision spatially specific spatially invariant temporally specific (fast) temporally invariant

motortouchauditionvision spatially specific spatially invariant temporally Specific (fast) temporally invariant Prediction (spatially and temporally specific) MacKay, Mumford, Softky, Rao & Ballard

motortouchauditionvision spatially specific spatially invariant temporally fast temporally invariant Prediction (spatially and temporally specific) Q1. Why make predictions? Q2. How do we make predictions? Q3. How do we form invariant representations?

Q1. Why make predictions Non-mammalian brain Sophisticated senses Complex behavior

Posterior Neocortex: sensory prediction Predictions allow brain to react prior to events, to “see” into the future. Sophisticated senses Complex behavior Mammalian posterior neocortex

Anterior Neocortex: motor sequences Sophisticated senses Complex behavior Mammalian posterior neocortex Human anterior neocortex

Q2. How do we make predictions? - Store sequence of patterns: allows prediction of future events - Invariant representations cannot make specific predictions invariant representations specific afferents ……… time

Q2. How do you make predictions? - Store sequence of patterns: allows prediction of future events - Invariant representations cannot make specific predictions - invariant prediction + input[t-1] = specific prediction[t] invariant representations specific afferents ……… + time

Q3. How do we form invariant representations? Spatially invariant representations require - convergence of features that constitute object - divergence to unite objects that although different represent the same thing (x 1 ⋂ x 2 ⋂ x 3 … ) ⋃ (x 4 ⋂ x 5 ⋂ x 6 … ) ⋃ (x 7 ⋂ x 8 ⋂ x 9 … ) …

Top down summary Every cortical region: - Forms representations by convergence of features - Forms invariant representations by divergence - Stores and recalls sequences of invariant representations sequence memory - Recalls pattern sequences auto-associatively - Combines recalled patterns with input to: make predictions of sensory afferents drive motor efferents

Top down summary Every cortical region: - Forms representations by convergence of features L4, Thalamus - Forms invariant representations by divergence L2,3 horiz - Stores and recalls sequences of invariant representations L1,2,3 sequence memory - Recalls pattern sequences auto-associatively - Combines recalled patterns with input to: L5,6 make predictions of sensory afferents drive motor efferents

Bottom up example: Auditory memory (melodies) - Representations are invariant to pitch recognized and recalled in any pitch - Stored as sequences of associated patterns have repeated elements (ggge- fffd ggge- aaag) each note has a stored duration - Prediction: we “hear” notes prior to occurrence - Hierarchical representation, e.g. AABA structure (temporal invariance/reduction)

A1 L freq H Thalamus

C D E F G A B C 1 D 1 E 1 A1 A2 C-C’ D-D’ E-E’ F-F’ G-G’A-A’ B-B’ octave (x 1 ⋂ x 2 ⋂ x 3 … ) ⋃ (x 4 ⋂ x 5 ⋂ x 6 … ) ⋃ (x 7 ⋂ x 8 ⋂ x 9 … ) … ( C ⋂ C’ ) ⋃ ( D ⋂ D’ ) ⋃ ( E ⋂ E’ ) … frequency intervals Pitch invariance = interval representation

A2 L freq H A1 L freq H Thalamus

A2 L freq H A1 L freq H L H Thalamus Intersecting inputs in layer 4 define all possible intervals

A2 L freq H A1 L freq H L H Thalamus Iso-interval bands up down

A2 L freq H A1 L freq H L H Thalamus Freq invariant interval bands up down L2,3 L4 - Intersecting inputs to L4 - Spread of activation in L2,3

How do we store the sequence of interval activations? How do we represent unique intervals in unique songs? GGGE- FFFD GGGE- AAAG How do we store and recall the precise time duration of each unique interval?

L2,3 L1 L4 L5 L6 Layer 2,3 cells Dense and small High local mutual excitation High local mutual inhibition Long distance excitatory coll. Dendrites in L1 Axon synapses in L5

L2,3 L1 L4 L5 L6 Layer 2,3 is sparsely active Mutual excitation drives all Strong inhibition prevents most cells from firing Layer1 plays role in deciding who is active

L2,3 L1 L4 L5 L6 Layer 1 is context 1. Context from higher areas 2. Local context from L2,3 3. Input from matrix thalamus (time)

L2,3 L1 L4 L5 L6 Layer 1 context Layer 2,3 unique representations of freq invariant intervals There is a unique sparse L2,3 activation pattern for each instance of this interval ever learned. Each unique pattern represents a particular interval in a particular melody.

Layer 4 Freq specific intervals Converging inputs form object representations L freq H L H Layer 2,3 Freq invariant intervals Horizontal connections join objects to form spatially invariant representations Layer 1 State: time & location L1 axons link representations in sequence. Unique representations link to unique representations Song is represented as a sequence of freq invariant interval bands. Each invariant interval has a unique representation and is associatively linked to its predecessor.

Representing “class” and “individuality” Activation area defines object class Unique activation pattern defines individual object

How do we store and recall the precise time duration of each unique interval? - Actual duration vs. relative duration (actual) - Duration must be stored in-situ with interval Proposal … - Matrix thalamic nuclei emits a clock pattern to L1 - Part of L1 changes on each clock tick - L5 cell resets clock on L4 transition or L1 match

L2,3 L1 L4 L5 L6 New input arrives at L4, causes L5 cell to burst, inhibition shuts down L4 L5 burst teaches L5 cell to fire when exact pattern in L1 is seen in future L5 burst also sets matrix thalamic nuclei to a deterministic state (resets clock) causing interval state transition L5 cells encode duration of a particular state (note in song): when the elapsed time of a particular state occurs, they burst fire Matrix Thalamus

How do you predict next note in proper key? invariant prediction + input[t-1] = specific prediction[t] invariant representations specific afferents ……… + time

L2,3 L1 L4 L6a L6b A1(t-1)Th(t) freq Pattern from A1

L2,3 L1 L4 L6a L6b A1(t-1)Th(t) freq Th(t) freq Pattern from A1 Simple interval

L2,3 L1 L4 L6a L6b A1(t-1)Th(t) freq Th(t) freq Pattern from A1 Simple interval Invariant unique interval

L2,3 L1 L4 L6a L6b A1(t-1)Th(t) freq Th(t) freq Pattern from A1 Simple interval Invariant unique interval Associative spread

L2,3 L1 L4 L6a L6b A1(t) freq Pattern from A1(t) Predicted next interval

L2,3 L1 L4 L6a L6b A1(t) freq A1(t) + predicted interval Predicted next interval

L2,3 L1 L4 L6a L6b freq A1(t) + predicted interval Predicted next interval Next predicted note back to Thalamus

L2,3 L1 L4 L6a L6b freq A1(t) + predicted interval Predicted next interval Horizontal projections from stored previous rich pattern to apical dendrites of predicted pattern copies rich attributes

Hierarchical representation words / melodies phrases / songs sentences

Hierarchical representation words / melodies phrases / songs sentences Problem The number of state transitions must decrease as you ascend the hierarchy. However L2,3 projects to upper areas and it changes on every event.

Hierarchical representation Solution Some cells in L2,3 learn to be stable over repeated patterns.

Hierarchical representation Solution Some cells in L2,3 learn to be stable over repeated patterns. Therefore we should see L2,3 cells that stay active over longer periods of time. Only these cells should project to next higher cortical area.

How generic is this model? Performs a non-trivial memory processing function - invariant, rich predicting, branching, hierarchical, sequence memory Aligns well with top down constraints Accounts for much of known cortical anatomy - involves all layers, excitatory and inhibitory spread - how could other areas of cortex be fundamentally different? Other cortical areas are likely variations on this theme Other principles are likely in use as well

A2 as I have drawn it A2 as it might appear - limited to octave intervals - appearance of tonotopy Redrawing A2

Compares input from two ears - inter-aural delay accentuated subcortically - predicts location of sounds in body space Possible interpretation of A1 Narrowly tuned Broader tuned, sweep low freq high

Summary 1)Converging L4 inputs define objects 2) Horizontal connections in L2,3 create spatially invariant representations 2)Sparse activation in Layers 2,3 encodes unique instances of invariant representations 3) L1 mediates memory of sequences 4)L5 thalamo-cortical loops encode duration of events 5) Sustained activity in some L2,3 cells establishes basis for temporal invariance 6) L6 cells make specific predictions from L2,3 and afferents

Summary 1)Converging L4 inputs define objects 2) Horizontal connections in L2,3 create spatially invariant representations 2)Sparse activation in Layers 2,3 encodes unique instances of invariant representations 3) L1 mediates memory of sequences 4)L5 thalamo-cortical loops encode duration of events 5) Sustained activity in some L2,3 cells establishes basis for temporal invariance 6) L6 cells make specific predictions from L2,3 and afferents Testable - buildable - a start

Thank - - -

“It is not that most neurobiologists do not have some general concept of what is going on. The trouble is that the concept is not precisely formulated. Touch it and it crumbles. What is conspicuously lacking is a broad framework of ideas within which to interpret these different approaches.” Francis Crick 1979

There is “no evidence whatsoever for differences in intrinsic structure or function. This suggests that the necortex is everywhere functionally much more uniform than hitherto supposed and that its avalanching enlargement in mammals and particularly in primates has been accomplished by replication of a basic neural module without the appearance of wholly new neuron types or qualitatively different modes of intrinsic organization.” “Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance.” Vernon Mountcastle, 1978 All cortical regions