1. Statistical and dynamical properties of large cortical network models: insights into semantic memory and language
Emilio Kropff
Thesis presentation
September 19, 2007

2. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Cerebral cortex – Braitenberg & Schüz, 1991
# of neurons >> # of input fibers
Modifiable synapses
No prefered direction in the connections
Two-level associative memory with formation of cell assemblies
Mostly excitatory synapses
Great convergence & divergence
Connections are very weak

3. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Auto-associative memories
- Pattern #2 active
- Pattern #3 active
No activity
- Pattern #1 active
Hebbian Learning !!

4. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Testing the memory
- Pattern #2 active

5. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Testing the memory
Network damage

6. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Two-level associative memory with formation of cell assemblies
Anatomical studies – Braitenberg & Schüz
Embodied theories of semantic memory
Feature representation

7. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
The model: the trick
S: number of alternative states of a unit
S: number of alternative features of a unit
a (global sparseness): average number of features describing a concept
a (global sparseness): average number of units that are active in a global memory …
state 1
state 0
state 3
state S
state 2

8. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Testing Kanter’s Potts network (Kanter,1988)
The patterns ξ are constructed randomly and stored in the network by modifying J.
The state of the network is set to some initial value (e.g: random or some stored memory if we want to test its stability).
A unit i is picked randomly and the fields hik are calculated.
The S states of unit i are updated following:

9. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Reviewing and extending the results of Kanter, 1988
Kanter’s result for low S is
We find high S behaviour is

10. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Adding a zero state and sparseness a
Sparse Hopfield (Buhmann, 1989, Tsodyks, 1988)
Potts without sparseness (Kanter, 1988)
Sparse Potts (Kropff & Treves, 2005) αc
~ 1/a
~ 0.14 S(S-1)
S2/a

11. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Highly diluted approximation
Two units speak to each other with probability cM/N
Two states speak to each other with probability e

12. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA

13. The Kanter result for Potts networks has a logarithmic correction for high S.
If well defined, a sparse Potts network reaches an optimal storage capacity
In highly diluted networks this result applies, with  = p/(cM e) .
These results are in line with the conjecture of a limit in the amount of information per synapse that a network can store.

14. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Dynamics of the network
retrieval
+ adaptation
+ correlation
latching!
time

15. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA

16. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
In addition, the transition matrix is not symmetric

17. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Dynamics: latching of 2 patterns
time

18. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Dynamics: latching of 2 patterns
Units active in both patterns in different states
‘alternative features’
Units active in one of the two patterns
Unitsactive in both in the same state
Weakly: units active in both patternsin different states
‘pathological case’
c: shared units
Units active in one of the two patterns
Unitactive in both in the same state
‘shared features’
d: shared features

19. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Does latching present a natural rudimentary grammar?

20. Correlation seems to be at least one of the main properties determining latching.
However, the transition matrix is not symmetric, which means that there are other important factors.
The equilibrium between global inhibition and local self excitation can control the complexity of symbolic chains.
Three types of latching transition exist, each in a restricted region of parameters. Do they organize in time?

21. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Category specific deficits
Patients were found with a significant impairment in their knowledge about living things (animals + foodstuffs) as opposed to manmade artifacts (Warrington & Shallice, 1984).
Impairment for nonliving has also been reported → double dissociation. Current ratio: 23% vs 77% (Capitani, 03)
Theoretical accounts
The selective impairments respond to differences in the networks representing different categories: sensory/functional theory (Warrington & Shallice, 84), domain-specific hypothesis (Caramazza & Shelton, 98).
The network sustaining semantic memory is quite homogeneous but different categories have different typical correlation properties (McRee et al, 97; Tyler & Moss, 01; Sartori & Lombardi, 04).

22. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Hopfield memories
If patterns are randomly correlated (Tsodyks,88),
However, if patterns have a non-trivial structure of correlations, the storage capacity colapses.
Solution #1: Orthogonalize the patterns before feeding the network. (i.e: Dentate Gyrus in Hippocampus)
In semantic memory correlation between stored patterns seems to play a major role.

23. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Solution #2 ??
We assume that pattern 1 is being retrieved
We split hi into the contribution of pattern 1 (signal) and the rest (noise)
We minimize the noise

24. Jij= Σμ (ξiμ – a ).(ξjμ – a )
Classical result: hebbian learning supports uncorrelated memories
Jij= Σμ (ξiμ – a ).(ξjμ – a )
Jij= Σμ (ξiμ - ai).(ξjμ - aj)
Classical result: catastrophe associated to correlated memories
popularity: ak= 1/p Σμ ξkμ
New result: a modification that supports correlated memories
New result: the performance is the same with uncorrelated memories

25. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with finite α, C ≈ ln(N)
GAUSSIAN noise (If there is independence between neurons i and j).

26. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with finite α, C ≈ ln(N)
GAUSSIAN noise (If there is independence between neurons i and j).

27. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with finite α, C ≈ ln(N)
F(x)
...
(uncorrelated patterns)
If F(x) decays fast enough

28. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with finite α, C ≈ ln(N)
F(x)
If F(x) decays exponentially
If F(x) decays fast enough
If F(x) decays as a power law

29. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Storing memories
Damaging the network

30. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Storage capacity ~ fixed p
Entropy Sf= Σi ai (1-ai)
summed over active neurons in the pattern
Connectivity
C (# of afferent connections per neuron)
Performance 1
0.2
0.4 10 20 30 40 50 60 70
Popularity ai
Number of neurons
Cc1
Cc2 Cc

31. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Category specific effects
McRae feature norms
541 concepts described in terms of 2526 features
i=1 if feature i is included in the description of concept  and i =0 otherwise
living
non living
Probability Distribution
Entropy Sf of objects

32. When memories are correlated, they have variable degrees of ressistance to damage.
The robustness of a memory is inverse to how informative it is (Sf).
In addition, popular neurons affect negatively the general performance (decay of F(x)).
These results show how the current trend in category specific deficits (‘living’ weaker than ‘non living’) could emerge even in a purely homogeneous network.

33. A singel cortical network with Potts units including addaptation and storing correlated patterns of activity in its long range synapses, presents all the properties studied in this thesis.
Thank you!

34. McRae’s feature norms
In the semantic memory literature, auto-associative networks are often presented as weak models. Why?
popularity
# of features
To convince psychologists one must show an auto-associative memory that is able to store feature norms.

35. McRae’s feature norms Performance of the network
theoretical prediction
simulations
Size of the subgroup of patterns

36. McRae’s feature norms a – average sparseness If – average information
Number of patterns p in the subgroup

37. McRae’s feature norms Performance of the network
theoretical prediction
simulations
Size of the subgroup of patterns

38. McRae’s feature norms Why the real network performs poorly?
Independence between features is not valid (e.g: beak and wings). Is this effect strong enough? In case it is, there would be a storage capacity colapse.
The system works but the approximation of diluted connectivity is not good.

39. McRae’s feature norms: the full solution
 +  2+ 

40. McRae’s feature norms: the full solution
Performance of the network
highly diluted
full solution
simulations
Size of the subgroup of patterns

41. a b McRae’s feature norms: strategies to store more patterns
1- add unpopular neurons
2- eliminate popular neurons
Total number of neurons
% of patterns retrieved
3000
4000
5000
6000 20 40 60 80
100 a b
2490
2500
2510
2520

42. McRae’s feature norms: strategies to store more patterns
4- popularity deppendent connectivity
3- recombination
neurons i and j have high popularity: their coincidence will be less popular. If applied massively, this principle could change the whole distribution.

43. The (episodic) memory pyramid
popularity calculation
final storage
orthogonalization
Hippo-campus
Coding
Consolidation
Entorhinal cortex
Perirhinal and parahippocampal cortex
Unimodal and polymodal association areas
Primary cortex: sensory motor areas

44. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with α ≈ 0, C ≈ ln(N)

45. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with α ≈ 0, C ≈ ln(N)
ac
ac

46. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Propeties with α ≈ 0, C ≈ ln(N)
If you want to be an attractor, you should pick at least some unpopular units.
Lowering U can make any pattern retrievable -> ATTENTION

47. Potts Networks – Latching – Correlated patterns
Thesis presentation. September 19th, 2007
Potts Networks – Latching – Correlated patterns
Emilio Kropff, LIMBO-CNS-SISSA
Dynamics: latching of 2 patterns
Units active in both patterns in different states
‘alternative features’
Units active in one of the two patterns
Unitsactive in both in the same state
Weakly: units active in both patternsin different states
‘pathological case’
c: alternative features
Units active in one of the two patterns
Unitactive in both in the same state
‘shared features’
d: shared features