1. Abstract Neuron { o u t p u t y i n p u t i i2 in i1 . . . w0 i0=1 w2
1 if net > 0
0 otherwise { w0
i0=1 w2 wn w1

2. Link to Vision: The Necker Cube

4. Constrained Best Fit in Nature
inanimate animate
physics
lowest energy state
chemistry
molecular
minima
biology
fitness, MEU
Neuroeconomics
vision
threats,
friends
language
errors,
NTL

5. Computing other relations
The 2/3 node is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active
Such a node is also called a triangle node and will be useful for lots of representations.

6. Triangle Nodes: Encoding relational information with abstract neurons
The triangle node (aka 2/3 node) is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active
Such a node will be useful for lots of representations.

7. Triangle nodes and McCullough-Pitts Neurons
Relation (A)
Object (B)
Value (C) A B C

8. Basic Ideas Parallel activation streams.
Top down and bottom up activation combine to determine the best matching structure.
Triangle nodes bind features of objects to values
Mutual inhibition and competition between structures
Mental connections are active neural connections

9. 5 levels of Neural Theory of Language
Pyscholinguistic experiments
Spatial Relation
Motor Control
Metaphor
Grammar
Cognition and Language
Computation
Structured Connectionism
abstraction
Neural Net
SHRUTI
Computational Neurobiology
Triangle Nodes
Biology
Neural Development
Quiz
Midterm
Finals

10. Behavioral Experiments
Identity – Mental activity is Structured Neural Activity
Spreading Activation — Psychological model/theory behind priming and interference experiments
Simulation — Necessary for meaningfulness and contextual inference
Parameters — Govern simulation, strict inference, link to language

11. Bottom-up vs. Top-down Processes
Bottom-up: When processing is driven by the stimulus
Top-down: When knowledge and context are used to assist and drive processing
Interaction: The stimulus is the basis of processing but almost immediately top-down processes are initiated

12. Stroop Effect
Interference between form and meaning

13. Name the words Book Car Table Box Trash Man Bed
Corn Sit Paper Coin Glass House Jar
Key Rug Cat Doll Letter Baby Tomato
Check Phone Soda Dish Lamp Woman

14. Name the print color of the words
Blue Green Red Yellow Orange Black Red
Purple Green Red Blue Yellow Black Red
Green White Blue Yellow Red Black Blue
White Red Yellow Green Black Purple

15. Body-Specificity Hypothesis
If concepts and word meanings are constituted, in part, by mental simulations our own perceptions and actions…
…then their neurocognitive representations should differ for people with different kinds of bodies, who perceive and act upon the environment in systematically different ways.
(Casasanto, in review, link on course page)

16. Testing body-specificity
pinch
chew
In between the red and blue boxes is a white box filled with hundreds of clear glass marbles.
Ss would see one word at a time in the center of the screen, in red or blue font, and move one marble into the box that matched the color of the font.
Manual Action
Non-Manual Action

17. Testing body-specificity
For one block of trials they’d use only their left hand, and the other block only their right hand -- block order was counterbalaced across Ss.
Left handed movements
for one block of trials
Right handed movements
for the other block of trials

18. Design 96 Words (48 manual verbs, 48 non-manual verbs)
2 blocks (LH movements, RH movements)
2 groups of Ss (Left handers, Right handers)
Dependent measures: RT & Surprise recognition (Old/New)
So, here’s the full design: There were 96 words, all of which were action verbs -- unbeknownst to the subject, half of them named Manual actions that we normally perform with one hand -- our dominant hand -- and the other half named Non-Manual actions.

19. Move marbles by word color
caress
jab
fling
grip
pound
tap
yank
erase
dial
sigh
cough
growl
watch
crawl
peek
tumble
dance
say
Importantly, the only instruction was to move marbles according to the color of the word -- word meaning was completely irrelevant to the task. But we expected people would read the word they saw in front of them, and would automatically activate they meaning, whether they wanted to or not, and we make the following predictions about what that meaning is made of:
Manual Action
Non-Manual Action

20. Predictions
If action word meanings are constituted, in part, by mental simulation of perceptuo-motor experiences, then:
Both online and offline effects of congruity should be found between manual motor actions and the meanings of manual action verbs (but not non-manual action verbs).
Right and left handed participants should show opposite effects of using their right and left hands to move marbles during incidental encoding of manual action verbs.
(Casasanto, in review)

21. Non-Manual Action Verb
Reaction Time Results
Non-Manual Action Verb ns
***
***
Marble movement time in ms
Here’s what we found:
Manual Action Verb
(Casasanto, in review)

22. Handedness-ness predicts congruity effect
ABS Laterality Quotient
Congruity effect in ms
Furthermore, We gave Ss the Edinburgh handedness inventory…

23. Recognition Memory Resultsns
Non-Manual Action Verb
0.80
LeftHand
RightHand
0.75
***
***
0.70
Proportion correct recognition
0.65
0.60
0.55
0.50
LEFTIES
RIGHTIES
Manual Action Verb
(Casasanto, in review)

24. Procedure for experiment that demonstrates the word-superiority effect
Procedure for experiment that demonstrates the word-superiority effect. First the word is presented, then the mask XXXX’s, then the letters.

25. Word-Superiority Effect Reicher (1969)
Which condition resulted in faster & more accurate recognition of the letter?
The word condition
Letters are recognized faster when they are part of a word then when they are alone
This rejects the completely bottom-up feature model
Also a challenge for serial processing

26. Connectionist Model McClelland & Rumelhart (1981)
Knowledge is distributed and processing occurs in parallel, with both bottom-up and top-down influences
This model can explain the Word-Superiority Effect because it can account for context effects

27. Connectionist Model of Word Recognition

28. Basic Ideas Parallel activation streams.
Top down and bottom up activation combine to determine the best matching structure.
Triangle nodes bind features of objects to values
Mutual inhibition and competition between structures
Mental connections are active neural connections

29. Interaction in language processing: Pragmatic constraints on lexical access
Jim Magnuson
Columbia University

30. Information integration
A central issue in psycholinguistics and cognitive science:
When/how are such sources integrated?
Two views
Interaction
Use information as soon as it is available
Free flow between levels of representation
Modularity
Protect and optimize levels by encapsulation
Staged serial processing
Reanalyze / appeal to top-down information only when needed

32. Reaction Times in Milliseconds after: “They all rose”
0 delay ms. delay
flower
685
659
stood
677
623
desk
711
652

33. Example: Modularity and word recognition
Tanenhaus et al. (1979) [also Swinney, 1979]
Given a homophone like rose, and a context biased towards one sense, when is context integrated?
Spoken sentence primes ending in homophones:
They all rose vs. They bought a rose
Secondary task: name a displayed orthographic word
Probe at offset of ambiguous word: priming for both “stood” and “flower”
200 ms later: only priming for appropriate sense
Suggests encapsulation followed by rapid integration
But the constraint here is weak -- overestimates modularity?
How could we examine strong constraints in natural contexts?

35. “They all rose” triangle nodes:
when two of the abstract neurons fire, the third also fires
model of spreading activation

36. Allopenna, Magnuson & Tanenhaus (1998)
Eye
Eye camera
tracking
computer
Scene camera
‘Pick up the beaker’

37. Do rhymes compete?
Cohort (Marlsen-Wilson): onset similarity is primary because of the incremental nature of speech
(serial/staged; Shortlist/Merge)
Cat activates cap, cast, cattle, camera, etc.
Rhymes won’t compete
NAM (Neighborhood Activation Model; Luce): global similarity is primary
Cat activates bat, rat, cot, cast, etc.
Rhymes among set of strong competitors
TRACE (McClelland & Elman): global similarity constrained by incremental nature of speech
Cohorts and rhymes compete, but with different time course
TRACE predictions

38. Allopenna et al. Results

39. Study 1 Conclusions Time locked to speech at a fine grain
As predicted by interactive models, cohorts and rhymes are activated, with different time courses
Eye movement paradigm
More sensitive than conventional paradigms
More naturalistic
Simultaneous measures of multiple items
Transparently linkable to computational model
Time locked to speech at a fine grain

40. Theoretical conclusions
Natural contexts provide strong constraints that are used
When those constraints are extremely predictive, they are integrated as quickly as we can measure
Suggests rapid, continuous interaction among
Linguistic levels
Nonlinguistic context
Even for processes assumed to be low-level and automatic
Constrains processing theories, also has implications for, e.g., learnability

41. Producing words from pictures or from other words: A comparison of aphasic lexical access from two different input modalities
Gary Dell
with
Myrna Schwartz, Dan Foygel, Nadine Martin, Eleanor Saffran, Deborah Gagnon, Rick Hanley, Janice Kay, Susanne Gahl, Rachel Baron, Stefanie Abel, Walter Huber

42. Boxes and arrows in the linguistic system
Semantics
Syntax
Lexicon
Output
Phonology
Input
Phonology

43. Picture Naming Task
Semantics
Say: “cat”
Syntax
Lexicon
Output
Phonology
Input
Phonology

44. A 2-step Interactive Model of Lexical Access in Production
Semantic Features
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

45. Step 1 – Lemma Access Activate semantic features of CAT FOG DOG CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

46. Step 1 – Lemma Access Activation spreads through network FOG DOG CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

47. Step 1 – Lemma Access
Most active word from proper category is selected and linked to syntactic frame NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

48. Step 2 – Phonological Access
Jolt of activation is sent to selected word NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

49. Step 2 – Phonological Access
Activation spreads through network NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

50. Step 2 – Phonological Access
Most activated phonemes are selected
FOG
DOG
CAT
RAT
MAT
Syl
On Vo Co f r d k m ae o t g
Onsets
Vowels
Codas

51. Semantic Error – “dog” NP N
Shared features activate semantic neighbors NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

52. Formal Error – “mat” NP N
Phoneme-word feedback activates formal neighbors NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

53. Mixed Error – “rat”
Mixed semantic-formal neighbors gain activation from both top-down and bottom-up sources NP N
FOG
DOG
CAT
RAT
MAT f r d k m ae o t g
Onsets
Vowels
Codas

54. Errors of Phonological Access- “dat” “mat”
Selection of incorrect phonemes
FOG
DOG
CAT
RAT
MAT
Syl
On Vo Co f r d k m ae o t g
Onsets
Vowels
Codas

55. A Test of the Model: Picture-naming Errors in Aphasia
“cat”
175 pictures of concrete nouns–Philadelphia Naming Test
94 patients (Broca,Wernicke, anomic, conduction)
60 normal controls

56. Response Categories Correct Semantic Formal Mixed Unrelated Nonword
CAT DOG MAT RAT LOG DAT
Continuity Thesis:
Normal Error Pattern: 97% Correct
Random Error Pattern: 80% Nonwords
cat dog mat rat log dat
cat dog mat rat log dat

57. Implementing the Continuity Thesis
2. Set processing parameters of the model so that its error pattern matches the normal controls.
Random Pattern
Model Random Pattern
cat dog mat rat log dat
Normal Controls
Model Normal Pattern
1.Set up the model lexicon so that when noise is very large, it creates an error pattern similar to the random pattern.
cat dog mat rat log dat

58. Lesioning the model: The semantic-phonological weight hypothesis
Semantic Features
Semantic-word weight: S
FOG
DOG
CAT
RAT
MAT
Phonological- word weight: P f r d k m ae o t g
Onsets
Vowels
Codas

59. Patient CAT DOG MAT RAT LOG DAT
Correct Semantic Formal Mixed Unrelated Nonword LH
s=.024 p= IG
s=.019 p= GL
s=.010 p=

60. Representing Model-Patient Deviations
Root Mean Square Deviation (RMSD)
LH .016 IG
GL .043

61. 94 new patients—no exclusions
94.5 % of variance accounted for

62. Conclusions The logic underlying box-and-arrow- models
is perfectly compatible with connectionist models.
Connectionist principles augment the boxes and arrows with
-- a mechanism for quantifying degree of damage
-- mechanisms for error types and hence an explanation of the error patterns
Implications for recovery and rehabilitation

63. Behavioral and Imaging Experiments Ben Bergen and Shweta Narayan
Do Words and Images Match?
Behavioral – Image First
Does shared effector slow negative response?
Imaging – Simple sentence using verb first
Does verb evoke activity in motor effector area?

64. Structured Neural Computation in NTL
The theory we are outlining uses the computational modeling mechanisms of the Neural Theory of Language (NTL).
NTL makes use of structured connectionism (Not PDP connectionism!).
NTL is ‘localist,’ with functional clusters as units.
Localism allows NTL to characterize precise computations, as needed in actions and in inferences.

65. Simulation
To understand the meaning of the concept grasp, one must at least be able to imagine oneself or someone else grasping an object.
Imagination is mental simulation, carried out by the same functional clusters used in acting and perceiving.
The conceptualization of grasping via simulation therefore requires the use of the same functional clusters used in the action and perception of grasping.

66. Parameters
All actions, perceptions, and simulations make use of parameters and their values. Such neural parameterization is pervasive.
E.g., the action of reaching for an object makes use of the parameter of direction; the action of grasping an object makes use of the parameter of force.
The same parameter values that characterize the internal structure of actions and simulations of actions also characterize the internal structure of action concepts.

67. Advantages of Structured Connectionism
Structured connectionism operates on structures of the sort found in real brains.
From the structured connectionism perspective, the inferential structure of concepts is a consequence of the network structure of the brain and its organization in terms of functional clusters.

68. Multi-Modal Integration
Cortical premotor areas are endowed with sensory properties.
They contain neurons that respond to visual, somatosensory, and auditory stimuli.
Posterior parietal areas, traditionally considered to process and associate purely sensory information, alsos play a major role in motor control.

69. Somatotopy of Action Observation
Foot Action
Hand Action
Mirror Neurons: Parietal, Premotor (Area ), and Broca’s (45)
A: mimicked actions; no parietal activation
B: Action on objects; Parietal activation.
Point: location determined by action.
Mouth Action
Buccino et al. Eur J Neurosci 2001

70. The Simulation Hypothesis
How do mirror neurons work?
By simulation.
When the subject observes another individual doing an action, the subject is simulating the same action.
Since action and simulation use some of the same neural substrate, that would explain why the same neurons are firing during action-observation as during action-execution.

71. Mirror Neurons Achieve
Partial Universality, since they code
an action regardless of agent, patient,
modality (action/observation/hearing),
manner, location.
Partial Role Structure, since they code
an agent role and a purpose role.
The Agent Role:
In acting, the Subject is an agent of that action.
In observing, the Subject identifies the agent of
the action as having the same role as he has
when he is acting – namely, the agent role.
The Purpose Role: Mirror neurons fire only for
purposeful actions.

72. The Sensory-Motor System Is Sufficient
Conclusion 1
The Sensory-Motor System Is Sufficient
For at least one concept, grasp, functional clusters, as characterized in the sensory-motor system and as modeled using structured connectionist binding and inference mechanisms, have all the necessary conceptual properties.

73. The Neural Version of Ockham’s Razor
Conclusion 2
The Neural Version of Ockham’s Razor
Under the traditional theory, action concepts have to be disembodied, that is, to be characterized neurally entirely outside the sensory motor system.
If true, that would duplicate all the apparatus for characterizing conceptual properties that we have discussed. Unnecessary duplication of this sort is highly unlikely in a brain that works by neural optimization.

74. Behavioral Experiments Ben Bergen and Shweta Narayan
Do Words and Images Match?
Does shared effector slow negative response?
Behavioral – Image First
Does verb evoke activity in motor effector area?

76. WALK

78. GRASP

80. WALK

81. Preliminary Behavior Results
Same Action Other Effector Same Effector
788
804
871
767
785
825
40 Native Speakers
Eliminate RT > 2 sec.

82. 5 levels of Neural Theory of Language
Spatial Relation
Motor Control
Metaphor
Grammar
Cognition and Language
Computation
Structured Connectionism
abstraction
Neural Net
SHRUTI
Computational Neurobiology
Triangle Nodes
Biology
Neural Development
Quiz
Midterm
Finals