1. Abstract Neuron w2w2 wnwn w1w1 w0w0 i 0 =1 o u t p u t y i2i2 inin i1i1... i n p u t i 1 if net > 0 0 otherwise {

2. Link to Vision: The Necker Cube

4. physicslowest energy state chemistrymolecular minima biology fitness, MEU N euroeconomics vision threats, friends language errors, NTL Constrained Best Fit in Nature inanimate animate

5. Computing other relations The 2/3 node is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active Such a node is also called a triangle node and will be useful for lots of representations.

6. Triangle nodes and McCullough- Pitts Neurons? Object (B)Value (C) Relation (A) ABC

7. “They all rose” triangle nodes: when two of the abstract neurons fire, the third also fires model of spreading activation

8. Basic Ideas Parallel activation streams. Top down and bottom up activation combine to determine the best matching structure. Triangle nodes bind features of objects to values Mutual inhibition and competition between structures Mental connections are active neural connections

9. Behavioral Experiments Identity – Mental activity is Structured Neural Activity Spreading Activation — Psychological model/theory behind priming and interference experiments Simulation — Necessary for meaningfulness and contextual inference Parameters — Govern simulation, strict inference, link to language

10. Bottom-up vs. Top-down Processes Bottom-up: When processing is driven by the stimulus Top-down: When knowledge and context are used to assist and drive processing Interaction: The stimulus is the basis of processing but almost immediately top- down processes are initiated

11. Stroop Effect Interference between form and meaning

12. Name the words Book Car Table Box Trash Man Bed Corn Sit Paper Coin Glass House Jar Key Rug Cat Doll Letter Baby Tomato Check Phone Soda Dish Lamp Woman

13. Name the print color of the words Blue Green Red Yellow Orange Black Red Purple Green Red Blue Yellow Black Red Green White Blue Yellow Red Black Blue White Red Yellow Green Black Purple

14. Procedure for experiment that demonstrates the word-superiority effect. First the word is presented, then the XXXX’s, then the letters.

15. Word-Superiority Effect Reicher (1969) Which condition resulted in faster & more accurate recognition of the letter? –The word condition –Letters are recognized faster when they are part of a word then when they are alone –This rejects the completely bottom-up feature model –Also a challenge for serial processing

16. Connectionist Model McClelland & Rumelhart (1981) Knowledge is distributed and processing occurs in parallel, with both bottom-up and top-down influences This model can explain the Word- Superiority Effect because it can account for context effects

17. Connectionist Model of Word Recognition

18. Interaction in language processing: Pragmatic constraints on lexical access Jim Magnuson Columbia University

19. Information integration A central issue in psycholinguistics and cognitive science: –When/how are such sources integrated? Two views –Interaction Use information as soon as it is available Free flow between levels of representation –Modularity Protect and optimize levels by encapsulation Staged serial processing Reanalyze / appeal to top-down information only when needed

21. Reaction Times in Milliseconds after: “They all rose” flower685659 stood677623 desk711652 0 delay 200ms. delay

22. Example: Modularity and word recognition Tanenhaus et al. (1979) [also Swinney, 1979] –Given a homophone like rose, and a context biased towards one sense, when is context integrated? Spoken sentence primes ending in homophones: –They all rose vs. They bought a rose Secondary task: name a displayed orthographic word –Probe at offset of ambiguous word: priming for both “stood” and “flower” –200 ms later: only priming for appropriate sense Suggests encapsulation followed by rapid integration But the constraint here is weak -- overestimates modularity? How could we examine strong constraints in natural contexts?

24. Eye tracking computer Eye camera Scene camera Allopenna, Magnuson & Tanenhaus (1998) ‘Pick up the beaker’

25. Do rhymes compete? Cohort (Marlsen-Wilson) : onset similarity is primary because of the incremental nature of speech ( serial/staged; Shortlist/Merge ) –Cat activates cap, cast, cattle, camera, etc. –Rhymes won’t compete NAM (Neighborhood Activation Model; Luce) : global similarity is primary –Cat activates bat, rat, cot, cast, etc. –Rhymes among set of strong competitors TRACE (McClelland & Elman) : global similarity constrained by incremental nature of speech –Cohorts and rhymes compete, but with different time course TRACE predictions

26. Allopenna et al. Results

27. Study 1 Conclusions As predicted by interactive models, cohorts and rhymes are activated, with different time courses Eye movement paradigm –More sensitive than conventional paradigms –More naturalistic –Simultaneous measures of multiple items –Transparently linkable to computational model Time locked to speech at a fine grain

28. Theoretical conclusions Natural contexts provide strong constraints that are used When those constraints are extremely predictive, they are integrated as quickly as we can measure Suggests rapid, continuous interaction among –Linguistic levels –Nonlinguistic context Even for processes assumed to be low-level and automatic Constrains processing theories, also has implications for, e.g., learnability

29. Producing words from pictures or from other words : A comparison of aphasic lexical access from two different input modalities with Myrna Schwartz, Dan Foygel, Nadine Martin, Eleanor Saffran, Deborah Gagnon, Rick Hanley, Janice Kay, Susanne Gahl, Rachel Baron, Stefanie Abel, Walter Huber Gary Dell

30. Boxes and arrows in the linguistic system Semantics Lexicon Output Phonology Input Phonology Syntax

31. Semantics Lexicon Output Phonology Input Phonology Syntax Picture Naming Task Say: “cat”

32. A 2-step Interactive Model of Lexical Access in Production FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Semantic Features

33. Step 1 – Lemma Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Activate semantic features of CAT

34. Step 1 – Lemma Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Activation spreads through network

35. Step 1 – Lemma Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Most active word from proper category is selected and linked to syntactic frame NP N

36. Step 2 – Phonological Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Jolt of activation is sent to selected word NP N

37. Step 2 – Phonological Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Activation spreads through network NP N

38. Step 2 – Phonological Access FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Most activated phonemes are selected Syl On Vo Co

39. Semantic Error – “dog” FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Shared features activate semantic neighbors NP N

40. Formal Error – “mat” FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Phoneme-word feedback activates formal neighbors NP N

41. Mixed Error – “rat” FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Mixed semantic-formal neighbors gain activation from both top-down and bottom-up sources NP N

42. Errors of Phonological Access- “dat” “mat” FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Selection of incorrect phonemes Syl On Vo Co

43. A Test of the Model: Picture-naming Errors in Aphasia “cat” 175 pictures of concrete nouns–Philadelphia Naming Test 94 patients (Broca,Wernicke, anomic, conduction) 60 normal controls

44. Response Categories Normal Error Pattern: 97% Correct Random Error Pattern: 80% Nonwords Correct Semantic Formal Mixed Unrelated Nonword CAT DOG MAT RAT LOG DAT Continuity Thesis : cat dog mat rat log dat

45. Implementing the Continuity Thesis 1.Set up the model lexicon so that when noise is very large, it creates an error pattern similar to the random pattern. Random Pattern Model Random Pattern 2. Set processing parameters of the model so that its error pattern matches the normal controls. Normal Controls Model Normal Pattern cat dog mat rat log dat

46. Lesioning the model: The semantic- phonological weight hypothesis FOGDOGCATRATMAT frdkmaeotg Onsets VowelsCodas Semantic Features Semantic-word weight: S Phonological- word weight: P

47. Patient CAT DOG MAT RAT LOG DAT Correct Semantic Formal Mixed Unrelated Nonword LH.71.03.07.01.02.15 IG.77.10.06.03.01.03 GL.29.04.22.03.10.32 s=.019 p=.032.77.09.06.01.04.03 s=.024 p=.018.69.06.06.01.02.17 s=.010 p=.016.31.10.15.01.13.30

48. Representing Model-Patient Deviations Root Mean Square Deviation (RMSD) LH.016 IG.016 GL.043

49. 94 new patients—no exclusions 94.5 % of variance accounted for

50. Conclusions The logic underlying box-and-arrow- models is perfectly compatible with connectionist models. Connectionist principles augment the boxes and arrows with -- a mechanism for quantifying degree of damage -- mechanisms for error types and hence an explanation of the error patterns Implications for recovery and rehabilitation

51. Behavioral and Imaging Experiments Ben Bergen and Shweta Narayan Do Words and Images Match? Behavioral – Image First Does shared effector slow negative response? Imaging – Simple sentence using verb first Does verb evoke activity in motor effector area? Metaphor follow-on experiment Will “kick the idea around” evoke motor activity?

52. Structured Neural Computation in NTL The theory we are outlining uses the computational modeling mechanisms of the Neural Theory of Language (NTL). NTL makes use of structured connectionism (Not PDP connectionism!). NTL is ‘localist,’ with functional clusters as units. Localism allows NTL to characterize precise computations, as needed in actions and in inferences.

53. Simulation To understand the meaning of the concept grasp, one must at least be able to imagine oneself or someone else grasping an object. Imagination is mental simulation, carried out by the same functional clusters used in acting and perceiving. The conceptualization of grasping via simulation therefore requires the use of the same functional clusters used in the action and perception of grasping.

54. Parameters All actions, perceptions, and simulations make use of parameters and their values. Such neural parameterization is pervasive. E.g., the action of reaching for an object makes use of the parameter of direction; the action of grasping an object makes use of the parameter of force. The same parameter values that characterize the internal structure of actions and simulations of actions also characterize the internal structure of action concepts.

55. Advantages of Structured Connectionism Structured connectionism operates on structures of the sort found in real brains. From the structured connectionism perspective, the inferential structure of concepts is a consequence of the network structure of the brain and its organization in terms of functional clusters.

56. Multi-Modal Integration Cortical premotor areas are endowed with sensory properties. They contain neurons that respond to visual, somatosensory, and auditory stimuli. Posterior parietal areas, traditionally considered to process and associate purely sensory information, alsos play a major role in motor control.

57. Somatotopy of Action Observation Foot Action Hand Action Mouth Action Buccino et al. Eur J Neurosci 2001

58. The Simulation Hypothesis How do mirror neurons work? By simulation. When the subject observes another individual doing an action, the subject is simulating the same action. Since action and simulation use some of the same neural substrate, that would explain why the same neurons are firing during action-observation as during action-execution.

59. Mirror Neurons Achieve Partial Universality, since they code an action regardless of agent, patient, modality (action/observation/hearing), manner, location. Partial Role Structure, since they code an agent role and a purpose role. The Agent Role: In acting, the Subject is an agent of that action. In observing, the Subject identifies the agent of the action as having the same role as he has when he is acting – namely, the agent role. The Purpose Role: Mirror neurons fire only for purposeful actions.

60. Conclusion 1 The Sensory-Motor System Is Sufficient For at least one concept, grasp, functional clusters, as characterized in the sensory-motor system and as modeled using structured connectionist binding and inference mechanisms, have all the necessary conceptual properties.

61. Conclusion 2 The Neural Version of Ockham’s Razor Under the traditional theory, action concepts have to be disembodied, that is, to be characterized neurally entirely outside the sensory motor system. If true, that would duplicate all the apparatus for characterizing conceptual properties that we have discussed. Unnecessary duplication of this sort is highly unlikely in a brain that works by neural optimization.

62. Behavioral and Imaging Experiments Ben Bergen and Shweta Narayan Do Words and Images Match? Does shared effector slow negative response? Imaging – Simple sentence using verb first Behavioral – Image First Does verb evoke activity in motor effector area?

64. WALK

66. GRASP

68. WALK

69. Preliminary Behavior Results 788804871 767785825 Same Action Other Effector Same Effector 40 Native Speakers Eliminate RT > 2 sec.

70. 5 levels of Neural Theory of Language Cognition and Language Computation Structured Connectionism Computational Neurobiology Biology MidtermQuiz Finals Neural Development Triangle Nodes Neural Net Spatial Relation Motor Control Metaphor SHRUTI Grammar abstraction