1. The Past Tense Neural Networks and Non-Symbolic Computation

2. Abstraction (again!) Powerful, but costly
How much is needed in human language? Model system: English past tense

3. Classic Developmental Story
Initial mastery of regular and irregular past tense forms
Overregularization appears only later (e.g. goed, comed)
‘U-Shaped’ developmental pattern taken as evidence for learning of a morphological rule V + [+past] --> stem + /d/

4. Rumelhart & McClelland 1986
Model learns to classify regulars and irregulars, based on sound similarity alone. Shows U-shaped developmental profile.

5. What is really at stake here?
Abstraction
Operations over variables
Symbol manipulation
Algebraic computation
Learning based on input
How do learners generalize beyond input?
y = 2x

6. Gary Marcus

7. Functions
Input Output 4,
2, 3 5 1, 6,
341,

8. Functions Input Output rock rock sing sing alqz alqz dark dark
lamb lamb

9. Functions
Input Output

10. Functions Input Output look looked rake raked sing sang go went
want wanted

11. Functions Input Output John left 1 Wallace fed Gromit 1
Fed Wallace Gromit 0
Who do you like Mary and? 0

12. Learning Functions
Learners are shown examples of what the function generates, and have to figure out what the function is.
Think of language/grammar as a very big function (or set of functions). Learning task is similar – learner is presented with examples of what the function generates, and has to figure out what the system is.
Main question in language acquisition: what does the learner need to know in order to successfully figure out what this function is?
Questions about Neural Networks
How can a network represent a function?
How can the network discover what this function is?

13. What is not at stake here
Feedback, negative evidence, etc.

14. Who has the most at stake here?
Those who deny the need for rules/variables in language have the most to lose here …if the English past tense is hard, just wait until you get to the rest of natural language!
…but if they are successful, they bring with them a simple and attractive learning theory, and mechanisms that can readily be grounded at the neural level
However, if the advocates of rules/variables succeed here or elsewhere, they face the more difficult challenge at the neuroscientific level

17. Pinker
Ullman

19. Beyond Sound Similarity
Regulars and Associative Memory
1. Are regulars different? 2. Do regulars implicate operations over variables?
Neuropsychological Dissociations
Other Domains of Morphology

20. Beyond Sound Similarity
Regulars and Associative Memory
1. Are regulars different? 2. Do regulars implicate operations over variables?
Neuropsychological Dissociations
Other Domains of Morphology

21. Beyond Sound Similarity
Zero-derived denominals are regular
Soldiers ringed the city
*Soldiers rang the city
high-sticked, grandstanded, …
*high-stuck, *grandstood, …
Productive in adults & children
Shows sensitivity to morphological structure [[ stem N] ø V]-ed

22. (Pinker & Ullman 2002)

23. Beyond Sound Similarity
Zero-derived denominals are regular
Soldiers ringed the city
*Soldiers rang the city
high-sticked, grandstanded, …
*high-stuck, *grandstood, …
Productive in adults & children
Shows sensitivity to morphological structure [[ stem N] ø V]-ed
Provides good evidence that sound similarity is not everything
But nothing prevents a model from using richer similarity metric
morphological structure (for ringed)
semantic similarity (for low-lifes)

24. Beyond Sound Similarity
Regulars and Associative Memory
1. Are regulars different? 2. Do regulars implicate operations over variables?
Neuropsychological Dissociations
Other Domains of Morphology

25. Two types of arguments
Storage of regulars
Default forms

26. Regulars & Associative Memory
Regulars are productive, need not be stored
Irregulars are not productive, must be stored
But are regulars immune to effects of associative memory?
frequency
over-irregularization
Pinker & Ullman:
regulars may be stored
but they can also be generated on-the-fly
‘race’ can determine which of the two routes wins
some tasks more likely to show effects of stored regulars

27. Base frequencies matched

28. singular freq. matched
base freq. matched

29. English
Singular frequency matched

30. Child vs. Adult Impairments
Specific Language Impairment
Early claims that regulars show greater impairment than irregulars are not confirmed
Pinker & Ullman 2002b
‘The best explanation is that language-impaired people are indeed impaired with rules, […] but can memorize common regular forms.’
Regulars show consistent frequency effects in SLI, not in controls.
‘This suggests that children growing up with a grammatical deficit are better at compensating for it via memorization than are adults who acquired their deficit later in life.’

32. (Clahsen, 1999)

33. Low-Frequency Defaults
German Plurals
die Straße die Straßen die Frau die Frauen
der Apfel die Äpfel die Mutter die Mütter
das Auto die Autos der Park die Parks die Schmidts
-s plural low frequency, used for loan-words, denominals, names, etc.
Response
frequency is not the critical factor in a system that focuses on similarity
distribution in the similarity space is crucial
similarity space with islands of reliability
network can learn islands
or network can learn to associate a form with the space between the islands

34. Beyond Sound Similarity
Regulars and Associative Memory
1. Are regulars different? 2. Do regulars implicate operations over variables?
Neuropsychological Dissociations
Other Domains of Morphology

35. Neuropsychological Dissociations
Ullman et al. 1997
Alzheimer’s disease patients
Poor memory retrieval
Poor irregulars
Good regulars
Parkinson’s disease patients
Impaired motor control, good memory
Good irregulars
Poor regulars
Striking correlation involving laterality of effect
Marslen-Wilson & Tyler 1997
Normals
past tense primes stem
2 Broca’s Patients
irregulars prime stems
inhibition for regulars
1 patient with bilateral lesion
regulars prime stems
no priming for irregulars or semantic associates

36. Alzheimer’s Disease

37. Parkinson’s Disease

38. Morphological Priming
Lexical Decision Task
CAT, TAC, BIR, LGU, DOG
press ‘Yes’ if this is a word
Priming
facilitation in decision times when related word precedes target (relative to unrelated control)
e.g., {dog, rug} - cat
Marslen-Wilson & Tyler 1997
Regular {jumped, locked} - jump
Irregular {found, shows} - find
Semantic {swan, hay} - goose
Sound {gravy, sherry} - grave

40. Neuropsychological Dissociations
Bird et al. 2003
complain that arguments for selective difficulty with regulars are confounded with the phonological complexity of the word-endings
Pinker & Ullman 2002
weight of evidence still supports dissociation; Bird et al.’s materials contained additional confounds

41. Brain Imaging Studies Münte et al. 1997 Is this evidence decisive?
Jaeger et al. 1996, Language
PET study of past tense
Task: generate past from stem
Design: blocked conditions
Result: different areas of activation for regulars and irregulars
Is this evidence decisive?
task demands very different
difference could show up in network
doesn’t implicate variables
Münte et al. 1997
ERP study of violations
Task: sentence reading
Design: mixed
Result:
regulars: ~LAN
irregulars: ~N400
Is this evidence decisive?
allows possibility of comparison with other violations

42. Regular
Irregular
Nonce
(Jaeger et al. 1996)

46. Beyond Sound Similarity
Regulars and Associative Memory
1. Are regulars different? 2. Do regulars implicate operations over variables?
Neuropsychological Dissociations
Other Domains of Morphology

47. Abstraction Phonological categories, e.g., /b/
Treating different sounds as equivalent
Failure to discriminate members of the same category
Treating minimally different words as the same
Efficient memory encoding
Morphological concatenation, e.g., V + ed
Productivity: generalization to novel words, novel sounds
Frequency-insensitivity in memory encoding
Association with other aspects of ‘procedural memory’

48. Gary Marcus

49. Generalization Training Items Test Item Input: 1 0 1 0 Output: 1 0 1 0

50. Generalization Training Items Test Item Input: 1 0 1 0 Output: 1 0 1 0
(Humans)
(Network)

51. Generalization Training Items Test Item
Input: Output:
Input: Output:
Input: Output:
Input: Output:
Test Item
Input: Output ? ? ? ?
Generalization fails because learning is local
(Humans)
(Network)

52. Generalization Training Items Test Item
Input: Output:
Input: Output:
Input: Output:
Input: Output:
Test Item
Input: Output ? ? ? ?
Generalization succeeds because representations are shared
(Humans)
(Network)

53. Now another example…

54. Shared Representation
Copying 1:
Copying 2:
“The key to the representation of variables is whether all inputs in a class are represented by a single node.”

55. Generalization
“In each domain in which there is generalization, it is an empirical question whether the generalization is restricted to items that closely resemble training items or whether the generalization can be freely extended to all novel items within some class.”

56. How far can a model generalize to novel forms?
All novel forms that it can represent
Only some of the novel forms that it can represent
Velar fricative [x], e.g., Bach
What would lead an English speaker to generate the correct plural for Bach?

58. Hebrew Word Formation Roots Word patterns lmd learning dbr talking
CiCeC limed ‘he learned’
CiCeC diber ‘he talked’
CaCaC lamad ‘he studied’
CiCuC limud ‘study’
hitCaCeC hitlamed ‘he taught himself’

60. English phonemes absent from Hebrew
j (as in jeep)
ch (as in chair)
th (as in thick) <-- features absent from Hebrew
w (as in wide)
Do speakers generalize the Obligatory Contour Principle (OCP) constraint effects?
XXY < YXX
jjr < rjj

61. Root position vs. word position
*jjr
jajartem
hijtajartem hiCtaCaCtem

62. Ratings derived from rankings for word-triples 1 = best, 3 = worst, scores subtracted from 4

64. What have we learned …?
Storage of rule-generated word forms + decomposition
Anomaly detection insensitive to stem frequency
Ongoing debates over criteria for establishing ‘defaults’
Neuropsychological dissociations
Generalization beyond trained items
Does evidence from morphological processing implicate a qualitative distinction between memorized forms and rules (operations involving variables)?