1. Linguistic Neuroscience: Extending Perceptual Neuroscience to Language Ling 411 – 12

2. “Linguistic Neuroscience”?  Applying the findings of perceptual neuroscience to language  Perceptual neuroscience as in Mountcastle’s 1998 book  Mountcastle doesn’t say anything about language  But his findings can be applied

3. Findings relating to columns ( Mountcastle, Perceptual Neuroscience, 1998)  The column is the fundamental module of perceptual systems probably also of motor systems  This columnar structure is found in all mammals that have been investigated  The theory is confirmed by detailed studies of visual, auditory, and somatosensory perception in living cat and monkey brains

4. Adjacency and the Proximity Principle  Neighboring areas for closely related functions The closer the function the closer the proximity  Consequences Members of same category will be in same area  Why? Same category because similar functions Competitors will be neighbors in the same area  Why? Neighbors in same area have same general function along with additional differentiating function They compete w.r.t. the differentiating function

5. Inhibitory connections Based on Mountcastle (1998)  Columnar specificity is maintained by pericolumnar inhibition (190) Activity in one column can suppress that in its immediate neighbors (191)  Inhibitory cells can also inhibit other inhibitory cells (193)  Inhibitory cells can connect to axons of other cells (“axoaxonal connections”)  Large basket cells send myelinated projections as far as 1-2 mm horizontally (193)

6. Extrapolation to Language?  Our knowledge of cortical columns comes mostly from studies of perception in cats, monkeys, and rats  Such studies haven’t been done for language Cats and monkeys don’t have language That kind of neurosurgical experiment isn’t done on human beings  Are they relevant to language anyway? Relevant if language uses similar cortical structures Relevant if linguistic functions are like perceptual functions

7. Perception and Language  Why haven’t such studies been done for language? 1.That kind of neurosurgical experiment isn’t done on human beings 2.Cats and monkeys don’t have language  Are they relevant to language anyway? 1.Relevant if language uses similar cortical structures 2.Relevant if linguistic functions are like perceptual functions

8. Relevance to Language  These studies of perception are relevant if Perceptual structure and functions are basically the same across modalities  Including associative areas (higher-level) Linguistic comprehension is basically a perceptual process

9. Objection  Cats and monkeys don’t have language  Language (as we know it) is a unique human faculty  Therefore language must have unique properties of its structural representation in the cortex  Answer: Yes, language is different, but The differences are a consequence not of different (local) structure but differences of connectivity The neurocognitive network does not have different kinds of structure for different kinds of information  Rather, different connectivities

10. Justifying extrapolation  Hypothesis: Extrapolation of findings about cortical columns can be extended to humans linguistic and conceptual structures  Why? Summary of the argument  Cortical structure, viewed locally, is uniform across mammalian species uniform across different cortical regions Exceptions in primary visual and primary auditory areas  Different cortical regions have different functions because of differences in connectivity not because of differences in structure

11. Essence of the argument  Cortical structure and function, locally, are essentially the same in humans as in cats and monkeys and rats  Moreover, in humans, The regions that support language have the same structure locally as other cortical regions

12. Uniformity of cortical function  Claim: Locally, all cortical processing is the same The apparent differences of function are consequences of differences in larger- scale connectivity  Conclusion (if the claim is supported): Understanding language, even at higher levels, is basically a perceptual process

13. Argument for local uniformity of representation  Different types of cortical information Perceptual Conceptual Grammatical Phonological  How are they different?  Two possibilities 1. They differ in their structural form 2. They differ based on their connections  Claim: Possibility #2 is the correct one The “connectionist claim”

14. Support for the connectionist claim  Lines and nodes (i.e., columns) are approximately the same all over  Uniformity of cortical structure Same kinds of columnar structure Same kinds of neurons Same kinds of connections  Conclusion: Different areas have different functions because of what they are connected to

15. Linguistic Information in the Cortex  Problem: Linguistic information is usually described symbolically  In the symbolic mode of description, different kinds of linguistic information appear to have different kinds of structure Phonology Morphology  Regular and irregular inflections Syntax Semantics  Claim: If the information is viewed as connectional instead of symbolic, it turns out to have a high degree of uniformity

16. Uniformity of cortical structure  Six layers throughout, with similar structure  Columns throughout  Same neuron types everywhere – pyramidal most frequent, spiny stellate in layer IV, etc.  Inhibitory and excitatory connections throughout  Same neurotransmitters everywhere Excitatory: glutamate Inhibitory: GABA  But: What about the different Brodmann areas? 1.The differences are relatively minor 2.They may be based on experience

17. Structural Uniformity? A closer look  Differences are found at lower levels Primary sensory areas have specialized structures These are called heterotypical areas Properties of columns depend on afferent inflow  More uniformity in higher-level areas “Homotypical” areas (i.e., same type) Relatively uniform structure Makes them flexible, adaptable Properties depend on intracortical processing Different homotypical areas differ not because of their structures but because of their connections

18. A heterotypical area: Visual motion perception An area in the posterior bank of the superior temporal sulcus of a macaque monkey (“V-5”) Albright et al. 1984 400-500 μ

19. Auditory areas in a cat’s cortex (Heterotypical) AAF – Anterior auditory field A1 – Primary auditory field PAF – Posterior auditory field VPAF – Ventral posterior auditory field A1

20. Exceptions: Diversity in cortical function  Perception vs. production Back brain vs. front brain  Sharpness of contrast Phonology and morphology require sharp contrasts Conceptual categories have fuzzy definitions  Left vs. right hemisphere RH may have..  Larger minicolumns  Less lateral inhibition

21. Uniformity in LH Associative Areas  The associative areas are homotypical  The structure that subserves language understanding is the same as perceptual structure Columns of similar structure With similar kinds of connections  Claim: Understanding language is the same process as perception To support this claim, must look more closely at cortical function Subclaim: Locally, all columns function alike

22. Primary areas and higher-level areas (LH) These are homotypical

23. The uniformity of information?  Different types of cortical information Perceptual Conceptual Grammatical Phonological  How are they different?  Two possibilities 1. They differ in their form of representation 2. They differ based on their connections  Claim: Possibility #2 is the correct one The “connectivity claim”

24. Uniformity of cortical function  If cortical function is uniform across mammals and across different cortical areas, then the findings presented by Mountcastle can be extended to language  Claims: Locally, all cortical processing is the same The apparent differences of function are consequences of differences in larger-scale connectivity  Conclusion (if the claim is supported): Understanding language, even at higher levels, is basically a perceptual process

25. Testing the claim  Claim: The apparent differences of function are consequences of differences in larger-scale connectivity  To test, we need to understand cortical function  That means we have to better understand the function of the cortical column

26. Columns do not store symbols!  They only Receive activation Maintain activation Inhibit competitors Transmit activation  Important consequence: We have linguistic information represented in the cortex without the use of symbols It’s all in the connectivity  The Challenge: How? This claim goes against most of the history of linguistics

27. Operation of the Network  The linguistic system operates as distributed processing of multiple individual components – cortical columns  Columnar Functions Integration: A column is activated if it receives enough activation from other columns  Can be activated to varying degrees  Can keep activation alive for a period of time Broadcasting: An activated column transmits activation to other columns  Exitatory – contribution to higher level  Inhibitory – dampens competition at same level  Columns do not store symbols!

28. Why the usual approach won’t work  Let us suppose that words are stored in some kind of symbolic form  What form?  If written, there has to be.. something in there that can read them something in there that can write them something in there that can move them around, from one place to another something in there to compare them with forms entering the brain as it hears someone speaking – otherwise, how can an incoming word be recognized?

29. Why the usual approach won’t work (cont’d)  If not written, then represented in some other medium  Doesn’t solve the problem  You still need whatever kind of sensory detectors can sense the symbols in whatever medium you choose  Plus means of performing all those other operations

30. Compare imagery  Visual images Little pictures? If so, what is in there to see them?  Auditory images Little sounds vibrating in the brain? If so, what is in there to hear them?  There has to be another way!

31. There must be another way  Visual imagery (e.g. of your grandmother) Reactivation of some of the same nodes and connections that operate when actually seeing her  Auditory imagery (e.g. of a tune) Reactivation of some of the same nodes and connections that operate in actually hearing it

32. Another way, for language  A syllable Activation of the nodes and connections needed to recognize or produce it  A word Activation of the nodes and connections needed to recognize it  A syntactic construction Activation of the nodes and connections needed to recognize or produce it

33. The postulation of objects as something different from the terms of relationships is a superfluous axiom and consequently a metaphysical hypothesis from which linguistic science will have to be freed. Louis Hjelmslev Prolegomena to a Theory of Language (1943: 61) Hjelmslev’s view

34. Columnar Functions: Integration and Broadcasting  Integration: A column is activated if it receives enough activation from Other columns Thalamus  Can be activated to varying degrees  Can keep activation alive for a period of time  Broadcasting: An activated column transmits activation to other columns Exitatory Inhibitory  Learning: adjustment of connection strengths and thresholds

35. Integration and Broadcasting  Broadcasting To multiple locations In parallel  Integration

36. Integration and Broadcasting Integration Broadcasting Wow, I got activated! Now I’ll tell my friends!

37. What matters is not ‘what’ but ‘where’  What distinguishes one kind of information from another is what it is connected to  Lines and nodes are approximately the same all over  Hence, uniformity of cortical structure Same kinds of columnar structure Same kinds of neurons Same kinds of connections  Different areas have different functions because of what they are connected to

38. Operations in neurocognitive networks  Activation moves along lines and through nodes Integration Broadcasting  Connection strengths are variable A connection becomes stronger with repeated successful use A stronger connection can carry greater activation

39. Can language be accounted for by such simple operations?  Phonology  Words and their meanings  Syntax and morphology  Conceptual relationships

40. Sequence  In language, sequence is very important Word order Order of phonological elements in syllables Etc.  Also important in many non-linguistic areas Dancing Eating a meal  Can cortical columns handle sequences?

41. Lasting activation in minicolumn Subcortical locations Connections to neighboring columns not shown Cell Types Pyramidal Spiny Stellate Inhibitory Recurrent axon branches keep activation alive in the column – Until is is turned off by inhibitory cell

42. Notation for lasting activation > Thick border for a node that stays active for a relatively long time > Thin border for a node that stays active for a relatively short time

43. Recognizing items in sequence This link stays active a b Node c is satisfied by activation from both a and b If satisfied it sends activation to output connections Node a keeps itself active for a while Suppose that node b is activated after node a Then c will recognize the sequence ab c This node recognizes the sequence ab

44. Recognizing stop consonants  Consider stop consonants, e.g. t, d  At the time of closure For voiceless stops there is no sound to hear For voiced stops, very little sound  The stops are identified by transitions To following vowel From preceding vowel

45. Demisyllables [di, de, da, du] F1 and F2 For [a] It is unlikely that [d] is represented as a unit in perception

46. Recognizing a syllable and its demisyllables dim di- -im Cardinal node for dim Functional subweb for dim Auditory features of [di-] Auditory features of [-im] Just labels

47. Another syllable and its demisyllables bil bi- -il Cardinal node for bill Subweb for bill

48. Multiple connections of -il bil hil kil bi- -il Bill hill mill kill etc. One and the same /-il/ in all of them

49. Multiple connections of -il bil hil kil bi- -il Bill hill mill kill etc. Similarly for multiple connections of bi- bit, bib, bid, etc.

50. Multiple connections of -il bil hil kil bi- -il Bill hill mill kill etc. To lower level nodes, for phonological features

51. Syntactic Recognition – same principle This link stays active a b Let node a represent Noun Phrases (Subject) and let b represent Predicates (Verb Phrases etc.) Then c represents Clauses: the sequence ab c This node recognizes the sequence ab

52. Syntactic Recognition: higher-level perception This link stays active a b The whole process is one of recognition, just as at lower levels (e.g., phonological recognition) Same structures, different connections c This node recognizes the sequence ab

53. Perhaps All of linguistic structure is relational?  Remains to be shown for Syntax (beyond the essence: recognizing sequence) Regular and irregular inflection Lexical structure  If it can be shown, then:  The whole of linguistic structure is a connectionist system No symbols – it’s all relationships  Good thing, since that is exactly the kind of system that the cortex is built to represent and to operate with

54. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography  Property III: Nodal specificity  Property IV: Adjacency  Property V: Extension of II-IV to larger columns  Property VI: Competition

55. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function Therefore, the column behaves as a single unit  A node of the linguistic network  Property II: Cortical topography  Property III: Nodal specificity  Property IV: Adjacency  Property V: Extension of II-IV to larger columns  Property VI: Competition

56. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography Linguistic structure as a two-dimensional array of nodes  Property III: Nodal specificity  Property IV: Adjacency  Property V: Extension of II-IV to larger columns  Property VI: Competition

57. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography  Property III: Nodal specificity Every linguistic node has a specific function  Property IV: Adjacency  Property V: Extension of II-IV to larger columns  Property VI: Competition

58. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography  Property III: Nodal specificity  Property IV: Adjacency Adjacent linguistic nodes have similar linguistic functions  For example, nodes for stop consonants  Property V: Extension of II-IV to larger columns  Property VI: Competition

59. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography  Property III: Nodal specificity  Property IV: Adjacency  Property V: Extension of II-IV to larger columns Linguisic categories in neighboring cortical areas  Property VI: Competition

60. Deductions from findings about cortical columns  If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:  Property I: Intra-column uniformity of function  Property II: Cortical topography  Property III: Nodal specificity  Property IV: Adjacency  Property V: Extension of II-IV to larger columns  Property VI: Competition Contiguous linguistic nodes are in competition  E.g., stop consonants

61. Operation of the Network  The linguistic system operates as distributed processing of multiple individual components – cortical columns  Columnar Functions Integration: A column is activated if it receives enough activation from other columns  Can be activated to varying degrees  Can keep activation alive for a period of time Broadcasting: An activated column transmits activation to other columns  Exitatory – contribution to higher level  Inhibitory – dampens competition at same level  Columns do not store symbols! Review

62. end