The ICSI/Berkeley Neural Theory of Language Project ECG Learning early constructions (Chang, Mok) VERY brief overview, to situate my project: The general tactic is to view complex cognitive phenomena as having more than one level of analysis, here 5 levels. (Cog/ling level that cogsci people study; computational level with relatively standard CS rep’ns; neural networks with structure, etc.) Importantly, this reduction or abstraction is constrained in that structures or representations used at one level should have equivalent translations or implementations at the more concrete or biologically inspired levels. [Caveat: it’s the NTL project; “neural” is the goal, not necessarily the status quo. Any input welcome.] e.g. SHRUTI (binding via temporal synchrony) Most of the previous work has been at the top several levels; my research fits in there too. I will give a brief illustration of the general approach and show how that leads naturally to the model I’ll be describing (learning early constructions).

Connectionist Model of Word Recognition (Rumelhart and McClelland)

Constraints on Connectionist Models 100 Step Rule Human reaction times ~ 100 milliseconds Neural signaling time ~ 1 millisecond Simple messages between neurons Long connections are rare No new connections during learning Developmentally plausible

Can we formalize/model these intuitions What is a neurally plausible computational model of spreading activation that captures these features. What does semantics mean in neurally embodied terms What are the neural substrates of concepts that underlie verbs, nouns, spatial predicates?

Abstract Neuron { o u t p u t y i n p u t i w0 I0 = 1 w2 wn w1 i2 in 1 if net > 0 0 otherwise { w0 I0 = 1 w2 wn w1 i2 in i1 . . . i n p u t i

Computing with Abstract Neurons McCollough-Pitts Neurons were initially used to model pattern classification size = small AND shape = round AND color = green AND location = on_tree => unripe linking classified patterns to behavior size = large OR motion = approaching => move_away size = small AND direction = above => move_above McCollough-Pitts Neurons can compute logical functions. AND, NOT, OR

Distributed vs Localist Rep’n John 1 Paul George Ringo John 1 Paul George Ringo What are the drawbacks of each representation?

Distributed vs Localist Rep’n John 1 Paul George Ringo John 1 Paul George Ringo What happens if you want to represent a group? How many persons can you represent with n bits? 2^n What happens if one neuron dies? How many persons can you represent with n bits? n

Sparse Distributed Representation

Visual System 1000 x 1000 visual map For each location, encode: orientation direction of motion speed size color depth Blows up combinatorically! …

Coarse Coding info you can encode with one fine resolution unit = info you can encode with a few coarse resolution units Now as long as we need fewer coarse units total, we’re good

Coarse-Fine Coding Y X G G but we can run into ghost “images” Coarse in F2, Fine in F1 Feature 1 e.g. Orientation Y X Y-Orientation G G X-Orientation but we can run into ghost “images” Coarse in F1, Fine in F2 Y-Dir X-Dir Feature 2 e.g. Direction of Motion

Connectionist Models in Cognitive Science Structured PDP Hybrid Neural Conceptual Existence Data Fitting

Models of Learning Hebbian ~ coincidence Recruitment ~ one trial – Lecture 14 Supervised ~ correction (backprop) Reinforcement ~ delayed reward Unsupervised ~ similarity

Computing other relations The 2/3 node is a useful function that activates its outputs (3) if any (2) of its 3 inputs are active Such a node is also called a triangle node and will be useful for lots of representations.

Triangle nodes and McCullough-Pitts Neurons? B C A A B C

“They all rose” triangle nodes: when two of the neurons fire, the third also fires model of spreading activation

Spreading activation and feature structures Parallel activation streams. Top down and bottom up activation combine to determine the best matching structure. Triangle nodes bind features of objects to values Mutual inhibition and competition between structures Mental connections are active neural connections

Representing concepts using triangle nodes

Feature Structures in Four Domains Barrett Ham Container Push dept~CS Color ~pink Inside ~region Schema ~slide sid~001 Taste ~salty Outside ~region Posture ~palm emp~GSI Bdy. ~curve Dir. ~ away Schneider Pea Purchase Stroll dept~Ling Color ~green Buyer ~person Schema ~walk sid~002 Taste ~sweet Seller ~person Speed ~slow emp~Gra Cost ~money Dir. ~ ANY Goods ~ thing

5 levels of Neural Theory of Language Pyscholinguistic experiments Spatial Relation Motor Control Metaphor Grammar Cognition and Language Computation Structured Connectionism abstraction Neural Net and learning SHRUTI Triangle Nodes Computational Neurobiology Biology Neural Development Quiz Midterm Finals

Categories and concepts- introduction CS182/Ling109/CogSci110 Spring 2008

Lecture Outline Categories Aspects of a Neural Theory of concepts Basic Level Prototype Effects Neural Evidence for Category Structure Aspects of a Neural Theory of concepts Image Schemas Description and types Behavioral Experiment on Image Schemas Event Structure and Motor Schemas

Embodiment Alan Turing (Intelligent Machines,1948) Of all of these fields, the learning of languages would be the most impressive, since it is the most human of these activities. This field, however, seems to depend rather too much on the sense organs and locomotion to be feasible. Alan Turing (Intelligent Machines,1948)

The WCS Color Chips Basic color terms: Single word (not blue-green) Frequently used (not mauve) Refers primarily to colors (not lime) Applies to any object (not blonde)

Concepts What Concepts Are: Basic Constraints Concepts are the elements of reason, and constitute the meanings of words and linguistic expressions.

Concepts Are: Universal: they characterize all particular instances; e.g., the concept of grasping is the same no matter who the agent is or what the patient is or how it is done. Stable. Internally structured. Compositional. Inferential. They interact to give rise to inferences. Relational. They may be related by hyponymy, antonymy, etc. Meaningful. Not tied to the specific word forms used to express them.

Concepts: Traditional Theory The Traditional Theory Reason and language are what distinguish human beings from other animals. Concepts therefore use only human-specific brain mechanisms. Reason is separate from perception and action, and does not make direct use of the sensory-motor system. Concepts must be “disembodied” in this sense.

The neural theory Human concepts are embodied. Many concepts make direct use of sensory-motor, emotional, and social cognition capacities of our body-brain system. Many of these capacities are also present in non-human primates. Continuity Principle of Am. Pragmatists

Classical vs prototype model of categorization Classical model Category membership determined on basis of essential features Categories have clear boundaries Category features are binary Prototype model Features that frequently co-occur lead to establishment of category Categories are formed through experience with exemplars

Prototype theory Certain members of a category are prototypical – or instantiate the prototype Categories form around prototypes; new members added on basis of resemblance to prototype No requirement that a property or set of properties be shared by all members Features/attributes generally gradable Category membership a matter of degree Categories do not have clear boundaries

Prototype theory Certain members of a category are prototypical – or instantiate the prototype Category members are not all equal a robin is a prototypical bird, but we may not want to say it is the prototype, rather it instantiates (manifests) the prototype or ideal -- it exhibits many of the features that the abstract prototype does “It is conceivable that the prototype for dog will be unspecified for sex; yet each exemplar is necessarily either male or female.” (Taylor)

Prototype theory Categories form around prototypes; new members can be added on the basis of resemblance to the prototype Categories may also be extended on the basis of more peripheral features house for apartment

Prototype theory 3. No requirement that a property or set of properties be shared by all members -- no criterial attributes Category where a set of necessary and sufficient attributes can be found is the exception rather than the rule Labov household dishes experiment Necessary that cups be containers, not sufficient since many things are containers Cups can’t be defined by material used, shape, presence of handles or function

Prototype theory Wittgenstein’s examination of game Generally necessary that all games be amusing, not sufficient since many things are amusing Board games, ball games, card games, etc. have different objectives, call on different skills and motor routines - categories normally not definable in terms of necessary and sufficient features

Prototype theory What about mathematical categories like odd or even numbers? Aren’t these sharply defined? (Armstrong et al.) Subjects asked to assign numbers a degree of membership to the categories odd number or even number  3 had a high degree of membership, 447 and 91 had a lower degree (all were rated at least ‘moderately good’)

Categories - who decides? Embodied theory of meaning- categories are not pre-formed and waiting for us to behold them. Our need for categories drives what categories we will have Basic level categories - not all categories have equal status. The basic level category has demonstrably greater psychological significance.

Basic-level categories --Last week, Prof. Lakoff talked about prototypes, and the internal structure of individual categories and how they couldn’t just be described in terms of necessary and sufficient conditions. --Today, I’m going to talk a bit about how categories are related to each other, particularly in terms of hierarchical structure. -Some of the early work in this area was done by Brent Berlin, who studied categorization of plants in Tzeltal. He found that when referring to plants, people tended to name them at the level of the genus rather than the species (e.g. oak rather than live oak) or higher level (e.g. tree). And it was also found that Tzeltal children first learn plant names at the genus level. Rosch and others extended this work, looking at ordinary objects in our world. What they discovered is that not all of levels in a category hierarchy or taxonomy have equal “status”. Some levels are psychologically more basic than others. Specifically, they found that the middle part of the hierarchy was more basic. The categories at this level have been called basic-level categories.

easy chair rocking chair chair desk chair easy chair rocking chair furniture lamp desk lamp floor lamp table dining room table coffee table Superordinate Basic Subordinate So if you look at the hierarchical category structure for, say, furniture. At one end, have furniture, and at the other end, very specific types of furniture {read}. In between, in the middle level – e.g. chair, table, lamp – are basic-level categories The other levels can be named too: Superordinate – the level above the basic-level category, contains basic level categories Subordinate is the level below But in what respects are these basic-level categories considered psychologically or cognitively basic? [next slide]

Categories & Prototypes: Overview Furniture Superordinate Sofa Desk Basic-Level Category leather sofa fabric sofa L-shaped desk Reception disk Subordinate Three ways of examining the categories we form: relations between categories (e.g. basic-level category) internal category structure (e.g. radial category) instances of category members (e.g. prototypes)

Basic-level -- Criteria Perception – overall perceived shape single mental image fast identification Basic in four main respects (WFDT p. 47): The first way they are basic is in regards to perception Perception – The basic-level is the highest category level at which: category membership can be determined by overall perceived shape. A single mental image can represent the entire category,  applies to chair, but not to furniture So shape plays an important role… People can identify category members fastest at the basic level  

Basic-level -- Criteria Perception Function – motor program for interaction Another respect these are basic is in terms of functional interaction. e.g. sit on chair – changing posture, bending legs put things on tables, move things around on them turn lights on and off --No set motor routine for interacting with furniture, --And same motor routine for different subordinate types

Basic-level -- Criteria Perception Function Words – shortest first learned by children first to enter lexicon A third respect is in terms of communication. The words used to label these categories are READ

Basic-level -- Criteria Perception Function Communication Knowledge organization – most attributes are stored at this level A fourth respect is in terms of how knowledge is organized or stored. Superordinate level -- Very few attributes stored – furniture – usually bigger than a breadbox? Used inside? Basic level - most attributes –know a lot about things at this level -- Chairs have seats, legs, you sit on them, etc… Subordinate – few additional attributes (unless an expert) Can also think in terms of similarities and differences -Many more differences between basic-level categories than subordinate categories, e.g. chairs and lamps, vs. desk chairs and an easy chairs. -And there is more similarity within basic-level categories than within superordinate categories e.g. And other categories can be analysed this way -- CLASS SUGGESTIONS? NAME AN OBJECT

Basic-Level Category Perception: Function: What constitutes a basic-level category? Perception: similar overall perceived shape single mental image (gestalt perception) fast identification Function: general motor program Communication: shortest most commonly used contextually neutral first to be learned by children first to enter the lexicon Knowledge Organization: most attributes of category members stored at this level

Other Basic-level categories Objects Colors Motor-routines The criteria just listed seem to apply mostly to objects. But the notion of basic-level can be applied to other domains as well Basic color terms met many of these same criteria in terms of short words (red vs. vermillion), earliest learned, and fast identification. And similar things might be done with actions, e.g. basic motor actions might be ??? [walk, run – not saunter and sprint]

Concepts are not categorical

Mother The birth model The genetic model The nurturance model The person who gives birth is the mother The genetic model The female who contributes the genetic material is the mother The nurturance model The female adult who nurtures and raises a child is the mother of the child The marital model The wife of the father is the mother The genealogical model The closest female ancestor is the mother (WFDT Ch.4, p.74, p.83)

Radial Structure of Mother Genetic mother Stepmother Unwed mother Adoptive mother Central Case Surrogate mother Birth mother Biological mother Natural mother Foster mother The radial structure of this category is defined with respect to the different models

Marriage What is a marriage? What are the frames (or models) that go into defining a marriage? What are prototypes of marriage? What metaphors do we use to talk about marriages? Why is this a contested concept right now?

Concepts and radial categories Concepts can get to be the "prototype" of their category in various ways. Central subcategory (others relate to this) Amble and swagger relate to WALK Shove relates to PUSH Essential (meets a folk definition: birds have feathers, beaks, lay eggs) Move involves change of location. Typical case (most are like this: "sparrow") Going to a conference involves air travel. Ideal/anti-ideal case (positive social standard: "parent"); anti-ideal case (negative social standard: "terrorist") Stereotype (set of attributes assumed in a culture: "Arab") Salient exemplar (individual chosen as example)

Category Structure Classical Category: Radial Category: necessary and sufficient conditions Radial Category: a central member branching out to less-central and non-central cases degrees of membership, with extendable boundary Family Resemblance: every family member looks like some other family member(s) there is no one property common across all members (e.g. polysemy) Prototype-Based Category Essentially-Contested Category (Gallie, 1956) (e.g. democracy) Ad-hoc Category (e.g. things you can fit inside a shopping bag) My original example for an ad-hoc category was “things you’ll bring to a picnic”, but a keen student pointed out that it is in fact a frame-based category (the picnic frame).

Prototype Cognitive reference point Social stereotypes standards of comparison Social stereotypes snap judgments defines cultural expectations challengeable Typical case prototypes default expectation often used unconsciously in reasoning Ideal case / Nightmare case e.g. ideal vacation can be abstract may be neither typical nor stereotypical Paragons / Anti-paragons an individual member that exhibits the ideal Salient examples e.g. 9/11 – terrorism act Generators central member + rules e.g. natural number = single-digit numbers + arithmetic

Neural Evidence for category structure Are there specific regions in the brain to recognize/reason with specific categories?

Category Naming and Deficits People with brain injury have selective deficits in their knowledge of categories. Some patients are unable to identify or name man made objects and others may not be able to identify or name natural kinds (like animals)

A PET Study on categories (Nature 1996)

Study 16 adults (8M, 8F) participated in a PET (positron emission tomography) study. Involves injecting subject with a positron emitting radioactive substance (dye) Regions with more metabolic activity will absorb more of the substance and thus emit more positrons Positron-electron collisions yield gamma rays, which are detected Increased rCBF (regional changes in cerebral blood flow) was measured When subjects viewed line drawings of animals and tools.

The experiment Subjects looked at pictures of animals and tools and named them silently. They also looked at noise patterns (baseline 1) And novel nonsense objects (baseline 2) Each stimulus was presented for 180ms followed by a fixation cross of 1820 ms. Drawings were controlled for name frequency and category typicality

medial lateral

Left middle temporal gyrus ACC Premotor

Calcarine Sulcus

Conclusions Both animal and tool naming activate the ventral temporal lobe region. Tools differentially activate the ACC, pre-motor and left middle temporal region (known to be related to processing action words). Naming animals differentially activated left medial occipital lobe (early visual processing) The object categories appear to be in a distributed circuit that involves activating different salient aspects of the category.