How language incorporates affordances Lucia Riggio Department of Neuroscience University of Parma Cambridge 3 September 2013

Affordances refer to properties of an object or to specific parts of it (typically the handle) that trigger components of actions such as reaching or grasping Visual perception of objects activates the very system responsible for their actual manipulation, that transforms object pragmatic features into appropriate motor programs for their interaction Affordances

Handle graspability [ Buccino et al 2009 ] Visual presentation of manipulable objects in which the most important feature relevant for action is violated (e.g., a cup with a broken handle) prevents motor activation from occurring.

Object graspability [ Taylor & Zwaan 2010] Dynamic responses (i.e. the grip force) to a set of spheres with increasing size were no longer correlated with the apparent weight of the stimuli, if the experimenter referred to them as planets, rather than graspable solids

Language Embodied theories predict that nouns of graspable objects should also activate affordances since understanding of language is achieved by recruiting the same sensorimotor systems activated when we experience the action or object a word refers to.

Presentation of action verbs associated with different effectors results in somatotopic activation of motor areas (e.g. Pulvermüller et al 2001; Hauk et al 2004; Tettamanti et al 2005) Modulation in the MEP for a muscle of a given effector associated with the verb expressed in a sentence (e.g. Buccino et al 2005) Modulation of RTs when the effector used to respond is also involved in the action expressed by the linguistic material (e.g. Buccino et al 2005; Boulenger et al 2006; Sato et al 2008; Dalla Volta et al 2009) or when the response requires the execution of a movement in the same direction of the action described by the sentence (e.g., Glenberg & Kaschak 2002) Verbs

Verbs : interference and facilitation Interference effects within ms after the presentation of action-related word: slowing–down of the RTs and decrease of the MEPs when either the same effector as that expressed by the action verb or sentence (e.g. Sato et al 2008; Buccino et al 2005) or the same directional movement are involved (de Vega et al 2013). Facilitation effects in the matching conditions for longer time intervals (e.g. Glenberg & Kaschak 2002; Zwaan & Taylor 2006; de Vega et al 2013).

Nouns Is the early involvement of the motor system, observed during the understanding of action-related sentences and verbs, also present for nouns?

Nouns: Materials Hand-related nouns Foot-related nouns Abstract nouns Forbici (scissors)Pedale (pedal)Tattica (tactics) Tazzina (cup)Pantofola (slipper)Superbia (arrogance) Spazzola (brush)Pedana (footboard)Ritegno (reluctance) Forchetta (fork)Pattini (skate)Elogio (praise) Pettine (comb)Gradino (step)Deferenza (deference) Matita (pencil)Mocassino (moccasin)Gelosia (jealousy) Accendino (lighter)Scalinata (staircase)Lusinga (allurement) Martello (hammer)Ciabatta (slipper)Inganno (deceit) [Marino et al 2013]

Nouns: behavioural experiment

Nouns: TMS experiment

*

Nouns: conclusions Both experiments indicate the early involvement of the left motor cortex in the representation of nouns of graspable objects, in line with a causal role of the motor system in language understanding. This modulation is comparable to that previously found with verbs or action related sentences (e.g. Hauk & Pulvermüller 2004; Buccino et al 2005; Boulenger et al 2006; Sato et al 2008). The differential pattern of results between the two hemispheres is consistent with the dominance of the left hemisphere for language and tool processing.

Nouns, as their external referents, elicit motor information.

[Borghi & Riggio 2009] Stable and variable affordances Stable affordances: stable aspects of the interaction with objects, which can be included in an object representation stored in memory Variable affordances: variable aspects, which change depending on the way the object is presented Canonical affordances: variable aspects associated with a canonical interaction The comprehension of words referring to a particular object activates only stable and canonical affordances.

Objects and nouns in space The activation of motor information associated with visual objects is spatially constrained (e.g. Costantini et al 2010; 2011) Is motor information elicited by nouns spatially constrained?

[Ferri et al 2011] Task: Participants had to signal whether an object, presented either within or outside their reachable space, was natural or manufactured by making reach-to-precision or reach-to-power grasp responses. Experiment 1: objects Reach-to-grasp response device Stimuli were presented in a 3-D room created by means a 3-D Studio Max TM

Experiment 2: nouns Task: Participants had to signal whether the object (near or far) was the same (congruent) as that indicated by the word, or different, producing reach-to-precision or reach-to- power grasp responses.

An action compatibility effect between the response and the grip evoked by the object, only in the reaching space and only for artefacts Experiment 1: objectsExperiment 2: nouns An action compatibility effect between the response and the grip evoked by the noun that was not modulated by the spatial position of the visual object Variable features such as distance and location are not integrated in the motor representation of the objects elicited by language. Results

Conclusions Action information relative to the size of an object is spatially constrained for visual objects but not for their nouns. Space information elicited by a seen object is different from that elicited by nouns.

Examples of stimuli [ Yoon et al 2010 ] Converging results Task: Right-handed participants had to decide whether the paired objects would typically be used together. Results: Decision times faster when stimuli were in the correct locations for a right-hand action (active member on the right) only when objects were presented.

Do concrete verbs and nouns combined in single sentences have a different weight on the modulation of the motor system? Combinations of nouns and verbs

Material and Procedure Sensible and non-sensible sentences formed by concrete verbs and nouns were presented in a sensibility judgment task. Verbs could express an action that could be performed in a specific way (low degrees of freedom) or in very many different ways (high degrees of freedom). Nouns referred to graspable and non-graspable objects. Sensible Non-sensible To squeeze the orangeTo water the handle VLDoFNoun of graspable object To recycle the bottleTo consult the onion To water the flowerbedTo squeeze the sunset VHDoF Noun of non-grasp. object To consult the archives To recycle the herd Sensorimotor specificity of verbs and nouns [Marino et al 2012]

Results With Verbs with LDoF performance is independent on the noun type: activation of specific features of objects on which the action is usually performed With Verbs with HDoF performance is affected by the noun type

The motor system is modulated by concrete nouns in a way comparable to concrete verbs Nouns seem to activate only stable and canonical affordances. Distance and location, related to the reaching of the object, seem not to be represented. When nouns are combined with verbs the sensorimotor specificity of both words could have a role in the motor activation. In short