Understanding Action Verbs- Embodied Verbal Semantics Approach Pavan Kumar Srungaram M.Phil Cognitive Science (09CCHL02) Supervisor: Prof. Bapi Raju, S. University of Hyderabad

Language in Two Different Perspectives Chomskyan view: Chomskyan view: Based on structures and rules. Based on structures and rules. Syntax is autonomous free of meaning and general cognition Syntax is autonomous free of meaning and general cognition Human language is an abstract formal system Human language is an abstract formal system

Problems In many cases it was observed that background knowledge, concept, meaning have come into the rules governing syntax. In many cases it was observed that background knowledge, concept, meaning have come into the rules governing syntax.

Language as a part of General Cognition Lakoff, Fauconnier – Language not to be studied in terms of structure/ Grammar but the way it is used Lakoff, Fauconnier – Language not to be studied in terms of structure/ Grammar but the way it is used Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it. Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it. Metaphors We Live By ! Metaphors We Live By !

Embodiment Acknowledge the role of body and its sensorimotor processes can and do play in cognition. Acknowledge the role of body and its sensorimotor processes can and do play in cognition. Understanding cognition in the context of biological function to support the activities of the body. Understanding cognition in the context of biological function to support the activities of the body. Cognition as real-time situated activity, inseparable from perception and action. Cognition as real-time situated activity, inseparable from perception and action.

Behavioural Tests Considered the experiments by Bergen et al. (2010) and tried to replicate three of those tests in Telugu. Considered the experiments by Bergen et al. (2010) and tried to replicate three of those tests in Telugu. In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific

Behavioural Experiment There are 4 tasks in this experiment There are 4 tasks in this experiment Each subject participates only in one task Each subject participates only in one task Task 1: Image-Verb Matching Task Task 1: Image-Verb Matching Task Task 2: Verb-Image Matching Task Task 2: Verb-Image Matching Task Task 3: Verb-Verb Matching Task Task 3: Verb-Verb Matching Task Task 4: Image-Verb (auditory) Matching Task Task 4: Image-Verb (auditory) Matching Task

Hypothesis If verb comprehension requires the activation of the motor areas related to the same effector with which the action is performed. If verb comprehension requires the activation of the motor areas related to the same effector with which the action is performed. Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions

Materials and Methods Computer Based Test Computer Based Test Two Stimuli presented at specific times Two Stimuli presented at specific times Subjects are to judge as quickly as possible whether the two stimuli match Subjects are to judge as quickly as possible whether the two stimuli match The response times of the subjects are recorded and analyzed The response times of the subjects are recorded and analyzed

Materials and Methods The stimuli used in the tasks are: The stimuli used in the tasks are: a) Images: These contain a single character performing some action a) Images: These contain a single character performing some action b) Action Verbs written (or) typed both in program compatible formats. b) Action Verbs written (or) typed both in program compatible formats. c) Action Verbs (audio) c) Action Verbs (audio) All these three are recorded and used in program compatible formats. All these three are recorded and used in program compatible formats.

Algorithm Algorithm First Stimuli: Presented for 1000 milliseconds First Stimuli: Presented for 1000 milliseconds Inter stimulus interval: 500 milliseconds a) 450 milliseconds: Visual Mask Inter stimulus interval: 500 milliseconds a) 450 milliseconds: Visual Mask b) 50 milliseconds: Blank b) 50 milliseconds: Blank Second Stimuli presentation: This is until the subject makes his decision by pressing the designated key. Second Stimuli presentation: This is until the subject makes his decision by pressing the designated key.

Presentation of Stimuli

Task 1: Image-Verb Matching Task Image is shown for a period of 1000 milliseconds Image is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in typed format This is followed by a verb in typed format Show images and verbs! Show images and verbs!

Results Results The observation is in accordance with the hypothesis. The observation is in accordance with the hypothesis. Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition. Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition.

Means Table Means Table Condition Average RT in millisecondsStandard Deviation Matching Condition Non-Matching Same effector Condition Non-Matching Different effector Condition

ANOVA: Analysis of Variance ANOVA: Analysis of Variance RT Sum of SquaresdfMean SquareFSig. Between Groups Within Groups2.506E Total2.623E7143

Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

Results The results obtained in the test provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action. The results obtained in the test provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action.

Alternative explanation An alternative explanation for the observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection An alternative explanation for the observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection

Task 2:Verb-Image Matching Task Verb in typed format is shown for a period of 1000 milliseconds Verb in typed format is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by an image This is followed by an image

Means Table Means Table Condition Average RT (in milliseconds)Standard Deviation Matching Condition Non-Matching Same Effector Condition Non-Matching Different Effector Condition

Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

ANOVA ANOVA RT Sum of Squares dfMean SquareFSig. Between Groups Within Groups3.251E Total3.312E7149

Results As observed in task, the subjects took longer time to respond when the verb and image share the same effector. As observed in task, the subjects took longer time to respond when the verb and image share the same effector. Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed. Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed.

Task 3: Verb-Verb Matching Task Verb in typed format is shown for a period of 1000 milliseconds Verb in typed format is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in typed format This is followed by a verb in typed format

Means Table Means Table Condition Average RT (in milliseconds) Standard Deviation Matching Condition Non-Matching Same Effector Condition Non-Matching Different Effector Condition

ANOVA ANOVA RT Sum of Squares dfMean SquareFSig. Between Groups Within Groups1.889E Total1.968E7161

Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

Results When the images were replaced by verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed When the images were replaced by verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed

Question In all the above tests, the presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality In all the above tests, the presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality

Task 4: Image-Verb (Auditory) Matching Task Image is shown for a period of 1000 milliseconds Image is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in auditory mode. This is followed by a verb in auditory mode.

Means Table Means Table Condition Average RT (in milliseconds) Standard Deviation Matching Condition Non-Matching Same Non-Matching Different

ANOVA ANOVA RT Sum of SquaresdfMean SquareFSig. Between Groups Within Groups Total

Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

Result It is evident from the table that RT in the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant. It is evident from the table that RT in the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant..

Summary ConditionTask 1Task 2Task 3Task 4 Matching Condition Non-Matching same effector Non-Matching different effector

Summary We observe that irrespective of statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non- matching different effector condition. We observe that irrespective of statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non- matching different effector condition.

Summary Task TI: Picture- Lexical Matching Task T2: Lexical- Picture Matching Task T3: Lexical- Lexical Matching Task T4: Picture- Lexical (Auditory) Matching Task Matching condition 7.45%9.4%11%7.90% Non matching same effector condition 20.90%19.55%18%20.17% Non matching different effector condition 6.12%8.6%7.33%6.29% Average error responses per Subject %9.71%12.11%11.45%

Summary The higher rate of errors in the non- matching same effector conditions in all the tasks is to be tested and given an explanation. The higher rate of errors in the non- matching same effector conditions in all the tasks is to be tested and given an explanation.

Conclusion The results obtained in the various tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect. The results obtained in the various tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect.

Thank you Thank you