1. Understanding Action Verbs- Embodied Verbal Semantics Approach Pavan Kumar Srungaram M.Phil Cognitive Science (09CCHL02) 2009-10 Supervisor: Prof. Bapi Raju, S. University of Hyderabad

2. Language in Two Different Perspectives Chomskyan view: Chomskyan view: Based on structures and rules. Based on structures and rules. Syntax is autonomous free of meaning and general cognition Syntax is autonomous free of meaning and general cognition Human language is an abstract formal system Human language is an abstract formal system

3. Problems In many cases it was observed that background knowledge, concept, meaning have come into the rules governing syntax. In many cases it was observed that background knowledge, concept, meaning have come into the rules governing syntax.

4. Language as a part of General Cognition Lakoff, Fauconnier – Language not to be studied in terms of structure/ Grammar but the way it is used Lakoff, Fauconnier – Language not to be studied in terms of structure/ Grammar but the way it is used Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it. Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it. Metaphors We Live By ! Metaphors We Live By !

5. Embodiment Acknowledge the role of body and its sensorimotor processes can and do play in cognition. Acknowledge the role of body and its sensorimotor processes can and do play in cognition. Understanding cognition in the context of biological function to support the activities of the body. Understanding cognition in the context of biological function to support the activities of the body. Cognition as real-time situated activity, inseparable from perception and action. Cognition as real-time situated activity, inseparable from perception and action.

6. Behavioural Tests Considered the experiments by Bergen et al. (2010) and tried to replicate three of those tests in Telugu. Considered the experiments by Bergen et al. (2010) and tried to replicate three of those tests in Telugu. In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific

7. Behavioural Experiment There are 4 tasks in this experiment There are 4 tasks in this experiment Each subject participates only in one task Each subject participates only in one task Task 1: Image-Verb Matching Task Task 1: Image-Verb Matching Task Task 2: Verb-Image Matching Task Task 2: Verb-Image Matching Task Task 3: Verb-Verb Matching Task Task 3: Verb-Verb Matching Task Task 4: Image-Verb (auditory) Matching Task Task 4: Image-Verb (auditory) Matching Task

8. Hypothesis If verb comprehension requires the activation of the motor areas related to the same effector with which the action is performed. If verb comprehension requires the activation of the motor areas related to the same effector with which the action is performed. Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions

9. Materials and Methods Computer Based Test Computer Based Test Two Stimuli presented at specific times Two Stimuli presented at specific times Subjects are to judge as quickly as possible whether the two stimuli match Subjects are to judge as quickly as possible whether the two stimuli match The response times of the subjects are recorded and analyzed The response times of the subjects are recorded and analyzed

10. Materials and Methods The stimuli used in the tasks are: The stimuli used in the tasks are: a) Images: These contain a single character performing some action a) Images: These contain a single character performing some action b) Action Verbs written (or) typed both in program compatible formats. b) Action Verbs written (or) typed both in program compatible formats. c) Action Verbs (audio) c) Action Verbs (audio) All these three are recorded and used in program compatible formats. All these three are recorded and used in program compatible formats.

11. Algorithm Algorithm First Stimuli: Presented for 1000 milliseconds First Stimuli: Presented for 1000 milliseconds Inter stimulus interval: 500 milliseconds a) 450 milliseconds: Visual Mask Inter stimulus interval: 500 milliseconds a) 450 milliseconds: Visual Mask b) 50 milliseconds: Blank b) 50 milliseconds: Blank Second Stimuli presentation: This is until the subject makes his decision by pressing the designated key. Second Stimuli presentation: This is until the subject makes his decision by pressing the designated key.

12. Presentation of Stimuli

14. Task 1: Image-Verb Matching Task Image is shown for a period of 1000 milliseconds Image is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in typed format This is followed by a verb in typed format Show images and verbs! Show images and verbs!

15. Results Results The observation is in accordance with the hypothesis. The observation is in accordance with the hypothesis. Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition. Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition.

16. Means Table Means Table Condition Average RT in millisecondsStandard Deviation Matching Condition 1281.34444.4169 Non-Matching Same effector Condition 1502.599431.3717 Non-Matching Different effector Condition 1394.467386.6818

17. ANOVA: Analysis of Variance ANOVA: Analysis of Variance RT Sum of SquaresdfMean SquareFSig. Between Groups1175122.4032587561.2023.306.040 Within Groups2.506E7141177703.579 Total2.623E7143

18. Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

19. Results The results obtained in the test provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action. The results obtained in the test provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action.

21. Alternative explanation An alternative explanation for the observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection An alternative explanation for the observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection

22. Task 2:Verb-Image Matching Task Verb in typed format is shown for a period of 1000 milliseconds Verb in typed format is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by an image This is followed by an image

23. Means Table Means Table Condition Average RT (in milliseconds)Standard Deviation Matching Condition 1020.568374.1628 Non-Matching Same Effector Condition 1176.36492.6629 Non-Matching Different Effector Condition 1111.11 529.8168

24. Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

25. ANOVA ANOVA RT Sum of Squares dfMean SquareFSig. Between Groups 612150.8292306075.4151.384.254 Within Groups3.251E7147221140.118 Total3.312E7149

27. Results As observed in task, the subjects took longer time to respond when the verb and image share the same effector. As observed in task, the subjects took longer time to respond when the verb and image share the same effector. Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed. Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed.

28. Task 3: Verb-Verb Matching Task Verb in typed format is shown for a period of 1000 milliseconds Verb in typed format is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in typed format This is followed by a verb in typed format

29. Means Table Means Table Condition Average RT (in milliseconds) Standard Deviation Matching Condition 1353.48 351.0843 Non-Matching Same Effector Condition 1419.04 396.6966 Non-Matching Different Effector Condition 1247.92 276.2357

30. ANOVA ANOVA RT Sum of Squares dfMean SquareFSig. Between Groups 795980.9912397990.4963.351.038 Within Groups1.889E7159118782.609 Total1.968E7161

31. Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

33. Results When the images were replaced by verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed When the images were replaced by verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed

34. Question In all the above tests, the presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality In all the above tests, the presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality

35. Task 4: Image-Verb (Auditory) Matching Task Image is shown for a period of 1000 milliseconds Image is shown for a period of 1000 milliseconds 500 milliseconds inter stimulus interval 500 milliseconds inter stimulus interval This is followed by a verb in auditory mode. This is followed by a verb in auditory mode.

36. Means Table Means Table Condition Average RT (in milliseconds) Standard Deviation Matching Condition 438.85138.5288 Non-Matching Same 508.95 219.089 Non-Matching Different 440.08 144.3107

37. ANOVA ANOVA RT Sum of SquaresdfMean SquareFSig. Between Groups 128683.860264341.9302.213.114 Within Groups3575614.99712329070.041 Total3704298.858125

38. Condition 1: Matching Condition Condition 2: Non-Matching Same effector Condition Condition 3: Non-Matching Different effector Condition

40. Result It is evident from the table that RT in the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant. It is evident from the table that RT in the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant..

41. Summary ConditionTask 1Task 2Task 3Task 4 Matching Condition 1281.34 1020.5681353.48438.85 Non-Matching same effector 1502.599 1176.361419.04508.95 Non-Matching different effector 1394.467 1111.111247.92440.08

42. Summary We observe that irrespective of statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non- matching different effector condition. We observe that irrespective of statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non- matching different effector condition.

43. Summary Task TI: Picture- Lexical Matching Task T2: Lexical- Picture Matching Task T3: Lexical- Lexical Matching Task T4: Picture- Lexical (Auditory) Matching Task Matching condition 7.45%9.4%11%7.90% Non matching same effector condition 20.90%19.55%18%20.17% Non matching different effector condition 6.12%8.6%7.33%6.29% Average error responses per Subject. 11.49%9.71%12.11%11.45%

44. Summary The higher rate of errors in the non- matching same effector conditions in all the tasks is to be tested and given an explanation. The higher rate of errors in the non- matching same effector conditions in all the tasks is to be tested and given an explanation.

45. Conclusion The results obtained in the various tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect. The results obtained in the various tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect.

46. Thank you Thank you