Hand trajectories reveal hidden cognitive states Ken Nakayama a Joo-Hyun Song b Matthew Finkbeiner c Alfonso Caramazza a a Harvard University b Smith-Kettlewell Eye Research Institute c Macquarie University

today’s talk Suggest there are new opportunities to study a wide range of internal cognitive states from the detailed study of motor behavior (Spivey et al., for language processing)

Remarkable power of psychophysical and behavioral methods Some examples Wertheimer, 1912 Hecht, Shlaer and Pirenne, 1943 Julesz 1960s Sperling 1960s

Early processing Later processing Cognition/decision Motor plan input

Early processing Later processing Cognition/decision Motor plan input fMRI EEG/MEG TMS

Early processing Later processing Cognition/decision Motor plan input Reaching Measure RT And accuracy PLUS Full details of trajectory Reveals otherwise hidden states

. Hand movement recording: 120Hz with Polhemus Fastrak Polhemus system: Sample rate (120 Hz) Recording 3-D movement of finger tip Visually-guided pointing Song and Nakayama (VSS 05,06)

I. Early hand trajectories reflect an underlying number representation –Song and Nakayama, Cognition (in press) II. pointing trajectories reveal influence of unseen words –Finkbeiner, Song, Nakayama, and Caramazza, Visual Cognition (in press) Two recent studies

Mental number line

IF X>5 respond “greater” Else respond “lesser” if digital number line BUT, RT and error rates systematically decrease as the numerical distance between two numbers increases. Moyer and Landauer (1967) Supporting “ Analog ” number line X In numeric comparison task3 vs. 5, 13 vs. 5 Same response to 3 vs. 5 &13 vs. 5 Numeric distance effect

Characteristics of number representations are mainly examined with discrete responses such as reaction time and accuracy. In the current study, to map invisible internal cognitive processes of numeric comparison in spatial domain over time, we measured trajectories.

X-Y trajectories Systematic shift of initial trajectories towards a hypothetical position on a number line intermediate between the numeral 1 and 5 positions

Mean trajectory deviation ≈ 480 ms after target onset ≈ 150 ms after the movement onset

Endpoint deviation

Continuous internal processes for number representations Early detection of mental numeric line than previous tasks Kinematic measurements: More reliable and robust with small numbers of subjects

I. Early hand trajectories reflect an underlying number representation –Song and Nakayama, Cognition (in press) II. Pointing trajectories reveal influence of unseen words –Finkbeiner, Song, Nakayama, and Caramazza, Visual Cognition (in press)

linkage between invisible text and actions? Goodale and Milner Dorsal Vs Ventral

spinach Point to color of the word ######## boy KMDKS spinach 500 ms 30ms 10 ms Until Response ######## red KMDKS spinach ######## green KMDKS spinach neutral incongruent congruent

######## boy KMDKS spinach 500 ms 30ms 10 ms Until Response ######## red KMDKS spinach ######## green KMDKS spinach neutral incongruent congruent

msec after movement onset Time after movement initiation Horizontal movement (in) incongruent congruent neutral

Replication with varying prime duration

Despite the unavailability of the invisible stimulus to visual awareness, the congruity of the prime stimulus with the target stimulus had a significant effect on participants’ pointing trajectories. Masked unseen word processing extends down to include the formulation of overt motor responses

Early processing Later processing Cognition/decision Motor plan input Reaching Full details of trajectory LEAKAGE

Suggest there are new opportunities to study a wide range of internal cognitive states from the detailed study of motor behavior