1. 1 Visually-guided Motor Actions in Humans Ken Nakayama and Joo-Hyun Song Harvard University VISION SCIENCES LABORATORY

2. 2 Actions: Interaction with The External World

3. 3. Discrete responses Continuous interaction Actions: Read Out Cognitive Processing Visually-guided actions: Instantaneously read out internal processes Yet, the importance of motor actions is overlooked

4. 4 Cognitive Theories in Human Behaviors Perception Cognitive decision Action Issue of motor control - Limited to post-cognitive decision-making - Mere reflection of completed cognitive decisions

5. 5 Target without distractorsTarget among distractors Visually-Guided Manual-Pointing Task + Single target Easy Task + Odd-color target Difficult Task

6. 6. Hand movement recording: 120Hz with Polhemus Fastrak Visually-Guided Manual-Pointing Task

7. 7 Outline Part 1: Role of focal attention on latencies and trajectories of manual pointing Part 2: Concurrent processing of manual pointing to competing stimuli Part 3: Automatic adjustment of visuo- motor readiness

8. 8 Outline Part 1: Role of focal attention on latencies and trajectories of manual pointing Part 2: Concurrent processing of manual pointing to competing stimuli Part 3: Automatic adjustment of visuo- motor readiness

9. 9 Visual Attention: Selection for Perception & Action

10. 10 Broadly distributed attention for entire displays is sufficient to detect and localize the target Distributed vs. Focused Attention Focused attention needs to be allocated to the target for feature discrimination ( Atkinson & Braddick, 1989; Folk & Egeth, 1989; Johnson & Pashler, 1990; Sagi & Julesz, 1985 a, b; Green, 1992)

11. 11 Diagnostic Visual Search Paradigm Mixed Condition Target:RedGreenRed + + + Blocked Condition Red Target: +++ (Bravo & Nakayama, 1992) Number of distractors Reaction Times Detection(distributed) Mixed Blocked Discrimination (focused) Mixed Blocked Saccadic tasks : No subtle discrimination required (McPeek et al., 1999)

12. 12 Role of Focused Attention on Latencies & Trajectories of Manual Pointing Number of distractors Reaction Times Detection (distributed) Mixed Blocked Discrimination (focused) Mixed Blocked + ++ B. Blocked Condition + ++ Green A. Mixed Condition Green Red Full Movement trajectories over the time course : Interaction between cognitive-decision making for the correct target & motor control

13. 13 Reaction Time Indexes (Initiation) Latency Movement Duration Total Time + Stimulus onset Finger lift-off Target touch Time

14. 14. Total Time (ms) 7 Subjects Exp 1: Effect of Perceptual Grouping of Distractors 680 700 720 740 760 25811 Number of distractors Mixed Blocked 280 300 320 340 25811 Number of distractors Mixed Blocked Initial latency (ms) 380 400 420 440 460 25811 Number of distractors Mixed Blocked Movement duration (ms)

15. 15. Exp 1: Effect of Perceptual Grouping of Distractors

16. 16 Exp 1: Effect of Perceptual Grouping of Distractors Initially toward distractors & corrected to the target Size & frequency of curved trajectories are correlated with the strength of target-distractor competition (McPeek, Han & Keller, 2003).

17. 17 Max Curvature=Max(D)/L -5 0 5 0 5 10 0 2 4 6 8 10 12 Forward Left Right D L Exp 1: Effect of Perceptual Grouping of Distractors

18. 18 680 700 720 740 760 25811 Number of distractors Mixed Blocked. Position within Same Color Sequence... 1 8 RT 5 Target: (Maljkovic & Nakayama, 1994; Bichot & Schall, 1999; McPeek et al., 1999) Exp 2: Effect of Perceptual Priming Total Time Attended target feature repetition

19. 19. Exp 2: Effect of Perceptual Priming 650 675 700 725 750 775 01234567 Order in sequence Blocked Baseline 7 Subjects Total time (ms) Order in sequence 400 425 450 475 500 01234567 Blocked Baseline Movement duration Order in sequence 240 245 250 255 260 265 270 01234567 Blocked Baseline Initial latency

20. 20. Maximum curvature Exp 2: Effect of Perceptual Priming 0.05 0.075 0.1 0.125 0.15 0.175 01234567 Order in sequence Maximum curvature Blocked Baseline

21. 21 Discussion Focused attention is closely associated with manual-pointing process though subtle discrimination is not required. Trajectory curvature is reduced as decreasing competition between target and distractors by perceptual grouping and cumulative priming Interactive processes between target selection and motor control may challenge assumption of serial processing in traditional cognitive theories

22. 22 Outline Part 1: Role of focal attention on latencies and trajectories of manual pointing Part 2: Concurrent processing of manual pointing to competing stimuli Part 3: Automatic adjustment of visuo- motor readiness

23. 23 Assumption of Serial Processing & Optimal Planning Perception Cognitive decision Action A single movement is planned & executed serially at a time An optimal trajectory is planned Straight trajectories Bell-shaped velocity profile ( Flash & Hogan, 1985; Uno, Kawato, & Suzuki, 1989; Engelbrecht, 2001)

24. 24.. Reaching trajectories swerves away from distractors ( Tipper, Lortie, & Baylis, 1992; Tipper, Howard, & Houghton, 1998, 2000) Dorsal premotor areas encode multiple action plans concurrently, even before cognitive decision is made (Cisek & Kalaska, 2002, 2005) Multiple Reaching Plans For potential Targets

25. 25 + Odd-color target Curved Trajectories Toward Distractors Interaction between cognitive decision making & motor planning Not a single optimal straight trajectory

26. 26 Are they concurrently planned? Do we execute every hand movement after target selection is finalized? Or, do we sometimes initiate hand movements before completing target selection? Can movements to competing stimuli be concurrently processed? When selecting a target among multiple distractors… + Odd-color target

27. 27 Concurrent Processing of Saccades Target Initial saccade Corrective saccade (McPeek, Skavenski, & Nakayama, 2000)

28. 28 Concurrent Processing of Visually- Guided Manual Pointing vs. Tiny mass/inertia High mass/inertia Hand/arm

29. 29 + Trial 1: Odd-color target Main Experiment + Trial 2: Single target (Baseline) Trial 3 Trial 4 + +..

30. 30.. Down Left Right Up Single target trials Movement Trajectories:Single Target

31. 31.. Down Left Right Up Movement Trajectories:Single Target Single target trials

32. 32.. Down Left Right Up Movement Trajectories:Single Target Single target trials

33. 33.. Odd-color target trials Down Left Right Up Movement Trajectories:Odd-Color Target Single target trials

34. 34.. Down Left Right Up Movement Trajectories:Odd-Color Target Odd-color target trials Single target trials

35. 35.. Down Left Right Up Movement Trajectories:Odd-Color Target Odd-color target trials Single target trials

36. 36.. Down Left Right Up Odd-color target trials Single target trials Movement Trajectories:Odd-Color Target

37. 37 Are Initial & Corrective Movements Planned Concurrently ? Curved trajectories - I nfluences of competing distractors on planning & execution of pointing movements Are initial & corrective movements concurrently planned, overlapping in time? Difficulty of decomposing curved trajectories into discrete two movements

38. 38 Estimate of Corrective Movement Onset Average Baseline Trajectory Time (ms) Position(inches) 04080120160200240280320360 -5 -4 -3 -2 0 1 2 3 4 5 Left Center Right -1.5SD +1.5SD Initial Movement Corrective Movement Stimulus Onset + Initial latency Corrective latency

39. 39 Corrective Latency vs. Initial Latency 8 subjects

40. 40 Difference Between Corrective & Initial Latencies 8 subjects

41. 41 Time Cost Of Executing two Concurrently Planned Movements Velocity along the curve 500 1000 0 10 20 30 40 Time(ms) CurvedStraight Curved Odd-Color + Non-optimal properties

42. 42 Time Cost Of Executing two Concurrently Planned Movements 8 subjects

43. 43 Displaced target Initial target ~150-250 ms + + + Fixation Time 35% Trials: Displaced 65% Trials: Baseline Time + + Fixation Control Experiment:Double-Step Task

44. 44 Control Experiment:Double-Step Task

45. 45 Discussion The cost of planning corrective movements may be negligible because these can be planned in parallel with initial movements. Like saccades, pointing movements to competing stimuli can be concurrently processed. Hand movements are sometimes initiated before cognitive decision-making for the correct target is completed. The visuo-motor system is less concerned with planning a single optimal trajectory as long as the ultimate talk goal is achieved

46. 46 Outline Part 1: Role of focal attention on latencies and trajectories of manual pointing Part 2: Concurrent processing of manual pointing to competing stimuli Part 3: Automatic adjustment of visuo- motor readiness

47. 47.. Down Left Right Up Single Target Fast initiation & straight trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Single target Easy Task

48. 48.. Single Target Down Left Right Up Fast initiation & straight trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Single target Easy Task

49. 49.. Single Target Down Left Right Up Fast initiation & straight trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Single target Easy Task

50. 50.. Odd-Color Target Slow initiation & straight trajectory Fast initiation but curved trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Odd-color target Difficult Task

51. 51.. Odd-Color Target Slow initiation & straight trajectory Fast initiation but curved trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Odd-color target Difficult Task

52. 52.. Odd-Color Target Slow initiation & straight trajectory Fast initiation but curved trajectory Influence of Trial Difficulty on Manual Pointing Strategies + Odd-color target Difficult Task

53. 53.. How does intermixing trials of varying difficulty influence latency & accuracy criteria of visually-guided manual pointing? What mechanism adjusts latency & accuracy criteria in visually-guided manual-pointing tasks? How does the time course of latency & accuracy criteria adjust? Questions

54. 54 Predictability Knowing upcoming trial type may lead to initiation latency & accuracy adjustment Trial Type Repetition Continuous repetition of the same type trials may lead to initiation latency & accuracy adjustment Initiation latency & accuracy criteria for trials of varying difficulty may be influenced by either … Predictability vs. Repetition

55. 55 Alternating (Odd-color) Mixed + + + +... + + + + (Single) + + + + + + + +... Exp1. Dissociation of Predictability & Past Experience To compare influences of predictability and past trial type repetition on initiation latency & accuracy adjustment Blocked

56. 56 Exp1. Dissociation of Predictability & Past Experience Prediction for Adjustment Cognitive knowledge of the next trial but frequent trial type switch

57. 57 200 250 300 350 400 BlockedMixedAlternated Single target Odd color target N=9 Initiation Latency √ Adjustment of Initiation Latency Criteria Alternating Initiation latency(ms) Not simply due to task switching No difference in error rates ( over 98%)

58. 58 Movement duration √ Adjustment of Initiation Latency Criteria

59. 59 Adjustment of Initiation Latency Criteria Alternating Homogenization: Initiation latency difference vanished in the mixed & alternating conditions No initiation latency differentiation by cognitive knowledge in the alternating condition Adjustment of initiation latency criteria by trial type repetitions ++ + + + Trial sequence 00101 + 2 Initiation Latency

60. 60 -50 -25 0 25 50 75 100 1over 2 Number of the same type trial repetition 0 ++ + + + Trial sequence 00101 + 2 Trial-by-Trial Initiation Latency Adjustment Moment-to-Moment cumulative learning

61. 61 Dissociation of Predictability & Past Experience in Initiation Latency Alternating -50 -25 0 25 50 75 100 1over 2 Number of the same type trial repetition 0 No initiation latency differentiation by cognitive knowledge Cumulative learning from recent experience Dissociation of the two sources Motor initiation latency is adjusted by recent experience not by cognitive knowledge

62. 62 Adjustment of Accuracy Criteria Cost of sub-optimal adjustment of latency criteria? Initiation Latency Alternating

63. 63 Movement Trajectories Left Right (Inch) Up (Inch) Down Single target Odd-color target BlockedMixedAlternating

64. 64 Maximum curvatures Alternating More curved trajectory trials in mixed & alternated conditions

65. 65 Exp2: Cumulative Learning in Predictable Sequence Alternating -50 -25 0 25 50 75 100 1over 2 Number of the same type trial repetition 0 Cumulative learning from recent experience Questions Time course with longer sequence? Predictable sequence?

66. 66. Exp2: Cumulative Learning in Predictable Sequence + + + + + 0 + + + ++ 123401234 Odd-color Singe target Odd-color target switching 5 single/odd-color target alternation Alternation informed in advance & Visual cue

67. 67 Intermixing trials of varying difficulty eliminates differences between initiation latencies of easy and difficult trials Discussion Alternating It is Not task switching : both easy & difficult trials slowed by intermixing (Allport et al., 1994) Initiation latency in Exp1

68. 68 Repetition of trials with the same difficulty gradually differentiates initiation latencies by cumulative learning Discussion Singe target Odd-color target It is not perceptual priming : both easy and difficult trials facilitated by repetition (Maljkovic & Nakayama, 1994) Initiation latency in Exp 2

69. 69 It is not specific to input or output systems : Also, no difference between easy and difficult trials in reading aloud tasks (Lupker et al., 1997) Discussion

70. 70 Fast initiation Central mechanism? : Determine sensorimotor readiness by very recent experience : Adjustable gain or threshold setting Discussion Time Threshold Decision(Activation) Slow initiation

71. 71 Part 1: Role of focal attention on latencies and trajectories of manual pointing Part 2: Concurrent processing of manual pointing to competing stimuli Part 3: Automatic adjustment of visuo- motor readiness

72. 72 Simple visually-guided action requires the allocation of focused attention to the target Multiple planning & execution of manual pointing toward competing stimuli can occur Visuo-motor action & target selection process are interacting continuously even after motor execution Conclusion

73. 73 Visuo-motor system is less concerned with planning a single optimal trajectory as long as the ultimate task goal is successfully achieved Application: Instantaneous read-out Perceptual/cognitive processing Conclusion

74. 74 Acknowledgements Ken Nakayama Patrick Cavanagh Yuhong Jiang Anne Grossetete Harvard Vision Lab

75. 75 Thank you!

76. 76 680 700 720 740 760 25811 Number of distractors Mixed Blocked. Position within Same Color Sequence... 1 8 RT 5 Target: (Maljkovic & Nakayama, 1994; Bichot & Schall, 1999; McPeek et al., 1999) Exp 2: Effect of Perceptual Priming Total Time Attended target feature repetition

77. 77. Exp 2: Effect of Perceptual Priming 650 675 700 725 750 775 01234567 Order in sequence Blocked Baseline 7 Subjects Total time (ms) Order in sequence 400 425 450 475 500 01234567 Blocked Baseline Movement duration Order in sequence 240 245 250 255 260 265 270 01234567 Blocked Baseline Initial latency

78. 78. Maximum curvature Exp 2: Effect of Perceptual Priming 0.05 0.075 0.1 0.125 0.15 0.175 01234567 Order in sequence Maximum curvature Blocked Baseline

79. 79 Curved trajectories : Short corrective latencies indicates that subjects can plan and execute multiple pointing toward competing stimuli : Visuo-motor action & target selection process are interacting continuously even after motor execution Perception Cognitive decision Action Conclusion