1. ON THE RELATIONSHIP BETWEEN MOTOR AND PERCEPTUAL BEHAVIOR – A SDT FRAMEWORK Andrei Gorea with Pedro Cardoso-Leite Florian Waszak Pascal Mamassian Laboratoire de Psychologie de la Perception CNRS & René Descartes University 71 Ave Edouard Vaillant, 92774 Boulogne-Billancourt, France

2. 1.Background elements 2.RT & Perceptual States with masked and not-masked stimuli 2.1 A one-path – two-decisions model 3.RT & Temporal Order Judgments 3.1 One-path – two-decisions model again 4.General conclusions SYNOPSYS

3. PART I SOME BACKGROUND

4. Sensory Input A. Layman’s view Action with perceptual awareness Perceptual Decision Verbal report aware / not aware

5. Priming & Metacontrast + RT  Fehrer, E. & Raab, D. (1962)  Harrison, K. & Fox, R. (1966)  Schiller, P. H. & Smith, M. C. (1966)  Neumann, O. (1982)  Neumann, O., Esselmann, U. & Klotz, W. (1993)  Klotz, W. & Neumann O. (1999)  Taylor, J.L. & McCloskey, D.I. (1990)  Steglich, C. & Neumann, O. (2000)  Schmidt, T. (2002)  Ogmen, H., Breitmeyer, B. & Melvin, R. (2003)  Scharlau, I. & Ansorge, U. (2003)  Scharlau, I. & Neumann, O. (2003)  Vorberg, D., Mattler, U., Heinecke, A., Schmidt, T. & Schwarzbach, J. (2003)  Rossetti, Y. & Pisella, L. (2002)  Breitmayer, B., Ro, T. & Singhal, N. S. (2004) Perception & Gaze pursuit  Beutter, B.B. & Stone, L.S. (1998)  Beutter, B.B. & Stone, L.S. (2000).  Gegenfurtner, K.R., Xing, D., Scott, B.H. & Hawken, M.J. (2003)  Osborne, L.C., Lisberger, S.G. & Bialek, B. (2005)  Morrone, M.C., Raffele, S., Ma-Wyatt, A. & Ross, J. (2005) TOJ & RT  Roufs, J.A.J. (1974)  Jaskowski, P. (1991, 1992, 1993, 1996)  Tappe, T., Niepel, M. & Neumann, O (1993)  Jaskowski, P. & Verleger, R. (2000)  Spence C., Baddeley R., Zampini M., James R. & Shore D.I. (2003)  Adams, W.J & Mamassian, P. (2004) Most frequently used experimental paradigms

6. Decision rule not specified Decision rule (implicit) Sensory Input Action with or without perceptual awareness Verbal report aware / not aware B. The 2 pathways view ? Lateral Interactions (implicit) ventral dorsal

7. Most frequently used experimental paradigms PARADIGMPerceptual taskMotor TaskComments Congruent/Incongruent Priming with/without Metacontrast Detection &/or Discrimination (typically bearing on the “prime”) Appearance (of the "prime“) Mostly choice- (but also a few simple-) RT (manual, saccades)  Despite consensus, dissociation was not formally proved: methodological problems of many sorts  Conceptually, choice-RT cannot be dissociated from perceptual decisions Temporal Order Judgments TOJSimple-RT, typically to one single stimulus (rather than the 2 used in the perceptual task)  Inconsistent results  Methodological problems Perception & gaze pursuit Discrimination (typically of speed) Pointing, gaze pursuit  Contradictory results (presumably due to technical problems)

8. None of the experimental paradigms used to explore the sensorimotor dissociation allowed a trial-by-trial analysis of the relationship between the motor behavior & the state of the perceptual system (Hits, FA, Misses, CR).

9. PART II RESPONSE TIME & PERCEPTUAL STATE

10. Simple RT Correlation S 1 -RT Correlation S 2 -RT T I M E T 0 : Start Clock 400-1000 0 d’ modulation criterion modulation Stimuli & Paradigm (one trial) SOA: 50-250 S 2 : Mask/Primed (p = 1) 20 Masked (metacontrast) S2S2 S 1 : Yes/No? Hits FA Misses CR 100-200 S 1 : Target/Prime (p =.2,.5,.8) 20 S1S1  7° SOA: 50-250 S 2 : Mask/Primed (p = 1) 20 NOT Masked S2S2 22.5° Waszak & Gorea (2004).

11. Of the 4 perceptual response categories, Hits & Misses are of particular interest: They tell us about the motor behavior when the Obs. says he senses and does not sense the test stimulus (S 1 ), hence establishing the relationship between perceptual and motor behavior*. * RT for FA are not reliable indices as they have an unknown temporal origin; RT for CR are simply used as reference for the relevant RT.

12. The motor system appears to react if and only if  the stimulus is present and  the observer is “aware” of it (i.e. only for Hits). The difference between RTs for Hits & Misses points against a full sensori-motor dissociation. S1S1 Masked (metacontrast) S2S2 SPACE 4 Sjs S 1 13 ms SOA variable S 2 36 ms 300 trials / d’ / Obs TIME t 20 ms SOA variable Waszak & Gorea (2004).

13. HitsMissesFACR AGSD.2.5 p[S 1 ].8 FW Waszak & Gorea (2004).

14. Rank Correlations between RT and S 1 -, S 2 -onsets as a fct. of d’ c. Correct Rejections -0.1 0.3 0.7 012345 r tR-tS2 a. Hits -0.1 0.3 0.7 012345 r tR-tS1 & r tR-tS2 Waszak & Gorea (2004). TIME Correlation S 1 -RT Correlation S 2 -RT S1S1 Speeded RT S2S2 SPACE S1S1 S2S2 SD AG FW Sj b. Misses -0.1 0.3 0.7 012345 r tR-tS1 & r tR-tS2 d. False Alarms -0.1 0.3 0.7 012345 r tR-tS2

15. Decision rule not specified Decision rule (implicit) Sensory Input Action with or without perceptual awareness Verbal report aware / not aware ? Lateral Interactions (implicit) ventral dorsal B. The 2 pathways view We’ve thus replaced the standard 2-pathways view… Action with perceptual awareness A’. Layman’s view modified …with a slightly modified layman’s view

16. Where does the discrepancy come from? (Aside from potential methodological problems in previous studies) Common denominator: Most of the previous (whether simple or choice RT) studies used 100% contrast targets whose ‘invisibility’ was ensured by strong backward masking. In contrast, our targets (S 1 ) yielded maximum contrasts of about 20%. To allow for higher target contrasts while keeping sensitivity constant, shorter SOA-s (48 ms instead of an average of 162 ms) were used in a second series of experiments; these entailed S 1 contrasts around 30%.

17. Waszak & Gorea (2004). 350 400 450 012345 RT (ms) FW p[S 1 ] =.5 012345 AG p[S 1 ] =.5 FA Misses CR Hits TIME t 20 ms SOA 52 ms S1S1 Masked (metacontrast) S2S2 SPACE

18. Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report aware / not aware C. Gorea & Waszak (2004) Lateral Interactions ventral dorsal Action with perceptual awareness The temptation was strong to conclude (Gorea & Waszak, 2004)…

19. RT for Misses drop with d’ (or contrast) only for the masked condition! RT appear to depend on Contrast rather than on d’! Perceptual HITS 220 240 260 280 300 320 0.511.522.5 d' RT 8% 13% 27% 13% 8% 13% 27% 13% Perceptual MISSES 0.511.522.5 d' S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 17 Obs 300 trials / d’ / Obs Masked NOT Masked Masked NOT Masked Masked S 1 “prime” S 2 “mask” t SOA 13 ms36 ms 52 ms Waszak & Gorea, new experiments.

20. However, the data are more intricate than that…

21. S1S1 Masked (metacontrast) S2S2 NOT MASKED (6 Obs) 010203040 S 1 Contrast (%) 2.6 3.3 3.2.14.45 1.2 2.1 d’ S1S1 S2S2 NOT Masked S2S2 S 1 “prime” S 2 “mask” t SOA 13 ms36 ms 52 ms MASKED (6 Obs) -50 -40 -30 -20 -10 0 10 0 203040 S 1 Contrast (%).21.32.75 1.8 2.1 2.4 2.6 2.8 3.2 3.1 3.2 3.4 d’ RT gain rel. to CR (ms) RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs HITS (6 Obs) Not Masked 300-900 trials / C / Obs Masked MISSES (6 Obs) RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs Not Masked 300-900 trials / C / Obs Masked Waszak & Gorea, new experiments.

22. S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 S 1 “prime” S 2 “mask” t SOA 13 ms36 ms 52 ms 01234 d’ S1 NOT MASKED (6 Obs) -50 -40 -30 -20 -10 0 10 01234 d’ S1 RT gain rel. to CR (ms) MASKED (6 Obs) RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs HITS (6 Obs) Not Masked 300-900 trials / C / Obs Masked MISSES (6 Obs) RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs Not Masked 300-900 trials / C / Obs Masked Waszak & Gorea, new experiments.

23. Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report aware / not aware C. Gorea & Waszak (2004) Lateral Interactions ventral dorsal Action with perceptual awareness …And a simpler, one pathway SDT model can account for them. Instead of: …we propose: Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report aware / not aware D. Current view Lateral Interactions ventral Action with perceptual awareness

24. A conceptual model

25. 40140240340 TIME (ms) S1S1 Masked (metacontrast) S2S2 TIME (ms) RESPONSE (Noise units) 40140240340 -2 0 1 2 3 4 5 6 7 RT d’=1 Motor threshold d’ Perceptual Misses S1S1 S2S2 NOT Masked S2S2 S 1 “prime” S 2 “mask” t SOA Perceptual Misses S1S1 SOA RT Mask S2S2 SOA RT Mask S2S2 RT d’=1 S1S1 S1S1 d’

26. S 1 “prime” S 2 “mask” t S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 RT d’=2 RT Mask RT d’=2 RT Mask SOA Perceptual Misses d’ Motor threshold SOA

27. S1S1 Masked (metacontrast) S2S2 S 1 “prime” S 2 “mask” t SOA Perceptual Misses d’ SOA 1 7 240340 TIME (ms) S1S1 S2S2 NOT Masked S2S2 RT d’=4 RT Mask -2 0 40140 RT Mask RT d’=4 Motor threshold -60 RESPONSE (Noise units) SOA 6 2 d’ Perceptual Misses 3 4 5

28. S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 S 1 “prime” S 2 “mask” t SOA 13 ms36 ms 52 ms MASKED 6 Obs 230 240 250 260 270 280 290 300 01234 d’ S1 RT (ms) NOT MASKED 6 Obs 01234 d’ S1 RT HITS -RT CR RT MISS -RT CR 300-900 trials / C / Obs Model Fits Motor Threshold .8 

29. Sensory Input A. Layman’s view Action with perceptual awareness Perceptual Decision Verbal report aware / not aware Decision rule not specified Decision rule implicit Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) B. The 2 pathways view ? Lateral Interactions implicit ventral dorsal Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) C. Gorea & Waszak (2004) Action with perceptual awareness Lateral Interactions dorsal ventral Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) D. Current view Action with perceptual awareness Lateral Interactions ventral dorsal ventral

30.  A one-pathway model with two distinct activation levels accounts for the observed perceptual-motor relationship under both masking and non-masking experimental conditions.  There is a fixed motor threshold (  0.8  )  to be contrasted with a variable perceptual criterion).  The motor threshold is measured in noise (  ) units as referred to the “absolute” perceptual detection “threshold” (i.e. in ref. to the internal noise).  RT for “unconscious” stimuli (i.e. Misses) depends on the reference (noise) level at which the perceptual task is performed;  as this reference level exceeds the motor threshold, the internal response associated with perceptual Misses also exceeds it and progressively contributes to shortening the RT. TAKE-HOME MESSAGES (Part II)

31. PART III RESPONSE TIME & TEMPORAL ORDER JUDGMENTS

32. Time C2 C1  Trigger-Delay  PSS PSS  RT C2  RT C1 Threshold Internal Response SOA PSS p(S 1 perceived first) slope  Temporal Order Judgments & RT IFF RT is strictly dependent on the sensory signal (as it determines the TOJ), then the slope and the PSS of the TOJ  -function should be direct indicators of the variance of the RT distributions and of their mean difference, respectively: this is a one-pathway sensorimotor model.

33. abc In contrast with previous studies:  TOJ and RT were measured in the same trial;  It was hence possible to assess RTs for correct & incorrect TOJs.  RT C2 C1 DG PSS = 0 ms DG 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -200-1000100200 SOA (ms) %S1 first C1-C1 C2-C2 C1-C2 PSS = 49 ms 500 ms 500-1300 ms SOA ~ 1700 ms RT Left / Right Stimuli & Paradigm (one trial)

34. Identical Different  -RT Correct / Incorrect Correct Incorrect S1 1 st S2 1 st

35. 375 IdenticalDifferent  -RT Correct / Incorrect  -RT Correct / Incorrect

36. ab  Red: C1 Green: C2 Blue: O1 Brown: O2  cd

37. ab  RT PSS RT R MM PCPC C High PT 1 PT 2 RT 1 PSS  RT RT 2 M  < P C M  > P C M  = P C R t C Low

38.  For 2 out of 4 Obs, RT to single stimuli also predict their TOJ behavior (PSS and slopes of the TOJ  -fct.); their behavior is hence compatible with a one pathway model (no sensorimotor dissociation).  For these 2 Obs, the relationship between RT-differences and PSS also suggests the existence of a motor threshold distinct from and higher than the perceptual criterion.  The behavior of the remaining 2 Obs does not follow any coherent pattern; it is likely that it was perturbed by perceptual response strategy factors, possibly under the influence of “transient Troxler masking” (Kanai & Kamitani, 2003).  Such factors may account for the various inconsistencies in the RT-TOJ literature. TAKE-HOME MESSAGES (Part III)

39.  The relationship between simple Response Times and perceptual states (Hits, FA, etc. and Temporal Order Judgments) can be accounted for by a single (one-path) system where two distinct decisions are made on the same incoming information.  The decision to act is based on a hard-wired threshold worth about 1  of the internal noise; the perceptual criterion is context dependent (in line with SDT).  TOJ data not complying with this model may reflect variable perceptual response strategies. GENERAL CONCLUSIONS

40. THANK YOU

41. Intermediate conclusion: ONE pathway, TWO outputs X Input Stimulus present + Awareness of it dorsal ventral MOTOR RESPONSE PERCEPTION Gorea & Waszak, 2004

42. S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 8% 13% 27% 13% Perceptual Hits 220 240 260 280 300 320 0.511.522.5 d' RT 8% 13% 27% 13% Perceptual Misses 0.511.522.5 d' DataModel Masked NOT Masked S 1 “prime” S 2 “mask” t SOA

43.  The faster the RT, the higher the likelihood that it was triggered by S 1 ;  The slower the RT, the higher the likelihood that it was triggered by S 2;  The correlations for Misses are null (as expected). Rank Correlations between RT and S 1 -, S 2 -onsets as a fct. of RT Hits only S1S1 S2S2 FastSlowFastSlowFastSlow Additional data from Waszak & Gorea (2004). S2S2 t S1S1

44.  The faster the RT, the higher the likelihood that it was triggered by S 1 ;  Conversely, the slower the RT, the higher the likelihood that it was triggered by S 2. Data from Waszak & Gorea (2004). Visual Cognition. Spearman Correlations between RT and S 1, S 2 onsets as a fct. of RT Hits only FastSlowFastSlow Perceptual Hits Motor Hits S1S1 N Perceptual FA CMCM d’ CPCP The fact that on a (variable) proportion of trials RTs are triggered by S 2 although S 1 was present and perceived (Hits) implies that the motor threshold is higher than the perceptual criteria. S1S1 S2S2

45. SOA 11 33 11 22 If motor threshold and perceptual criteria differ significantly, the same should be true for the temporal dispersion of R at each of these two levels (see Carpenter).

46. If the motor threshold and perceptual criteria differ significantly, the same should be true for the temporal dispersion of the internal response, R, at each of these two levels (see Carpenter et al.). SOA 11 22 Reddi & Carpenter (2000). Nature Neurosci., 3, 827-830.

48. Becker & Anstis (2004) Vision Research 44, 2537–2543 No backward masking with opposite polarity Test-Mask TIMESPACE OR What does the Obs. Compare? vs.A B Either, both? C =< => Most likely both (also depending on SOA)

49. TIMESPACE OR What do the Obs. compare? vs.A Either, both? C vs.B =< => Most likely both (also depending on SOA)

50. Sensory Input A. Layman’s view Action with perceptual awareness Perceptual Decision Verbal report aware / not aware Decision rule not specified Decision rule implicit Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) B. The 2 pathways view ? Lateral Interactions implicit ventral dorsal Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) C. Gorea & Waszak (2004) Action with perceptual awareness Lateral Interactions dorsal ventral Fixed Motor Threshold Variable Perceptual Criterion Sensory Input Action with or without perceptual awareness Verbal report (aware / not aware) D. Current view Action with perceptual awareness Lateral Interactions ventral dorsal ventral

51. Temporal Order Judgments & RT IFF RT is strictly dependent on the sensory signal (as it determines the TOJ), then the slope of the TOJ  -function should be a direct indicator of the variance of the RT distribution (see Adams & Mamassian, 2004). SOA 0 P(S 1 perceived first) slope  S2S2 +SOA (  d’) S1S1 titi t In contrast with previous studies:  TOJ and RT were measured in the same trial;  It was hence possible to assess RTs for correct & incorrect TOJs.

52. S 1 +NN c0 R S 1 + S 2 +NS 2 +N c =  R Misses  R Hits  R Misses  R Hits NOT MASKEDMASKED

53. T 0 : Start Clock 400-1000 0 100-200 S 1 : Target/Prime (p =.2,.5,.8) 20 S1S1  7° Simple RT Correlation S 1 -RT Correlation S 2 -RT T I M E d’ modulation criterion modulation SOA: 50-250 S 2 : Mask/Primed (p = 1) 20 Masked (metacontrast) S2S2 S 1 : Yes/No? Hits FA Misses CR SOA: 50-250 S 2 : Mask/Primed (p = 1) 20 NOT Masked S2S2 22.5°

54. S 1 “prime” S 2 “mask” t S1S1 Masked (metacontrast) S2S2 S1S1 S2S2 NOT Masked S2S2 RT d’=2 RT Mask RT d’=2 RT Mask SOA Perceptual Misses d’ Motor threshold SOA

55. Of main interest for further research  The direct assessment of the motor threshold  Still searching for the critical paradigm.  Test the reference noise level idea in metacontrast and revise metacontrast models accordingly  The empirical test could consist in manipulating the external noise under masked and unmasked conditions.

56. Simple RT to any (either S 1 or S 2 ) S 2 p(S 2 )=1 SOA S 1 : Yes / No? Hits FA Misses CR S 1 0<p(S 1 )<1 TIME a S1S1 Masked (metacontrast) S2S2  7° SPACE b S2S2 S1S1  7° Not Masked S2S2  3°  7° c SPACE