What made you respond face (or word)? Something in your brain made you decide face or word. Can we determine where this decision is made? Related domain: Motion Direction Discrimination and area MT

What determines the percept and the response? Observe a correlation between motion direction judgements and activity of cells in area MT. (Britten et al. 1996, Gold & Shadlen, 2000) If neurons that responded to leftward motion were highly active, the monkey chose 'left' as the decision. From Schall, 1999

N170 - distinct negative potential at 170 ms post-stimulus onset fairly early in processing This occurs at T6; smaller effect at T5 T = temporal lobe 5 = left 6 = right right hemisphere specialization

Evoked Response Potential (ERP) and Face Stimuli N170: negative-going potential at 170 ms Largest over the right parietal lobe, also on the left parietal lobe. N170 - distinct negative potential at 170 ms post-stimulus onset fairly early in processing This occurs at T6; smaller effect at T5 T = temporal lobe 5 = left 6 = right right hemisphere specialization From Tanaka and Curran (2001)

N170 Properties: Faces produce the largest amplitude. Strong evidence of expertise: Bird experts have larger N170's to pictures of birds than pictures of dogs. Dog experts show the reverse. (Tanaka & Curran 2001). Mainly perceptually based: prior exposure of a face does not produce large changes in the N170 for subsequent presentations (Rik Henson, AIC 2003). Scalp distribution and latency suggest that the N170 component reflects the perceptual processing of complex visual stimuli. fMRI studies show IT is active whether attending to faces or not (Tarr) No effect of familiarity (Bentin & Deouell, 2000) target status: Is the face-sensitive N170 the only ERP not affected by selective attention? (Caquil, Edmonds, & Taylor, 2000) Appears to be feed-forward perceptual processing of faces or other face-like stimuli

A Thought Question: What was going on in your perceptual regions when you thought you saw a face or a word? Could we capture the current state (at least indirectly) with the N170 component? Would the N170 be larger when you thought you saw a face? fMRI studies show IT is active whether attending to faces or not (Tarr) No effect of familiarity (Bentin & Deouell, 2000) target status: Is the face-sensitive N170 the only ERP not affected by selective attention? (Caquil, Edmonds, & Taylor, 2000) Appears to be feed-forward perceptual processing of faces or other face-like stimuli

Central Question: Can we relate the size of the N170 to the response in the noise-alone condition? Will it be larger when subjects think they see a face? fMRI studies show IT is active whether attending to faces or not (Tarr) No effect of familiarity (Bentin & Deouell, 2000) target status: Is the face-sensitive N170 the only ERP not affected by selective attention? (Caquil, Edmonds, & Taylor, 2000) Appears to be feed-forward perceptual processing of faces or other face-like stimuli

An Experiment Show Faces and Words Embedded in Noise: High Contrast Low Contrast Faces Low Contrast Words High Contrast Words Noise Alone

Methods Face, word and noise-alone trials were presented in random order. Ten naïve participants 120 trials per condition per subject On each trial: Did you see a face or a word? Subjects were told that a stimulus appeared on each trial, and that they should guess if they were unsure. Participants had to indicate whether there was a face or a word present Participants were told that a very low contrast stimulus was present in zero-contrast (i.e., noise alone) condition

EEG Recording Sites T5 - Left T6 - Right*

Methods continued Analyze the data according to the subject’s responses on the noise-alone trials. Incredibly important point: The noise was the same across ALL trials and stimuli. Physically the same. Not just identically distributed, but identical. There was only one noise field for the entire experiment. Together, these procedures hold the physical stimulus constant on noise-alone trials. Participants had to indicate whether there was a face or a word present Participants were told that a very low contrast stimulus was present in zero-contrast (i.e., noise alone) condition

Amplitude (V) ***RIGHT*** Two-tailed t-tests - average amplitude for each observer in window N170- 140-200 ms T6: t(9) = 2.74, p = .023 no significant results for P100 (80-130 ms) or P300 (260-340 ms) no significant results for other sites

Central Question: Will we see a larger N170 to the noise-alone stimulus when subjects think they see a face as opposed to a word? T5 - Left T6 - Right*

* Amplitude (V) ***RIGHT*** * t(9) = 2.74, p = .023, two-tailed Two-tailed t-tests - average amplitude for each observer in window N170- 140-200 ms T6: t(9) = 2.74, p = .023 no significant results for P100 (80-130 ms) or P300 (260-340 ms) no significant results for other sites

Main Result: On noise-alone trials: Larger N170 when observers report seeing a face than when report seeing a word. Occurs in 9 of the 10 subjects. No other differences in any other channel at the P100, N170 or P300 components. Unlikely to just reflect activity for an already-made decision. Relates activity in the perceptual processing areas to the behavioral response. Greater activity in the N170 neurons is associated with ‘face’ responses to the noise-alone stimulus. One interpretation: Greater activity in the face processing region biases the response towards a ‘face’ response. Inside-out gets at the idea that the internal response might be higher for whatever reason on that trial, and this leads observer to think saw a face, and to give an overt 'face' response. Note that our data is just correlational; it could be that the prior trial could have primed our N170 neurons, and that we were thinking about faces for other reasons (maybe frontal cortex was primed as well), and this leads to the correlation in the absence of causality. It might not be that the greater N170 is b/c they saw a face in the noise, but rather seeing a face on the previous trial primed them. In other words, do observers have larger N170 on present trial if a face was shown on the previous trial?

Alternative Explanations for this Greater Activity Attention to different spatial frequencies or face-like features in the noise Unlikely to see rapid changes in spatial frequency tuning in a mixed design. No P100 differences that might be associated with changes in activity in different spatial frequency channels. One obvious exception: Prior trial priming Seeing a face on the previous trial may leave residual activity in the face neurons or make subjects look for face-like features in the noise. Inside-out gets at the idea that the internal response might be higher for whatever reason on that trial, and this leads observer to think saw a face, and to give an overt 'face' response. Note that our data is just correlational; it could be that the prior trial could have primed our N170 neurons, and that we were thinking about faces for other reasons (maybe frontal cortex was primed as well), and this leads to the correlation in the absence of causality. It might not be that the greater N170 is b/c they saw a face in the noise, but rather seeing a face on the previous trial primed them. In other words, do observers have larger N170 on present trial if a face was shown on the previous trial?

Prior Trial Priming Amplitude (V) Face response not influenced by previous trial Word response IS influenced by previous trial, but in wrong direction for positive priming Point out how to recover data reported earlier - priming effect is only small part of what we found

Third Possibility: Stochastic Activity In the domain of binocular rivalry, Blake and Logothetis introduced the idea of a process that involved stochastic activity in perceptual regions, which could bias the response toward one percept or the other at different points in time. (Blake & Logothetis 2002). A similar process could be at work in the face processing neurons: When activity is high in the N170 neurons due to random stochastic fluctuations, the observer may be biased to respond ‘face’ on that trial. Inside-out gets at the idea that the internal response might be higher for whatever reason on that trial, and this leads observer to think saw a face, and to give an overt 'face' response. Note that our data is just correlational; it could be that the prior trial could have primed our N170 neurons, and that we were thinking about faces for other reasons (maybe frontal cortex was primed as well), and this leads to the correlation in the absence of causality. It might not be that the greater N170 is b/c they saw a face in the noise, but rather seeing a face on the previous trial primed them. In other words, do observers have larger N170 on present trial if a face was shown on the previous trial?

Implications for Internal Noise Internal noise does not just limit performance or decrease calculation efficiency, but also operates in feature space to bias the response toward one alternative or another. Manipulations such as varying the power of the noise or the stimulus pairs being compared may help constrain models of internal noise. Supports trial-by-trial variability in parameters: Starting point and drift-rate variability in Ratcliff’s Diffusion model. These sources of variability take on a perceptually-based interpretation. Link with perceptual brain areas. Inside-out gets at the idea that the internal response might be higher for whatever reason on that trial, and this leads observer to think saw a face, and to give an overt 'face' response. Note that our data is just correlational; it could be that the prior trial could have primed our N170 neurons, and that we were thinking about faces for other reasons (maybe frontal cortex was primed as well), and this leads to the correlation in the absence of causality. It might not be that the greater N170 is b/c they saw a face in the noise, but rather seeing a face on the previous trial primed them. In other words, do observers have larger N170 on present trial if a face was shown on the previous trial?

For Noise-Alone Trials: The Important Stuff: For Noise-Alone Trials: * t(9) = 2.74, p = .023 High Contrast Faces High Contrast Words Low Contrast Words Noise Alone Amplitude (V)

ERP and Faces Intracranial recordings reveal N200 at sites in IT/fusiform gyrus From Allison, Puce, Spencer, and McCarthy (1999)

Which Stimuli Evoke an N170/N200? Any face or face-like visual stimulus Face-specific N170/N200: Any face, regardless of sex, age, emotional expression, viewpoint High- and low-pass filtered faces Parts of faces; internal and external features Cartoon faces No face-specific N170/N200: butterflies, flowers, etc. words* scrambled faces hands

N170 Properties: Top-down Influences Could be just a face-detection system (Bentin, et. al. 1996) No effect of task demands such as selective attention to faces vs. objects. (Caquil, Edmonds, & Taylor, 2000) No effect of familiarity of the face (Bentin & Deouell, 2000) fMRI - IT active anytime there’s a face. (Gauthier et al. 1999) However, IT also active while imagining a face. (O’Craven & Kanwisher 2000) fMRI studies show IT is active whether attending to faces or not (Tarr) No effect of familiarity (Bentin & Deouell, 2000) target status: Is the face-sensitive N170 the only ERP not affected by selective attention? (Caquil, Edmonds, & Taylor, 2000) Appears to be feed-forward perceptual processing of faces or other face-like stimuli

Contextual Influences On N170 Bentin et al (2002) Block 1: Stimulus Set A Block 2: Set B (experimental) Set C (control) Set D (all; targets) Block 3: Set A again (all) Task: count schematic flowers

Contextual Influences On N170 Bentin et al (2002) Face context elicits N170 to schematic eyes Once the stimulus has been interpreted as containing face-like features, a stronger N170 is produced. Experimental and control group showed no N170 response to schematic eyes alone before priming Experimental showed N170 response to schematic eyes alone in Block 3 (after priming) Control group showed no N170 response to schematic eyes alone in Block 3 *We wanted to further explore non-perceptual influences

N170 Summary Represents activity of face-selective neurons most likely in area IT. Magnitude varies with the degree to which physiognomic information is perceived in the image. Is the N170 related to the response in the noise-alone condition? If so, tie the activity in face-selective cells to the percept and the response.

Would we get an N170 to the noise-alone trials at all? Previous work showed only a weak N170 to random noise stimuli. Pilot study: Blocked presentations of faces and words. Result: when looking for faces, get a larger N170 to noise-alone trials than when looking for words. Replicates Bentin et al., extends to images that contain no facial features. We see top-down effects on the N170 even with no face-like features.

Face Condition : Male or female? Task : Is the face male or female? Have hi-, low-, and no-contrast (i.e., noise alone) condition Told subjects there was actually a very low-contrast stimulus in the no-contrast condition

Word condition: Honesty or Trust? IMPORTANT POINT: NOISE STIMULI IDENTICAL FOR BOTH FACE AND WORD TRIALS

Experiment 1 Results - T5 Grand Average for T5 High-Contrast Face 8 6 4 High-Contrast Face Low-Contrast Face Zero-Contrast Face High-Contrast Word Low-Contrast Word Zero-Contrast Word 2 Voltage (microvolts) -2 Big mess - just look at crucial condition -4 -6 -8 -10 -12 -100 100 200 300 400 500 600 Time (ms)

Experiment 1 Results - T5 Zero-Contrast Word Zero-Contrast Face LEFT Grand Average for T5 Zero-Contrast Word Zero-Contrast Face 6 * 5 4 *t(8) = 2.62, p = .031 3 Voltage (microvolts) 2 1 LEFT Two-tailed t-tests - average amplitude for each observer in window from 140-200 ms T5: t(8) = 2.62, p = .031 no significant results for P100 (80-130 ms) or P300 (260-340 ms) N170 significant for all sites, but difficult to interpret -1 -2 -3 -100 100 200 300 400 500 600 Time (ms)

Neural Processing of Face Stimuli Cells in inferotemporal cortex (IT) are known to respond selectively to faces (single cell recording, fMRI) Slight right hemisphere dominance

Experiment 1 Results - T6 Zero-Contrast Word Zero-Contrast Face RIGHT Grand Average for T6 Zero-Contrast Word Zero-Contrast Face 5 * 4 3 * t(8) = 2.35, p = .047 2 Voltage (microvolts) 1 RIGHT Two-tailed t-tests - average amplitude for each observer in window from 140-200 ms T6: t(8) = 2.35, p = .047 no significant results for P100 (80-130 ms) or P300 (260-340 ms) N170 significant for all sites, but difficult to interpret -1 -2 -3 -100 100 200 300 400 500 600 Time (ms)

Experiment 1 Conclusion When an observer expects a face (versus a word) there is a greater N170 When an observer expects, or is looking for, a face (versus a word) there is a greater N170