Auditory cortical monitoring prevents speech errors before they happen Caroline A. Niziolek UCSF Depts. of Radiology and Otolaryngology – Head and Neck Surgery Biomagnetic Imaging Laboratory Speech Neuroscience Laboratory

The Speech Chain Denes & Pinson, 1993

The Speech Chain Denes & Pinson, 1993

The Speech Chain Denes & Pinson, 1993

The Speech Chain Denes & Pinson, 1993

The Speech Chain Denes & Pinson, 1993 deafness

The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility

The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility frontal motor lesion

The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility frontal motor lesion perceptual deficits

Understanding speech production via perception

MEG fMRI ECoG

Understanding speech production via perception Houde, Niziolek, et al., 2014, ISSP MEG fMRI ECoG

Understanding speech production via perception

How does auditory feedback affect speech output? Understanding speech production via perception

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? Understanding speech production via perception

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

Auditory feedback causes compensatory changes in our speech output

Focal acoustic changes evoke focal compensation frequenc y amplitud e

Focal acoustic changes evoke focal compensation frequenc y amplitud e f0 (pitch)

Focal acoustic changes evoke focal compensation frequenc y amplitud e formants (vowel)

Focal acoustic changes evoke focal compensation frequenc y amplitud e amplitude (loudness )

Chang, Niziolek, et al., 2013, PNAS f0 shift during vocalization

f0 shift of emphatic stress f0 contrast distance Patel et al., 2011, JSLHR

f0 shift of emphatic stress f0 contrast distance Patel et al., 2011, JSLHR ampl contrast distance

Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids

Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

Real-time formant alteration

Greater compensation to vowels shifted across a boundary Niziolek & Guenther, 2013, J. Neurosci.

Enhanced auditory error to shifts across a boundary Niziolek & Guenther, 2013, J. Neurosci.

Chang, Niziolek, et al., 2013, PNAS Niziolek & Guenther, 2013, J. Neurosci. f0formants Patel et al., 2011, JSLHR f0 contract distance

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS

Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL

Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED

Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED < feedback match: suppression

Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED < feedback match: suppression feedback mismatch: no suppression ≥

Auditory cortical neurons are suppressed during speech, but only when the feedback matches what is expected.

What causes this selective suppression? Can it be used for error detection?

Models of speech motor control

CONTROLLER : premotor, motor cortex Models of speech motor control

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs Models of speech motor control motor command

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex internal prediction Models of speech motor control motor command

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction Models of speech motor control motor command

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction Models of speech motor control motor command

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction corrective error signal Models of speech motor control motor command

CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction corrective error signal Models of speech motor control motor command

Artificially-altered feedback

Chang, Niziolek, et al., 2013, PNAS release from suppression

Artificially-altered feedback Chang, Niziolek, et al., 2013, PNAS change in speech output release from suppression

Are these error- correction processes at work in natural speech?

Multi-vowel task eat spea k Ed add list en eat Ed add Niziolek et al., 2013, J. Neurosci

Auditory M100

Owen et al., 2012, NeuroImageNiziolek et al., 2013, J. Neurosci Source localization: Champagne

Vowel space: center vs. periphery Niziolek et al., 2013, J. Neurosci

Vowel space: center vs. periphery Niziolek et al., 2013, J. Neurosci “eat ” “Ed” “add”

Are peripheral productions processed as errors?

centerperiphery F1 F2

centerperiphery F1 F2 predict variability off center = expected

centerperiphery F1 F2 predict variability off center = expected

centerperiphery F1 F2 predict variability off center = expected

centerperiphery F1 F2 predict variability off center = expected don’t predict variability off center = error

center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

Individual subject decrease in suppression Niziolek et al., 2013, J. Neurosci

Decrease in suppression is consistent in left AC p = Niziolek et al., 2013, J. Neurosci

Decrease in suppression is consistent in left AC Niziolek et al., 2013, J. Neurosci

center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error WINNER

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

Suppression falls off at the vowel periphery Niziolek et al., 2013, J. Neurosci Fall-off increases with acoustic distance

Acoustic error is coded by auditory suppression

The efferent prediction may reflect an acoustic target or goal (not merely a “copy” of motor commands)

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

Does error-like processing have behavioral consequences?

Behavioral consequences

Niziolek et al., 2013, J. Neurosci Behavioral consequences

If centering is partly driven by auditory feedback, it should: correlate with cortical responses to feedback decrease when you can’t hear yourself (i.e., in masking noise)

Centering correlates with suppression Niziolek et al., 2013, J. Neurosci

Masking noise reduces centering Niziolek et al., submitted quiet

Masking noise reduces centering quiet masking noise Niziolek et al., submitted

Masking noise reduces centering quiet masking noise Implications for hearing loss Niziolek et al., submitted

We detect deviations from an expected target Decreased suppression  ongoing error detection process (we make “errors” all the time!)

We correct our deviations before they become errors Decreased suppression  ongoing error detection process (we make “errors” all the time!) Centering  ongoing error correction process (partly mediated by feedback)

Ongoing & future directions: Assess error detection and correction capacities in patients with speech disorders

Speech error detection and correction in aphasia

detection impaired

Speech error detection and correction in aphasia centerperiphery F1 F2 detection impaired

Speech error detection and correction in aphasia centerperiphery F1 F2 detection impaired detection preserved

Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved

Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved sensory feedback training

Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved sensory feedback training motor skill training

Ongoing & future directions: How task-specific is the perceived error?

Niziolek et al., in prep Redefining center and periphery

Sort by formant Niziolek et al., in prep Redefining center and periphery F1 F2

Sort by formant Niziolek et al., in prep Redefining center and periphery F1 F2 ✓

F0 Sort by formant Sort by pitch Redefining center and periphery F1 F2 ✓

F0 Sort by formant Sort by pitch Redefining center and periphery F1 F2 ✓ ✗

Multi-pitch task Niziolek et al., in prep eat spea k eat list en eat z xc z xc

Owen et al., 2012, NeuroImageNiziolek et al., in prep Source localization: right hemi

F0 Sort by pitch Pitch task auditory suppression

F0 Sort by pitch Pitch task auditory suppression ✓ Niziolek et al., in prep p = 0.048

F0 Sort by formant Sort by pitch Pitch task auditory suppression F1 F2 ✓ Niziolek et al., in prep p = 0.048

F0 Sort by formant Sort by pitch Pitch task auditory suppression F1 F2 ✗ ✓ Niziolek et al., in prep p = 0.27 (n.s.) p = 0.048

Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task

Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task ✓ ✓

Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task ✓ ✓ (n.s.) ✗

Pitch centering Niziolek et al., in prep

Pitch centering Niziolek et al., in prep

Understanding speech production via perception

Altered auditory feedback causes compensatory changes to speech output Understanding speech production via perception

Altered auditory feedback causes compensatory changes to speech output Degree of auditory cortical suppression allows us to detect deviations from what we intend to say Understanding speech production via perception

Altered auditory feedback causes compensatory changes to speech output Degree of auditory cortical suppression allows us to detect deviations from what we intend to say This suppression may underlie a corrective behavior that serves to bring speech back on track Understanding speech production via perception

Thank you! NIH F32DC NIH R01DC NSF BCS Frank Guenther John Houde Sri Nagarajan Eddie Chang Danielle Mizuiri Susanne Honma

Decrease in suppression in both speak and listen Niziolek et al., 2013, J. Neurosci

No systematic changes in suppression in right AC Niziolek et al., 2013, J. Neurosci

No systematic changes in suppression in right AC Niziolek et al., 2013, J. Neurosci

Formant variability

Anti-centering masking noise Niziolek et al., in prep

Direct cortical recordings Chang, Niziolek, et al., 2013, PNAS

Altered feedback Normal feedback Chang, Niziolek, et al., 2013, PNAS

“SIS falloff” across acoustic space Niziolek et al., 2013, J. Neurosci