2. Auditory cortical monitoring prevents speech errors before they happen Caroline A. Niziolek UCSF Depts. of Radiology and Otolaryngology – Head and Neck Surgery Biomagnetic Imaging Laboratory Speech Neuroscience Laboratory

3. The Speech Chain Denes & Pinson, 1993

4. The Speech Chain Denes & Pinson, 1993

5. The Speech Chain Denes & Pinson, 1993

6. The Speech Chain Denes & Pinson, 1993

7. The Speech Chain Denes & Pinson, 1993 deafness

8. The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility

9. The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility frontal motor lesion

10. The Speech Chain Denes & Pinson, 1993 deafness decrease in intelligib ility frontal motor lesion perceptual deficits

11. Understanding speech production via perception

14. MEG fMRI ECoG

15. Understanding speech production via perception Houde, Niziolek, et al., 2014, ISSP MEG fMRI ECoG

16. Understanding speech production via perception

17. How does auditory feedback affect speech output? Understanding speech production via perception

18. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? Understanding speech production via perception

19. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

20. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

21. Auditory feedback causes compensatory changes in our speech output

24. Focal acoustic changes evoke focal compensation frequenc y amplitud e

25. Focal acoustic changes evoke focal compensation frequenc y amplitud e f0 (pitch)

26. Focal acoustic changes evoke focal compensation frequenc y amplitud e formants (vowel)

27. Focal acoustic changes evoke focal compensation frequenc y amplitud e amplitude (loudness )

28. Chang, Niziolek, et al., 2013, PNAS f0 shift during vocalization

29. f0 shift of emphatic stress f0 contrast distance Patel et al., 2011, JSLHR

30. f0 shift of emphatic stress f0 contrast distance Patel et al., 2011, JSLHR ampl contrast distance

31. Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids

32. Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

33. Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

34. Real-time formant alteration Niziolek & Guenther, 2013, J. Neurosci. Niziolek & Guenther, 2014, Frontiers for Kids x y frequency amplitude

35. Real-time formant alteration

47. Greater compensation to vowels shifted across a boundary Niziolek & Guenther, 2013, J. Neurosci.

48. Enhanced auditory error to shifts across a boundary Niziolek & Guenther, 2013, J. Neurosci.

49. Chang, Niziolek, et al., 2013, PNAS Niziolek & Guenther, 2013, J. Neurosci. f0formants Patel et al., 2011, JSLHR f0 contract distance

50. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

51. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

52. Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS

53. Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL

54. Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED

55. Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED < feedback match: suppression

56. Neural responses to feedback Chang, Niziolek, et al., 2013, PNAS NORMAL ALTERED < feedback match: suppression feedback mismatch: no suppression ≥

57. Auditory cortical neurons are suppressed during speech, but only when the feedback matches what is expected.

58. What causes this selective suppression? Can it be used for error detection?

59. Models of speech motor control

60. CONTROLLER : premotor, motor cortex Models of speech motor control

61. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs Models of speech motor control motor command

62. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex internal prediction Models of speech motor control motor command

63. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction Models of speech motor control motor command

64. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction Models of speech motor control motor command

65. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction corrective error signal Models of speech motor control motor command

66. CONTROLLER : premotor, motor cortex OUTPUT : vocal tract, articulato rs COMPARISON : auditory cortex auditory signal internal prediction corrective error signal Models of speech motor control motor command

67. Artificially-altered feedback

68. Chang, Niziolek, et al., 2013, PNAS release from suppression

69. Artificially-altered feedback Chang, Niziolek, et al., 2013, PNAS change in speech output release from suppression

70. Are these error- correction processes at work in natural speech?

71. Multi-vowel task eat spea k Ed add list en eat Ed add Niziolek et al., 2013, J. Neurosci

72. Auditory M100

73. Owen et al., 2012, NeuroImageNiziolek et al., 2013, J. Neurosci Source localization: Champagne

74. Vowel space: center vs. periphery Niziolek et al., 2013, J. Neurosci

75. Vowel space: center vs. periphery Niziolek et al., 2013, J. Neurosci “eat ” “Ed” “add”

76. Are peripheral productions processed as errors?

77. centerperiphery F1 F2

78. centerperiphery F1 F2 predict variability off center = expected

79. centerperiphery F1 F2 predict variability off center = expected

80. centerperiphery F1 F2 predict variability off center = expected

81. centerperiphery F1 F2 predict variability off center = expected don’t predict variability off center = error

82. center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

83. center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

84. center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

85. Individual subject decrease in suppression Niziolek et al., 2013, J. Neurosci

86. Decrease in suppression is consistent in left AC p = 0.002 Niziolek et al., 2013, J. Neurosci

87. Decrease in suppression is consistent in left AC Niziolek et al., 2013, J. Neurosci

88. center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error

89. center periphery F1 F2 predict variability off center = expected don’t predict variability off center = error WINNER

90. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

91. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

92. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

93. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

94. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

95. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

96. acoustic distance decrease in suppressio n ∝ Niziolek et al., 2013, J. Neurosci

97. Suppression falls off at the vowel periphery Niziolek et al., 2013, J. Neurosci Fall-off increases with acoustic distance

98. Acoustic error is coded by auditory suppression

99. The efferent prediction may reflect an acoustic target or goal (not merely a “copy” of motor commands)

100. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

101. How does auditory feedback affect speech output? How do we detect deviations from what we intend to say? What is the corrective behavior when a deviation is detected? Understanding speech production via perception

102. Does error-like processing have behavioral consequences?

103. Behavioral consequences

107. Niziolek et al., 2013, J. Neurosci Behavioral consequences

108. If centering is partly driven by auditory feedback, it should: correlate with cortical responses to feedback decrease when you can’t hear yourself (i.e., in masking noise)

109. Centering correlates with suppression Niziolek et al., 2013, J. Neurosci

110. Masking noise reduces centering Niziolek et al., submitted quiet

111. Masking noise reduces centering quiet masking noise Niziolek et al., submitted

112. Masking noise reduces centering quiet masking noise Implications for hearing loss Niziolek et al., submitted

113. We detect deviations from an expected target Decreased suppression  ongoing error detection process (we make “errors” all the time!)

114. We correct our deviations before they become errors Decreased suppression  ongoing error detection process (we make “errors” all the time!) Centering  ongoing error correction process (partly mediated by feedback)

115. Ongoing & future directions: Assess error detection and correction capacities in patients with speech disorders

116. Speech error detection and correction in aphasia

117. detection impaired

118. Speech error detection and correction in aphasia centerperiphery F1 F2 detection impaired

119. Speech error detection and correction in aphasia centerperiphery F1 F2 detection impaired detection preserved

120. Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved

121. Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved sensory feedback training

122. Speech error detection and correction in aphasia center periphery F1 F2 detection impaired detection preserved sensory feedback training motor skill training

123. Ongoing & future directions: How task-specific is the perceived error?

124. Niziolek et al., in prep Redefining center and periphery

125. Sort by formant Niziolek et al., in prep Redefining center and periphery F1 F2

126. Sort by formant Niziolek et al., in prep Redefining center and periphery F1 F2 ✓

127. F0 Sort by formant Sort by pitch Redefining center and periphery F1 F2 ✓

128. F0 Sort by formant Sort by pitch Redefining center and periphery F1 F2 ✓ ✗

129. Multi-pitch task Niziolek et al., in prep eat spea k eat list en eat z xc z xc

130. Owen et al., 2012, NeuroImageNiziolek et al., in prep Source localization: right hemi

131. F0 Sort by pitch Pitch task auditory suppression

132. F0 Sort by pitch Pitch task auditory suppression ✓ Niziolek et al., in prep p = 0.048

133. F0 Sort by formant Sort by pitch Pitch task auditory suppression F1 F2 ✓ Niziolek et al., in prep p = 0.048

134. F0 Sort by formant Sort by pitch Pitch task auditory suppression F1 F2 ✗ ✓ Niziolek et al., in prep p = 0.27 (n.s.) p = 0.048

135. Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task

136. Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task ✓ ✓

137. Task-specific suppression F0 forman t pitch F1 F2 vowel task pitch task ✓ ✓ (n.s.) ✗

138. Pitch centering Niziolek et al., in prep

139. Pitch centering Niziolek et al., in prep

141. Understanding speech production via perception

142. Altered auditory feedback causes compensatory changes to speech output Understanding speech production via perception

143. Altered auditory feedback causes compensatory changes to speech output Degree of auditory cortical suppression allows us to detect deviations from what we intend to say Understanding speech production via perception

144. Altered auditory feedback causes compensatory changes to speech output Degree of auditory cortical suppression allows us to detect deviations from what we intend to say This suppression may underlie a corrective behavior that serves to bring speech back on track Understanding speech production via perception

145. Thank you! NIH F32DC011249 NIH R01DC010145 NSF BCS0926196 Frank Guenther John Houde Sri Nagarajan Eddie Chang Danielle Mizuiri Susanne Honma

146. Decrease in suppression in both speak and listen Niziolek et al., 2013, J. Neurosci

147. No systematic changes in suppression in right AC Niziolek et al., 2013, J. Neurosci

148. No systematic changes in suppression in right AC Niziolek et al., 2013, J. Neurosci

149. Formant variability

151. Anti-centering masking noise Niziolek et al., in prep

152. Direct cortical recordings Chang, Niziolek, et al., 2013, PNAS

153. Altered feedback Normal feedback Chang, Niziolek, et al., 2013, PNAS

154. “SIS falloff” across acoustic space Niziolek et al., 2013, J. Neurosci