Modulation Spectrum Factorization for Robust Speech Recognition Wen-Yi Chu 1, Jeih-weih Hung 2 and Berlin Chen 1 Presenter : 張庭豪

2 Outline Introduction Nonnegative Matrix Factorization (NMF) Updating the Modulation Spectrum via NMF Experimental Setup Experimental Results and Discussions Conclusion and Future Work

Introduction NMF is a recently developed method for finding a linear and non- subtractive combination scheme to extract important ingredients that can correspond better with Most of the useful linguistic information is encapsulated in the modulation frequency components between 1 Hz and 16 Hz,with the dominant component centering around 4 Hz. We attempt to refine the features in the magnitude part of modulation spectra (which is always real and non-negative) via the technique of non-negative matrix factorization 3

Nonnegative Matrix Factorization (1/3) Nonnegative matrix factorization (NMF) is a subspace method that approximates data with an additive and linear combination of nonnegative components (or basis vectors) Given a nonnegative data matrix, NMF computes another two nonnegative matrices and such that V ≈ WH –r<< L and r<< M to ensure efficient encoding 4 ≈ X VWH (tall and thin) (short and wide) (basis)(encoding)

Nonnegative Matrix Factorization (2/3) V ≈ WH To find an approximate factorization as V ≈ WH, the cost function is defined as: With an initial (random) guess of W and H, the following multiplicative updating rule is employed to achieve a local minimum of L: and 5

Nonnegative Matrix Factorization (3/3) Procedures: 6

First, the time sequence x[n] for each utterance in the training set is converted to its spectrum x[k] via a 2L point DFT. Since the property of conjugate symmetry, only the first L+1 points of X[k] is reserved, and their magnitude parts (which are always nonnegative) form each column of the data matrix V. Accordingly, if the training set consists of M utterances, then V has M columns. Given the data matrix V and a chosen number r, we obtain the two nonnegative matrices W and H. 7 Updating the Modulation Spectrum via NMF(1/3)

Updating the Modulation Spectrum via NMF(2/3) The fixed W comes directly from the previous step, and the encoding vector h can be obtained via the updating rule. The vector V is a linearcombination of the basis vectors involved in W, which is created via the clean utterances. Therefore we expect that the vector V, representing the new magnitude spectrum, can highlight the important information for speech recognition and alleviate the effect of noise from the original V. A 2L-point inverse DFT is performed on the new modulation spectrum (with the conjugate symmetric last-half part being appended), which consists of the updated magnitude parts and the original phase parts, to obtain the new time sequence. 8

Updating the Modulation Spectrum via NMF(3/3) The basis spectra vectors of the MFCC c1 (r = 10) –Localized and sparse characteristics, which coincide with the fact that NMF often learns a parts-based representation of data –Capable of distilling or emphasizing the lower modulation frequency components of the speech features, which contains more speech information 9 (a) original MFCC c1 (b) MVN-processed MFCC c1

Experimental Setup –Feature type : 39-dimensional MFCC –The number of basis vectors, r, is varied from 5 to 20 –The DFT size :

Experimental Results (1/4) 11

Experimental Results (2/4) NMF : r = 5 NMF + CMVN : r = 15 12

Experimental Results (3/4) The power spectral density (PSD) curves of the feature streams at different signal-to-noise ratios (SNRs): 13 original c1 NMF-processed c1 noise causes significant mismatch in PSD of MFCC NMF reduces the PSD mismatch

Experimental Results (4/4) The power spectral density (PSD) curves of the feature streams at different signal-to-noise ratios (SNRs): 14 MVN-processed c1c1 processed by MVN and NMF MVN reduces the PSD mismatch more in the low frequency region NMF further alleviates the high-frequency PSD mismatch in MVN features

Conclusion and Future Work We have presented a novel use of NMF for deriving noise robust speech features –The basis spectra via NMF correspond well with the intuitive notion of the important components of modulation frequency. –NMF benefits both the plain and MVN-processed MFCC in recognition accuracy As to future work, we envisage the following two directions: (1)To further process the encoding vector H in the mapping process of NMF to give better recognition accuracy (2) To examine if some variants or extensions of NMF, such as probabilistic latent semantic analysis (PLSA), and other compressive sensing methods can further enhance the modulation spectrum 15