Language Transfer of Audio Word2Vec: Learning Audio Segment Representations without Target Language Data Introduction: 4:00 Training: 2:30 Language transfer: 6:30 STD: 5:30 Speaker: Hung-Yi Lee Chia-Hao Shen, Janet Y. Sung, Hung-Yi Lee

Outline Introduction Training of Audio Word2vec Language Transfer Application to Query-by-example Spoken Term Detection (STD) Concluding Remarks 0:50

Audio Word to Vector Model Model Model Model word-level audio segment Model Model Model Model As its name implies Learn from lots of audio without annotation

Audio Word to Vector The audio segments corresponding to words with similar pronunciations are close to each other. dog never dog never dogs 1:00 never ever ever

Language Transfer Model ? Not included in training audio Language X Model unsupervised :39 ? Audio collection without annotation Can we train an universal model that can be applied even on the unknown languages?

Language Transfer Why consider universal model for all languages? If you want to apply the model on language X, why don’t you simply train a model by the audio of language X. Many audio files are code-switched across several different languages. Applied the model on audio data on the Internet in hundreds of languages. Audio collection for model training may not cover all the languages. It would be beneficial to have an universal model. 1:30 Star War C-3PO six million galactic languages Cannot Ewok language The Yuzzum language is closely associated with the Ewok language C-3PO first communicated with the Ewoks using the Yuzzum language, and gradually pieced together enough to learn the Ewok language sufficiently to be conversational. C-3PO learnt the Ewok language through observation. https://scifi.stackexchange.com/questions/124394/how-did-c3po-know-the-ewok-language The 3PO-series protocol droids are equipped with a TranLang III communications module. It comes with up to six million galactic languages - common and obscure, organic and inorganic - at purchase. It also possessed phonetic pattern analysers that provides the capability to learn and translate new languages not in its existing database.

Outline Introduction Training of Audio Word2vec Language Transfer Application to Query-by-example Spoken Term Detection Concluding Remarks

Audio Word to Vector Model Model Model Model word-level audio segment There are lots of segmentation approaches. Model Model Model Model In the following discussion, assume that we already get the segmentation.

Sequence-to-sequence Auto-encoder vector audio segment We use sequence-to-sequence auto-encoder here The training is unsupervised. RNN Encoder The vector we want (similar to the model used in speech summarization) x1 x2 x3 x4 acoustic features audio segment

Sequence-to-sequence Auto-encoder Input acoustic features x1 x2 x4 x3 The RNN encoder and decoder are jointly trained. y1 y2 y3 y4 RNN Encoder RNN Decoder x1 x2 x3 x4 acoustic features audio segment

What does machine learn? Text word to vector: Audio word to vector (phonetic information) 𝑉 𝑅𝑜𝑚𝑒 −𝑉 𝐼𝑡𝑎𝑙𝑦 +𝑉 𝐺𝑒𝑟𝑚𝑎𝑛𝑦 ≈𝑉 𝐵𝑒𝑟𝑙𝑖𝑛 𝑉 𝑘𝑖𝑛𝑔 −𝑉 𝑞𝑢𝑒𝑒𝑛 +𝑉 𝑎𝑢𝑛𝑡 ≈𝑉 𝑢𝑛𝑐𝑙𝑒 V( ) - V( ) + V( ) = V( ) 2:30 IT CAT CATS GIRL PEARL PEARLS GIRLS V( ) - V( ) + V( ) = V( ) IT CAT CATS ITS [Chung, Wu, Lee, Lee, Interspeech 16)

Outline Introduction Training of Audio Word2vec Language Transfer Application to Query-by-example Spoken Term Detection Concluding Remarks

Language Transfer We train sequence-to-sequence auto-encoder on a source language with a large amount of data. Training Apply RNN encoder on a new language. source language source language RNN Encoder RNN Decoder trained by source language Testing target language vector representation z for target language RNN Encoder

Experimental Setup Using 1-layer GRU as encoder and decoder Training with SGD. Initial learning rate was 1 and decayed with a factor of 0.95 every 500 batches. Acoustic features: 39-dim MFCC We used forced alignment with reference transcriptions to obtain word boundaries. The results is kind of oracle. We address this issue in another ICASSP paper.

Experimental Setup - Corpus English is our source language, while the other languages are target languages. English: Librispeech Training data: 2.2M word-level audio segments Testing data: 250K audio segments French, German, Czech and Spanish: GlobalPhone Testing data: 20K audio segments

Phonetic Information ever never RNN RNN Encoder Encoder Edit Distance between Phoneme sequences EH V ER N EH V ER ever never =1 RNN Encoder RNN Encoder Cosine Similarity

Phonetic Information Model trained on English, tested on English Variance Larger phoneme sequence edit distance, smaller cosine similarity Cosine Similarity The same pronunciation very different pronunciation Phoneme Sequence Edit Distance

Phonetic Information Model trained on English, tested on other languages Audio word2vec still capture phonetic information even though the model has never heard the language. Cosine Similarity Phoneme Sequence Edit Distance:

Visualization Visualizing embedding vectors of each word RNN Encoder day Project to 2-D RNN Encoder day average RNN Encoder day

Visualization Learn on English, and apply on the other languages 6:30 左邊法文 右邊德文 French German

Outline Introduction Training of Audio Word2vec Language Transfer Application to Query-by-example Spoken Term Detection (STD) Concluding Remarks

Query-by-example Spoken Term Detection “ICASSP” spoken query user “ICASSP” “ICASSP” We have to mention some evolutions here!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!????????????????????? Also known as unsupervised spoken term detection, zero-resource spoken content retrieval, etc. Evaluation program Spoken Content Compute similarity between spoken queries and audio files on acoustic level, and find the query term

Query-by-example Spoken Term Detection Segmental DTW [Zhang, ICASSP 10], Subsequence DTW [Anguera, ICME 13][Calvo, MediaEval 14] DTW for query-by-example Adding slope-constraints [Chan & Lee, Interspeech 10] The blue path is better than the green one. Spoken Query Utterance

Query-by-example Spoken Term Detection Much faster than DTW Audio archive divided into variable-length audio segments Off-line Audio Word to Vector Spoken Query Audio Word to Vector Similarity On-line Search Result

Query-by-Example STD Baseline: Naïve Encoder [Tu & Lee, ASRU 11] [I.-F. Chen, Interspeech 13] … … … … … … … …

Query-by-Example STD ─ English 1K queries to retrieve 250K audio segments DTW? Audio Word2vec Naïve Encoder DTW is not tractable here due to computing issue The evaluation measure is Mean Average Precision (MAP) (the larger, the better).

Query-by-Example STD ─ Language Transfer : Naïve Encoder : Audio word2vec by target language (4K segments) 1K queries to retrieve 20K audio segments FRE GRE CZE ESP The performance based on audio word2vec is poor with limited training data.

Query-by-Example STD ─ Language Transfer : Naïve Encoder : Audio word2vec by target language (4K segments) : Audio word2vec by English (2.2M segments) FRE GRE CZE ESP Audio word2vec learned by English can be directly applied on French and German.

Query-by-Example STD ─ Language Transfer : Naïve Encoder : Audio word2vec by target language (4K segments) : Audio word2vec by English (2.2M segments) : Audio word2vec by English + target (2K segments) 5:30 FRE GRE CZE ESP Fine-tuning English model by target language is helpful.

Outline Introduction Training of Audio Word2vec Language Transfer Application to Query-by-example Spoken Term Detection Concluding Remarks

Concluding Remarks We verify the capability of language transfer of Audio Word2Vec. Audio Word2Vec learned from English captures the phonetic information of other languages. In Query-by-example STD, Audio Word2Vec learned from English outperformed the baselines on French and German.

SEGMENTAL AUDIO WORD2VEC Session: Spoken Language Acquisition and Retrieval Time: Wednesday, April 18, 16:00 - 18:00