LYU0103 Speech Recognition Techniques for Digital Video Library Supervisor : Prof Michael R. Lyu Students: Gao Zheng Hong Lei Mo

Outline of Presentation Project objectives Project objectives ViaVoice recognition experiments ViaVoice recognition experiments Speech information processor Speech information processor Audio information retrieval Audio information retrieval Summary Summary

Our Project Objectives Speech recognition Speech recognition Audio information retrieval Audio information retrieval

Last Term’s Work Extract audio channel (stereo 44.1 kHz) from mpeg video files into wave files (mono 22 kHz) Extract audio channel (stereo 44.1 kHz) from mpeg video files into wave files (mono 22 kHz) Segment the wave files into sentences by detecting its frame energy Segment the wave files into sentences by detecting its frame energy Realtime dictation with IBM ViaVoice (ViaVoice is a speech recognition engine developed by IBM) Realtime dictation with IBM ViaVoice (ViaVoice is a speech recognition engine developed by IBM) Developed a visual training tool Developed a visual training tool

Visual Training Tool Video Window; Dictation Window; Text Editor

IBM ViaVoice Experiments Employed 7 student helpers Employed 7 student helpers Produce transcripts of 77 news video clips Produce transcripts of 77 news video clips Four experiments: Four experiments:  Baseline measurement  Trained model measurement  Slow down measurement  Indoor news measurement

Baseline Measurement To measure the ViaVoice recognition accuracy using TVB news video To measure the ViaVoice recognition accuracy using TVB news video Testing set: 10 video clips Testing set: 10 video clips The segmented wav files are dictated The segmented wav files are dictated Employ the hidden Markov model toolkit (HTK) to examine the accuracy Employ the hidden Markov model toolkit (HTK) to examine the accuracy

Trained Model Measurement To measure the accuracy of ViaVoice, trained by its correctly recognized words To measure the accuracy of ViaVoice, trained by its correctly recognized words 10 videos clips are segmented and dictated 10 videos clips are segmented and dictated The correctly dictated words of training set are used to train the ViaVoice by the SMAPI function SmWordCorrection The correctly dictated words of training set are used to train the ViaVoice by the SMAPI function SmWordCorrection Repeat the procedures of “baseline measurement” after training to get the recognition performance Repeat the procedures of “baseline measurement” after training to get the recognition performance Repeat the procedures of using 20 videos clips Repeat the procedures of using 20 videos clips

Slow Down Measurement Investigate the effect of slowing down the audio channel Investigate the effect of slowing down the audio channel Resample the segment wave files in the testing set by the ratio of 1.05, 1.1, 1.15, 1.2, 1.3, 1.4, and 1.6 Resample the segment wave files in the testing set by the ratio of 1.05, 1.1, 1.15, 1.2, 1.3, 1.4, and 1.6 Repeat the procedures of “baseline measurement” Repeat the procedures of “baseline measurement”

Indoor News Measurement Eliminate the effect of noise Eliminate the effect of noise Select the indoor news reporter sentence Select the indoor news reporter sentence Dictate the test set using untrained model Dictate the test set using untrained model Repeat the procedure using trained model Repeat the procedure using trained model

Experimental Results Experiment Accuracy (Max. performance) Baseline25.27% Trained Model 25.87% (with 20 video trained) Slow Speech 25.67% (max. at ratio = 1.15) Indoor Speech (untrained model) 35.22% Indoor Speech (trained model) 36.31% (with 20 video trained) Overall Recognition Results (ViaVoice, TVB News )

Experimental Result Cont. Trained Video Number Untrained 10 videos 20 videos Accuracy25.27%25.82%25.87% Ratio Accuracy (%) Result of trained model with different number of training videos Result of using different slow down ratio

Analysis of Experimental Result Trained model: about 1% accuracy improvement Trained model: about 1% accuracy improvement Slowing down speeches: about 1% accuracy improvement Slowing down speeches: about 1% accuracy improvement Indoor speeches are recognized much better Indoor speeches are recognized much better Mandarin: estimated baseline accuracy is about 70 % ( >> Cantonese) Mandarin: estimated baseline accuracy is about 70 % ( >> Cantonese)

Experiment Conclusions Four reasons for low accuracy Four reasons for low accuracy  Language model mismatch  Voice channel mismatch  The broadcast is very fast and some characters are not so clear  The voice of video clips is too loud The first two reasons are the most critical ones The first two reasons are the most critical ones

Speech Recognition Approach We cannot do much acoustic model training with the ViaVoice API We cannot do much acoustic model training with the ViaVoice API Training is speaker dependent Training is speaker dependent Great difference between the news audio and the training speech for ViaVoice Great difference between the news audio and the training speech for ViaVoice The tool to adapt acoustic model is not currently available The tool to adapt acoustic model is not currently available Manually editing is necessary for producing correct subtitles Manually editing is necessary for producing correct subtitles

Speech Information Processor (SIP) Media player, Text editor, Audio information panel

Main Features Media playback Media playback Real-time dictation Real-time dictation Word time information Word time information Dynamic recognition text editing Dynamic recognition text editing Audio scene change detection Audio scene change detection Audio segments classification Audio segments classification Gender classification Gender classification

System Chart

Timing Information Retrieval Use ViaVoice Speech Manager API (SMAPI) Use ViaVoice Speech Manager API (SMAPI) Asynchronous callback Asynchronous callback The recognized text is organized in a basic unit called “firm word” The recognized text is organized in a basic unit called “firm word” SIP builds an index to store the position and time of firm words SIP builds an index to store the position and time of firm words Highlight corresponding firm words during video playback Highlight corresponding firm words during video playback

Highlight words during playback

Dynamic Index Alignment While editing recognized result, firm word structure might be changed While editing recognized result, firm word structure might be changed Word index need to be updated accordingly Word index need to be updated accordingly SIP captures WM_CHAR event of the text editor SIP captures WM_CHAR event of the text editor Then search for the modified words, and update the corresponding entries in the index Then search for the modified words, and update the corresponding entries in the index In practice, binary search provides good responding time In practice, binary search provides good responding time

Time Index Alignment Example Before EditingEditing After Editing

Audio Information Panel The entire clip is divided into segments separated by audio scene changes The entire clip is divided into segments separated by audio scene changes SIP classifies the segments into three categories, male, female, and non-speech SIP classifies the segments into three categories, male, female, and non-speech Click a segment to preview it Click a segment to preview it

Audio Information Retrieval

Detection of Audio Scene Change -- Motivations Segments of different properties can be handled differently Segments of different properties can be handled differently Apply unsupervised learning to different clusters Apply unsupervised learning to different clusters Assistant tool to video scene change detection Assistant tool to video scene change detection

Bayesian Information Criterion (BIC) Gaussian Distribution—model input stream Gaussian Distribution—model input stream Maximum Likelihood—detect turns Maximum Likelihood—detect turns BIC– make a decision BIC– make a decision

Principle of BIC Bayesian information criterion (BIC) is a likelihood criterion Bayesian information criterion (BIC) is a likelihood criterion The main principle is to penalize the system by the model complexity The main principle is to penalize the system by the model complexity

Detection of a single point change using BIC H 0 :x 1,x 2 …x N ~N(μ,Σ) H 1 :x 1,x 2 …x i ~N(μ 1,Σ 1 ), H 2 :x i+1,x i+2 …x N ~N(μ 2,Σ 2 ), The maximum likelihood ratio is defined as: R(I)=Nlog| Σ|-N 1 log| Σ 1 |-N 2 log| Σ 2 |

Detection of a single point change using BIC The difference between the BIC values of two models can be expressed as: The difference between the BIC values of two models can be expressed as: BIC(I) = R(I) – λP P=(1/2)(d+(1/2d(d+1)))logN If BIC value>0, detection of scene change If BIC value>0, detection of scene change

Detection of multiple point changes by BIC a. Initialize the interval [a, b] with a=1, b=2 a. Initialize the interval [a, b] with a=1, b=2 b. Detect if there is one changing point in interval [a, b] using BIC b. Detect if there is one changing point in interval [a, b] using BIC c. If (there is no change in [a, b]) c. If (there is no change in [a, b]) let b= b + 1 else let t be the changing point detected assign a = t +1; b = a+1; end d. go to step (b) if necessary

Advantages of BIC approach Robustness Robustness Thresholding-free Thresholding-free Optimality Optimality

Comparison of different algorithms

Gender Classification: Motivation and Purpose Allowing different speech analysis algorithms for each gender Allowing different speech analysis algorithms for each gender Facilitating speech recognition by cutting the search space in half Facilitating speech recognition by cutting the search space in half Helping us to build gender-dependent recognition model and better training of the system Helping us to build gender-dependent recognition model and better training of the system

Gender Classification MaleFemale

Speech/Non-Speech Classification Motivation Motivation One method we used : pitch tracking One method we used : pitch tracking

Speech/Non-Speech classification SpeechNon-Speech

Summary ViaVoice training experiments ViaVoice training experiments Speech recognition editing Speech recognition editing Dynamic index alignment Dynamic index alignment Audio scene change detection Audio scene change detection Speech classification Speech classification Integrated the above functions into a speech processor Integrated the above functions into a speech processor

Q & A