Hierarchical Multi-Stream Posterior Based Speech Recognition System Hamed Ketabdar, Herve Bourlard and Samy Bengio IDIAP Research Institute, Martigny, Switzerland MLMI 2005 Workshop, Edinburgh, UK gsggdgdfsgdfsg

Hierarchical Multi-stream Posterior Based Speech Recognition System Main idea (1) Estimating more informative posteriors by taking into account: Prior knowledge about the problem (e.g. phone transition probabilities, phone minimum durations) Contextual information Multiple stream of features Hierarchical Multi-stream Posterior Based Speech Recognition System

Hierarchical Multi-stream Posterior Based Speech Recognition System Main idea (2) Principled approach towards using posteriors in hierarchical structures Dividing the problem into multi-level sub problems. In each layer: Integrating relevant prior and contextual information Combining different kind of features Hierarchical Multi-stream Posterior Based Speech Recognition System

Posterior based speech recognition systems Posterior estimation: Based on local features Without taking into account prior knowledge about the problem. Posteriors usage: As features for a standard HMM/GMM recognizer (e.g. Tandem) As local scores for a decoder (e.g. hybrid HMM/ANN system). You should say that I am talking about phonemes but this approch and discussion and issue is genarl and can be applied to many cases of pattern recognition problems. Say that, As a review of state of the art posterior based speech recognition systems. The posteriors are usually estimated using MLPs. Most of the time a feature vector which is representing a slice of the MLP only see that slice of speech sifnal. It also does not know any thing about prior knowledge related to the problem, for example what is the minium duration which a phonme can appear, legl sequences of phonemes, etc. It just can see the current frame possibly concatineted with a small number of neighbor frames and decide about phoneme posteriors based on this knowledge. Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Prior knowledge, Contextual information Information about a phoneme is extended over time, therefore contextual information should be useful. There are usually some prior knowledge and assumptions about the problem. For example, transition between some phonemes cannot happen or it is less probable, lexical information Question: Is there any way to introduce prior and contextual information in posterior estimation? Make an example of the illegal sequence of phonemes. Say at the end, in order to address this problem, and then go to the next slide … you should say normally MLPs can not take into account this information about prior knowledeg or just they can concatinate feature vectors to some how take into account contextual information. Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

“Gamma” posterior estimation The idea: Estimate posteriors through an HMM, based on “Gamma” state posterior definition: Posterior estimation taking into account: Prior knowledge encoded in the model M Whole sequence contextual information You should say something about integration. You should say also something Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

“Gamma” posterior estimation The “gamma” posterior can be written based on forward and backward HMM recursions: Forward and Backward recursions: functions of observation likelihoods (or scaled likelihoods) and state transition probabilities. You should say: and in fact the transition probability term is the place by which we can introduce Prior knowledge in the form of topological constraints Formula for alpha (mention that the emission probability term can be likelihood or scaled likelihood and can be estimated using GMMs or MLPs. For furtur details refer to Ref1 Ref2 Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Example: introducing prior knowledge Phone posteriors estimated by MLP “Gamma” phone posteriors Prior knowledge: Minimum phone duration is 3 Hierarchical Multi-stream Posterior Based Speech Recognition System

Multi-stream “gamma” posterior The extension of the gamma posterior estimation idea to the multi-stream case. We define: Estimating more informative posteriors by Combining multiple feature streams having complementary information Taking into account prior knowledge (encoded in the Model M) and contextual information (whole observation sequence) You should say that the multi-stream case is the main concern in this talk, you should same recursion and usual HMM assumptions Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Multi-stream “gamma” posterior estimation We define multi-stream forward and backward recursions as follows: Multi-stream forward and backward recursions can be written in terms of single stream forward and backward recursions (having some independence assumptions) The multi-stream gamma posterior can be written based on multi-stream forward and backward recursions: Some details about the multistream posterior estimation. For multi-stream posterior estimation we need to estimate Here or in the next slide, you should say: “Well now we have a theretical frame work for combinig the streams and also posterior estimation taking into account contextual information and prior knowledge You should mention that it can be rewritten based on single stream forward and backward recursions having the independence assumption You should also mention about symbols and terms and definitions and ext Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Experiments with multi-stream posteriors Which streams to combine? The streams which are combined should have some complementary information. Candidates: TempoRAl Pattern (TRAP) features and PLP cepstral features. Here I explain, the first results of the multi-stream posterior estimation method. The first step, in building the system is to decide about the feature streams. Different feature stream should have some complementary information. TRAP and PLP features are good candidates since they carry some complementary information. The other candidates can be static and dynamic features like plp and delta plp features which carry some kind of complementary information. Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Hierarchical Multi-stream Posterior Based Speech Recognition System Feature streams PLP cepstral features: Showing whole spectrum for a limited period of time. TempoRAl Pattern (TRAP) features: Representing critical band spectral energies over long time. Hierarchical Multi-stream Posterior Based Speech Recognition System

Hierarchical multi-stream posterior based speech recognition system

Hierarchical Multi-stream Posterior Based Speech Recognition System Results CTS database OGI digits database Features WER PLP posteriors 48.7% TRAP posteriors 55.1% Inverse entropy combination Multi-stream gamma posteriors 46.8% Features WER PLP posteriors 3.6% TRAP posteriors 4.8% Inverse entropy combination 3.5% Multi-stream gamma posteriors 2.9% Databases: OGI digits: Recognition of continuous digits, 11 words Reduced vocabulary version of DARPA Conversational Telephone Speech (CTS) task, 1000 words Tel about the specifications of the databse, number of utterance, time and etc for each database. Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg

Hierarchical Multi-stream Posterior Based Speech Recognition System Conclusions Proposing a theoretical framework for Multiple feature stream combination Posterior estimation taking into account Prior knowledge Contextual information Design a hierarchical ASR system based on the multi-stream posterior estimation method You should say that till now, I estmated emition probabilities with MLPs. I can furthur work to estimate emission likelihoods by GMMs. The MLP uvery low sually gives a kind of yes/no answer. In wihich the posterios are eighter very high or low even for wrong cases. Hierarchical Multi-stream Posterior Based Speech Recognition System gsggdgdfsgdfsg