1. Conditional Random Fields for ASR Jeremy Morris July 25, 2006

2. Overview ► Problem Statement (Motivation) ► Conditional Random Fields ► Experiments  Attribute Selection  Experimental Setup ► Results ► Future Work

3. Problem Statement ► ► Developed as part of the ASAT Project   (Automatic Speech Attribute Transcription) ► ► Goal: Develop a system for bottom-up speech recognition using 'speech attributes'

4. Speech Attributes? ► ► Any information that could be useful for recognizing the spoken language   Phonetic attributes   Speaker attributes (gender, age, etc.)   Any other useful attributes that could be used for speech recognition   Note that there is no guarantee that attributes will be independent of each other ► ► One part of this project is to explore ways to create a framework for easily combining new features for experimental purposes /d/ manner: stop place of artic: dental voicing: voiced /t/ manner: stop place of artic: dental voicing: unvoiced /iy/ height: high backness: front roundness: nonround

5. Evidence Combination ► ► Two basic ways to build hypotheses hyp data hyp data Top Down Generate a hypothesis See if the data fits the hypothesis Bottom Up Examine the data Search for a hypothesis that fits

6. Top Down ► ► Traditional Automated Speech Recogintion Systems (ASR) use a top- down approach (HMMs)   Hypothesis is the phone we are predicting   Data is some encoding of the acoustic speech signal   A likelihood of the signal given the phone label is learned from data   A prior probability for the phone label is learned from the data  These are combined through Bayes Rule to give us the posterior probability /iy/ X P(/iy/) P(X|/iy/)

7. Bottom Up ► ► Bottom-up models have the same high-level goal – determine the label from the observation   But instead of a likelihood, the posterior probability is learned from the data ► ► Neural Networks have been used to learn these probabilities /iy/ X P(/iy/|X)

8. Speech is a Sequence ► ► Speech is not a single, independent event   It is a combination of multiple events over time ► ► A model to recognize spoken language should take into account dependencies across time /k/ /iy/

9. Speech is a Sequence ► ► A top down (generative) model can be extended into a time sequence as a Hidden Markov Model (HMM)   Now our likelihood of the data is over the entire sequence instead of a single phone /k/ /iy/ XXXXX

10. Speech is a Sequence ► ► Tandem is a method for using evidence bottom up (discriminative)   Hypothesis output of Neural Network is used to train an HMM   Not a pure discriminative method, but a combination of generative and discriminative methods /k//iy/ YYY XXX

11. Bottom up Modelling ► ► The idea is to have a system that combines evidence layer by layer   Speech attributes contribute to phone attribute detection   Phone attributes contribute to “syllable” attribute detection, and so on ► ► Each layer combines information from previous layers to form its hypotheses   We want to do this probabalistically – no hard decisions   Note that there is no guarantee of independence among the observed speech features – in fact, they are often very dependent.

12. Conditional Random Fields ► ► A form of discriminative modelling   Has been used successfully in various domains such as part of speech tagging and other Natural Language Processing tasks ► ► Processes evidence bottom-up   Combines multiple features of the data   Builds the probability P( sequence | data)

13. Conditional Random Fields ► ► CRFs are based on the idea of Markov Random Fields   Modelled as an undirected graph connecting labels with observations   Observations in a CRF are not random variables /k/ /iy/ XXXXX Transition functions add associations between transitions from one label to another State functions help determine the identity of the state

14. Conditional Random Fields State Feature Function f([x is stop], /t/) One possible state feature function For our attributes and labels State Feature Weight λ=10 One possible weight value for this state feature (Strong) Transition Feature Function g(x, /iy/,/k/) One possible transition feature function Indicates /k/ followed by /iy/ Transition Feature Weight μ=4 One possible weight value for this transition feature ► Hammersley-Clifford Theorem states that a random field is an MRF iff it can be described in the above form  The exponential is the sum of the clique potentials of the undirected graph

15. Conditional Random Fields ► ► Conceptual Overview   Each attribute of the data we are trying to model fits into a feature function that associates the attribute and a possible label ► ► A positive value if the attribute appears in the data ► ► A zero value if the attribute is not in the data   Each feature function carries a weight that gives the strength of that feature function for the proposed label ► ► High positive weights indicate a good association between the feature and the proposed label ► ► High negative weights indicate a negative association between the feature and the proposed label ► ► Weights close to zero indicate the feature has little or no impact on the identity of the label

16. Experiments ► ► Goal: Implement a Conditional Random Field Model on ASAT-style data   Perform phone recognition   Compare results to those obtained via a Tandem system ► ► Experimental Data   TIMIT read speech corpus   Moderate-sized corpus of clean, prompted speech, complete with phonetic-level transcriptions

17. Attribute Selection ► ► Attribute Detectors   ICSI QuickNet Neural Networks ► ► Two different types of attributes   Phonological feature detectors ► ► Place, Manner, Voicing, Vowel Height, Backness, etc. ► ► Features are grouped into eight classes, with each class having a variable number of possible values based on the IPA phonetic chart   Phone detectors ► ► Neural networks output based on the phone labels – one output per label   Classifiers were applied to 2960 utterances from the TIMIT training set

18. Experimental Setup ► ► Code built on the Java CRF toolkit on Sourceforge   http://crf.sourceforge.net   Performs training to maximize the log-likelihood of the training set with respect to the model   Uses a Limited Memory BGFS algorithm to minimize the gradient of the log-likelihood ► ► For CRF models, maximizing the log-likelihood of the empirical distribution of the data as predicted by the model is the same as maximizing the entropy (Berger et. al.)

19. Experimental Setup ► ► Output from the Neural Nets are themselves treated as feature functions for the observed sequence – each attribute/label combination gives us a value for one feature function   Note that this makes the feature functions non- binary features.

20. ResultsModel Phone Accuracy Phone Correct Tandem [1] (phones) 60.48%63.30% Tandem [3] (phones) 67.32%73.81% CRF [1] (phones) 66.89%68.49% Tandem [1] (features) 61.48%63.50% Tandem [3] (features) 66.69%72.52% CRF [1] (features) 65.29%66.81% Tandem [1] (phones/feas) 61.78%63.68% Tandem [3] (phones/feas) 67.96%73.40% CRF (phones/feas) 68.00%69.58%

21. Future Work ► More features  What kinds of features can we add to improve our transitions? ► Tuning  HMM model has parameters that can be tuned for better performance – can we tweak the CRF to do something similar? ► Word recogntion  How does this model do at the full word recognition level, instead of just phones ► Other corpora  Can we extend this method beyond TIMIT to different types of corpora? (e.g. WSJ)