Phoneme Alignment. Slide 1 Phoneme Alignment based on Discriminative Learning Shai Shalev-Shwartz The Hebrew University, Jerusalem Joint work with Joseph Keshet, Hebrew University Yoram Singer, Google Dan Chazan, IBM

Phoneme Alignment. Slide 2 The Alignment Problem Phonetic transcription: Waveform: Text: /hh ae v ey tcl t eh s tcl t/ Have a test

Phoneme Alignment. Slide 3 The Alignment Problem Setting alignment function acoustic representation phonetic representation start-time of phoneme p i in x /hh ae v ey tcl t eh s tcl t/

Phoneme Alignment. Slide 4 Acoustic Representation Short-time Fourier Transform

Phoneme Alignment. Slide 5 Comparing Alignments e.g.

Phoneme Alignment. Slide 6  insensitive Cost  insensitivity region

Phoneme Alignment. Slide 7 A Discriminative Learning Approach Training set: Learning Algorithm Hypotheses class Alignment function:

Phoneme Alignment. Slide 8 Outline of Solution 1.Define the hypotheses class - constitutes the template of our alignment function: a.Map each possible alignment into vectors in an abstract vector-space b.Devise a projection in the vector-space which order alignments in accordance to their quality 2.Derive a simple online learning algorithm 3.Convert the Online Alg. to a Batch procedure with some formal guarantees

Phoneme Alignment. Slide 9 Feature “Primitives” for Alignment feature primitive for alignment Assessing the quality of a suggested alignment acoustic and phonetic representation suggested alignment

Phoneme Alignment. Slide 10 Feature Primitive I Cumulative spectral change across the boundaries

Phoneme Alignment. Slide 11 Feature Primitives I Cumulative spectral change across the boundaries

Phoneme Alignment. Slide 12 Feature Primitives II Cumulative confidence in the phoneme sequence frame based phoneme classifier is the confidence that phoneme was uttered at frame (Dekel, Keshet, Singer, ‘04) Learn a static frame-based phoneme classifier

Phoneme Alignment. Slide 13 Feature Primitive III Phoneme duration model - average length of phoneme - standard deviation of the length of phoneme

Phoneme Alignment. Slide 14 Feature Primitive IV Spectogram at different rates of articulation (Pickett, 1980) Speaking-rate (“dynamics”)

Phoneme Alignment. Slide 15 Feature Functions for Alignment correct alignment slightly incorrect alignment grossly incorrect alignment Mapping all possible alignments into a vector space

Phoneme Alignment. Slide 16 Main Solution Principle grossly incorrect alignment correct alignment slightly incorrect alignment Find a linear projection that ranks alignments according to their quality

Phoneme Alignment. Slide 17 slightly incorrect alignment Main Solution Principle (cont.) example of low confidence projection correct alignment grossly incorrect alignment

Phoneme Alignment. Slide 18 slightly incorrect alignment Main Solution Principle (cont.) example of incorrect projection correct alignment grossly incorrect alignment

Phoneme Alignment. Slide 19 Online Learning Algorithm Hypotheses class Cumulative cost

Phoneme Alignment. Slide 20 Online Learning For Receive an instance Predict Receive true alignment and Pay cost If Set Update

Phoneme Alignment. Slide 21 Converting from Online to Batch Run online algorithm on the training set and generate w 1,…,w M Small online error  exists w 2 {w 1,…,w M } whose generalization error is low (Cesa-bianchi et al.) Choose w 2 {w 1,…,w M } which minimizes the error on a fresh validation set

Phoneme Alignment. Slide 22 Algorithmic aspects Running-time: If the “inference”,, can be performed in polynomial time (e.g. dynamic programming), then the entire algorithm operates in polynomial time as well. Worst case analysis for Online Learning: For any competitor u, Generalization error Online-to-batch conversion guarantees that: low online error  low generalization error

Phoneme Alignment. Slide 23 Experiments TIMIT corpus Phoneme representation: 48 phonemes (Lee & Hon, 1989) Acoustic Representation: MFCC+∆+∆∆ (ETSI standard) TIMIT training set: 500 utterances for training a frame classifier 3096 utterances for learning alignment function 100 utterances used for validation

Phoneme Alignment. Slide 24 Alternative Approaches Brugnara, Falavigna & Omologo, Automatic segmentation and labeling of speech based on HMM, Hosom, Automatic phoneme alignment on acoustic-phonetic modeling, Toledano, Gomez & Grande, Automatic Phoneme Alignment, 2003.

Phoneme Alignment. Slide 25 Results Brugnara, Falavigna and Omologo, “Automatic segmentation and labling of speech based on Hidden Markov Models”, Speech Comm., 12 (1993) Training size Test set t < 10 ms t < 20 ms t < 30 mst < 40 ms Discrim. Algo. 650 or core79.7%92.1%96.2%98.1% Brugnara et al core75.3%88.9%94.4%97.1% Discrim. Algo. 650 or entire80.0%92.3%96.4%98.2% Brugnara et al entire74.6%88.8%94.1%96.8%

Phoneme Alignment. Slide 26 Current and Future Work Discriminative learning methods for: Whole phoneme sequence classification 64% (ours) vs. 59% (HMM – IDIAP Torch3) Results without normalization of silences etc. Small vocabulary continuous speech recognition Segmentation of utterances to speakers Full online learning setting: real-time adaptation to Speaker/environment changes