. cmsc726: HMMs material from: slides from Sebastian Thrun, and Yair Weiss

Outline u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

Audio Spectrum Audio Spectrum of the Song of the Prothonotary Warbler

Bird Sounds Chestnut-sided WarblerProthonotary Warbler

Questions One Could Ask u What bird is this? u How will the song continue? u Is this bird sick? u What phases does this song have?  Time series classification  Time series prediction  Outlier detection  Time series segmentation

Other Sound Samples

Another Time Series Problem Intel Cisco General Electric Microsoft

Questions One Could Ask u Will the stock go up or down? u What type stock is this (eg, risky)? u Is the behavior abnormal?  Time series prediction  Time series classification  Outlier detection

Music Analysis

Questions One Could Ask u Is this Beethoven or Bach? u Can we compose more of that? u Can we segment the piece into themes?  Time series classification  Time series prediction/generation  Time series segmentation

CiteSeer.Com (Citation Index) Dave Rumelhart Takeo KanadeTom Mitchell Raj ReddyJim Morris

Questions One Could Ask u Shall UMD give tenure? u Shall UMD hire? u Shall UMD fire?  Time series classification  Time series prediction  Outlier detection Disclaimer: This is a joke!

The Real Question u How do we model these problems? u How do we formulate these questions as a inference/learning problems?

Outline For Today u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

Weather: A Markov Model Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75%5%

Ingredients of a Markov Model u States: u State transition probabilities: u Initial state distribution: Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75% 5%

Ingredients of Our Markov Model u States: u State transition probabilities: u Initial state distribution: Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75% 5%

Probability of a Time Series u Given: u What is the probability of this series?

Outline For Today u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

Hidden Markov Models Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75%5% Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75% 5% 60% 10% 30% 65% 5% 30% 50% 0% 50% NOT OBSERVABLE

Ingredients of an HMM u States: u State transition probabilities: u Initial state distribution: Observations: Observation probabilities:

Ingredients of Our HMM u States: u Observations: u State transition probabilities: u Initial state distribution: u Observation probabilities:

Probability of a Time Series u Given: u What is the probability of this series?

Calculating Data Likelihood u Problem: exponential in time u Is there a more efficient way?

The Forward Algorithm (1) S2S2 S3S3 S1S1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 …

Question u Does this solve our problem of calculating ? YESNO Count

Answer u And the answer is… Yes!

Exercise u What is the probability of observing AB? a. Initial state s 1 : b. Initial state chosen at random: s2s2 s1s B 0.7 A 0.2 B 0.8 A 0.2  (0.4   0.7) =  (0.5  0.3  (0.3   0.8)) =

Next Question: What is the probability that the state at time t was S i ? u Can we answer this? S2S2 S3S3 S1S1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 … No!

The Backward Algorithm (2) S2S2 S3S3 S1S1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 …

The Forward-Backward Algorithm (3) S2S2 S3S3 S1S1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 S2S2 S3S3 S1S1 O2O2 O3O3 O1O1 …

Summary (So Far)  For a given HMM, we can compute

Finding the best state sequence We would like to the most likely path (and not just the most likely state at each time slice) The Viterbi algorithm is an efficient method for finding the MPE: and we to reconstruct the path:

Outline u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

Hidden Markov Models Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75%5% Sunny Rainy Snowy 80% 15% 5% 60% 2% 38% 20% 75% 5% 60% 10% 30% 65% 5% 30% 50% 0% 50% NOT OBSERVABLE

Summary So Far u HMMs: generative probabilistic models of time series with hidden state u Forward-backward algorithm: Efficient algorithm for calculating

What about learning?

EM Problem: Find HMM  that makes data most likely  E-Step: Computefor given  M-Step: Compute new under these expectations (this is now a Markov model)

E-Step u Calculate using the forward-backward algorithm, for fixed model

The M Step: generate =( , a, b)

Summary (Learning)  Given observation sequence O  Guess initial model  u Iterate: Calculate expected times in state S i at time t (and in S j at time t  ) using forward-backward algorithm Find new model  by frequency counts

Outline For Today u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

Three Problems u What bird is this? u How will the song continue? u Is this bird abnormal?  Time series classification  Time series prediction  Outlier detection

Time Series Classification  Train one HMM l for each bird l  Given time series O, calculate

Outlier Detection  Train HMM  Given time series O, calculate probability u If abnormally low, raise flag u If high, raise flag

Time Series Prediction  Train HMM  Given time series O, calculate distribution over final state (via  ) and ‘hallucinate’ new states and observations according to a, b

Typical HMM in Speech Recognition 20-dim frequency space clustered using EM Use Bayes rule + Viterbi for classification Linear HMM representing one phoneme [Rabiner 86] + everyone else

Typical HMM in Robotics [Blake/Isard 98, Fox/Dellaert et al 99]

Problems with HMMs u Zero probabilities l Training sequence: AAABBBAAA l Test sequence: AAABBBCAAA u Finding “right” number of states, right structure u Numerical instabilities

Outline u Time Series u Markov Models u Hidden Markov Models u Learning HMMs with EM u Applying HMMs u Summary

HMMs: Main Lessons u HMMs: Generative probabilistic models of time series (with hidden state) u Forward-Backward: Algorithm for computing probabilities over hidden states u Learning models: EM, iterates estimation of hidden state and model fitting u Extremely practical, best known methods in speech, computer vision, robotics, … u Numerous extensions exist (continuous observations, states; factorial HMMs, controllable HMMs=POMDPs, …)