1 Hidden Markov Model: Overview and Applications in MIR MUMT 611, March 2005 Paul Kolesnik MUMT 611, March 2005 Paul Kolesnik

2 Contents  Introduction to HMM  Overview of Publications  Conclusion  Introduction to HMM  Overview of Publications  Conclusion

3 Introduction  Definition  A structure that is used to statistically characterize the behavior of sequences of event observations  Extension of a model known as Markov chains  “A double stochastic process with an underlying stochastic process which is not observable, but can only be observed through another set of stochastic process that produces the sequence of observed symbols” (Rabiner and Huang 1986)  Concepts  Any observable sequence can be represented as a succession of states, with each state representing a grouped portion of the observation values and containing its features in a statistical form  Definition  A structure that is used to statistically characterize the behavior of sequences of event observations  Extension of a model known as Markov chains  “A double stochastic process with an underlying stochastic process which is not observable, but can only be observed through another set of stochastic process that produces the sequence of observed symbols” (Rabiner and Huang 1986)  Concepts  Any observable sequence can be represented as a succession of states, with each state representing a grouped portion of the observation values and containing its features in a statistical form

4 Introduction  Concepts (ctd.)  HMM keeps track of  What state will the sequence start in  What state-to-state transitions are likely to take place  What values are likely to occur in each state  Corresponding parameters  Array of initial state probabilities  Matrix of state-to-state transitional probabilities  Matrix of state output probabilities  = ( , A, B)  Concepts (ctd.)  HMM keeps track of  What state will the sequence start in  What state-to-state transitions are likely to take place  What values are likely to occur in each state  Corresponding parameters  Array of initial state probabilities  Matrix of state-to-state transitional probabilities  Matrix of state output probabilities  = ( , A, B)

5 Introduction Markov Model Example. - x — States of the Markov model - a — Transition probabilities - b — Output probabilities - y — Observable outputs

6 Introduction  Three Main HMM Problems  Recognition  given an observation sequence and a Hidden Markov Model, calculate the probability that the model would produce this observation sequence.  Uncovering (Viterbi)  given an observation sequence and a Hidden Markov Model, calculate the optimal sequence of states that would maximize the likelihood of the HMM producing the observation.  Learning / Training  given an observation sequence (or a set of observation sequences and a Hidden Markov Model, adjust the model parameters, so that probability of the model is maximized.  Three Main HMM Problems  Recognition  given an observation sequence and a Hidden Markov Model, calculate the probability that the model would produce this observation sequence.  Uncovering (Viterbi)  given an observation sequence and a Hidden Markov Model, calculate the optimal sequence of states that would maximize the likelihood of the HMM producing the observation.  Learning / Training  given an observation sequence (or a set of observation sequences and a Hidden Markov Model, adjust the model parameters, so that probability of the model is maximized.

7 Introduction  HMM History  Basic concept developed by Markov  Theory for practical implementation summarized by Rabiner and Huang (1986)  Applied in different fields to data stream observation problems  Common in speech recognition  Has become increasingly popular in music information retrieval applications  HMM History  Basic concept developed by Markov  Theory for practical implementation summarized by Rabiner and Huang (1986)  Applied in different fields to data stream observation problems  Common in speech recognition  Has become increasingly popular in music information retrieval applications

8 Overview of Works Automatic Segmentation for Music Classification using Competitive Hidden Markov Models  (2000) Battle, Cano – University Pompeu Fabra  System classifies audio segments (automatic segmentation into abstract acoustic events)  can be applied to classify a database of audio sounds  allows fast indexing and retrieval of audio fragments  similar segment events are given the same label Automatic Segmentation for Music Classification using Competitive Hidden Markov Models  (2000) Battle, Cano – University Pompeu Fabra  System classifies audio segments (automatic segmentation into abstract acoustic events)  can be applied to classify a database of audio sounds  allows fast indexing and retrieval of audio fragments  similar segment events are given the same label

9 Overview of Works  (2000) Battle, Cano (ctd.)  First stage: parametrization, features obtained from audio signals  Mel-cepstrum analysis used to obtain feature vectors  Main classification engine: HMM-based  Traditional HMMs are not suited for blind learning  Competitive HMMs used instead  CoHMMs differ from HMMs only in training stage; recognition is exactly the same  (2000) Battle, Cano (ctd.)  First stage: parametrization, features obtained from audio signals  Mel-cepstrum analysis used to obtain feature vectors  Main classification engine: HMM-based  Traditional HMMs are not suited for blind learning  Competitive HMMs used instead  CoHMMs differ from HMMs only in training stage; recognition is exactly the same

10 Overview of Works Melody Spotting Using Hidden Markov Models  (2001) Durey, Clements–Georgia Institute of Technology  A melody-based database song retrieval system  Uses melody spotting procedure adopted from word spotting techniques in automatic speech recognition  Humming, whistling, keyboard as input  Main goal: develop a practical system for non-symbolic music representation (audio)  Word/melody-spotting: searching for a data segment in a data stream Melody Spotting Using Hidden Markov Models  (2001) Durey, Clements–Georgia Institute of Technology  A melody-based database song retrieval system  Uses melody spotting procedure adopted from word spotting techniques in automatic speech recognition  Humming, whistling, keyboard as input  Main goal: develop a practical system for non-symbolic music representation (audio)  Word/melody-spotting: searching for a data segment in a data stream

11 Overview of Works  (2001) Durey, Clements (ctd.)  Uses monophonic melodies, both audio and MIDI data  Left-to-right, 5-state HMM to represent each available note and a rest  Frequency and time-domain features for feature vectors  Constructs an HMM model from the input query  Runs all of the feature vectors from the songs in the database using Viterbi process  A ranked list of melody occurances in database songs is created  System presented as a proof-of-concept  (2001) Durey, Clements (ctd.)  Uses monophonic melodies, both audio and MIDI data  Left-to-right, 5-state HMM to represent each available note and a rest  Frequency and time-domain features for feature vectors  Constructs an HMM model from the input query  Runs all of the feature vectors from the songs in the database using Viterbi process  A ranked list of melody occurances in database songs is created  System presented as a proof-of-concept

12 Overview of Works Indexing Hidden Markov Models for Music Retrieval  (2002) Jin, Jagadish – University of Michigan  Music retrieval system  Paper describes traditional MIR HMM techniques as effective but not efficient  Efficient mechanism is suggested to index the HMMs  Each state is represented by an interval / inter onset interval ratio  Each transition is transformed into a 4-dimensional box Indexing Hidden Markov Models for Music Retrieval  (2002) Jin, Jagadish – University of Michigan  Music retrieval system  Paper describes traditional MIR HMM techniques as effective but not efficient  Efficient mechanism is suggested to index the HMMs  Each state is represented by an interval / inter onset interval ratio  Each transition is transformed into a 4-dimensional box

13 Overview of Works  (2002) Jin, Jagadish (ctd.)  All boxes are inserted into R-tree, an indexing structure for multidimensional data  HMMs are ranked by the number of boxes  Most likely candidates are selected for evaluation  The evaluation uses the traditional forward algorithm  (2002) Jin, Jagadish (ctd.)  All boxes are inserted into R-tree, an indexing structure for multidimensional data  HMMs are ranked by the number of boxes  Most likely candidates are selected for evaluation  The evaluation uses the traditional forward algorithm

14 Overview of Works Chord Segmentation and Recognition using EM-Trained Hidden Markov Models  (2003) Sheh, Ellis – Columbia University  Uses HMM for chord recognition, EM (Expectation- Maximization) Algorithm to train them  PCP (Pitch Class Profile) vectors used as features to train HMMs  HMM model for each chord type (ex. ‘A Minor’, etc.)  System able to successfully recognize chords in unstructured, polyphonic, multi-timbre audio Chord Segmentation and Recognition using EM-Trained Hidden Markov Models  (2003) Sheh, Ellis – Columbia University  Uses HMM for chord recognition, EM (Expectation- Maximization) Algorithm to train them  PCP (Pitch Class Profile) vectors used as features to train HMMs  HMM model for each chord type (ex. ‘A Minor’, etc.)  System able to successfully recognize chords in unstructured, polyphonic, multi-timbre audio

15 Overview of Works Effectiveness of HMM-Based Retrieval on Large Databases  (2003) Shifrin, Burmingham – University of Michigan  Investigates performance of an HMM-based QBH system on a large musical database  VocalSearch system, part of MusArt project  theme database (roughly songs)  Uses pair as feature vectors Effectiveness of HMM-Based Retrieval on Large Databases  (2003) Shifrin, Burmingham – University of Michigan  Investigates performance of an HMM-based QBH system on a large musical database  VocalSearch system, part of MusArt project  theme database (roughly songs)  Uses pair as feature vectors

16 Overview of Works  (2003) Shifrin, Burmingham (ctd)  Compared perfect and imperfect queries, simulated insertions and deletions  Discovered Trends:  Longer queries have a positive effect on evaluation performance  All experiments show an early saturation point  Performed well with imperfect queries on a large database  (2003) Shifrin, Burmingham (ctd)  Compared perfect and imperfect queries, simulated insertions and deletions  Discovered Trends:  Longer queries have a positive effect on evaluation performance  All experiments show an early saturation point  Performed well with imperfect queries on a large database

17 Overview of Works A HMM-Based Pitch Tracker for Audio Queries  (2003) Orio, Sette – University of Padova  HMM-based approach to transcription of musical queries  HMM used to model features related to singing voice  A sung query is considered as an observation of an unknown process – the melody the user has in mind  Two-level HMM: event-level (using pitches as labels), audio-level (attack-sustain-relst events)  A simple model is presented, low recognition percentages A HMM-Based Pitch Tracker for Audio Queries  (2003) Orio, Sette – University of Padova  HMM-based approach to transcription of musical queries  HMM used to model features related to singing voice  A sung query is considered as an observation of an unknown process – the melody the user has in mind  Two-level HMM: event-level (using pitches as labels), audio-level (attack-sustain-relst events)  A simple model is presented, low recognition percentages

18 Conclusion  HTML Bibliography  Questions  HTML Bibliography  Questions