Part 3 Vector Quantization and Mixture Density Model CSE717, SPRING 2008 CUBS, Univ at Buffalo.

Slides:

Advertisements

Similar presentations

2806 Neural Computation Self-Organizing Maps Lecture Ari Visa.

Advertisements

Clustering with k-means and mixture of Gaussian densities Jakob Verbeek December 3, 2010 Course website:

Segmentation and Fitting Using Probabilistic Methods

K-means clustering Hongning Wang

Machine Learning and Data Mining Clustering

Visual Recognition Tutorial

EE 290A: Generalized Principal Component Analysis Lecture 6: Iterative Methods for Mixture-Model Segmentation Sastry & Yang © Spring, 2011EE 290A, University.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Slide 1 EE3J2 Data Mining EE3J2 Data Mining Lecture 10 Statistical Modelling Martin Russell.

Part 4 b Forward-Backward Algorithm & Viterbi Algorithm CSE717, SPRING 2008 CUBS, Univ at Buffalo.

Unsupervised Learning: Clustering Rong Jin Outline  Unsupervised learning  K means for clustering  Expectation Maximization algorithm for clustering.

Part 2b Parameter Estimation CSE717, FALL 2008 CUBS, Univ at Buffalo.

Prénom Nom Document Analysis: Data Analysis and Clustering Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

L15:Microarray analysis (Classification). The Biological Problem Two conditions that need to be differentiated, (Have different treatments). EX: ALL (Acute.

Lecture 4 Unsupervised Learning Clustering & Dimensionality Reduction

Part 4 c Baum-Welch Algorithm CSE717, SPRING 2008 CUBS, Univ at Buffalo.

Unsupervised Learning

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

Maximum Likelihood (ML), Expectation Maximization (EM)

Expectation-Maximization

Visual Recognition Tutorial

What is it? When would you use it? Why does it work? How do you implement it? Where does it stand in relation to other methods? EM algorithm reading group.

Expectation-Maximization (EM) Chapter 3 (Duda et al.) – Section 3.9

Clustering with Bregman Divergences Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, Joydeep Ghosh Presented by Rohit Gupta CSci 8980: Machine Learning.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Clustering & Dimensionality Reduction 273A Intro Machine Learning.

Gaussian Mixture Models and Expectation Maximization.

Radial Basis Function Networks

Incomplete Graphical Models Nan Hu. Outline Motivation K-means clustering Coordinate Descending algorithm Density estimation EM on unconditional mixture.

Biointelligence Laboratory, Seoul National University

Binary Variables (1) Coin flipping: heads=1, tails=0 Bernoulli Distribution.

Segmentation Techniques Luis E. Tirado PhD qualifying exam presentation Northeastern University.

COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.

1 HMM - Part 2 Review of the last lecture The EM algorithm Continuous density HMM.

CHAPTER 7: Clustering Eick: K-Means and EM (modified Alpaydin transparencies and new transparencies added) Last updated: February 25, 2014.

Chapter 9 DTW and VQ Algorithm  9.1 Basic idea of DTW  9.2 DTW algorithm  9.3 Basic idea of VQ  9.4 LBG algorithm  9.5 Improvement of VQ.

ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: ML and Simple Regression Bias of the ML Estimate Variance of the ML Estimate.

MACHINE LEARNING 8. Clustering. Motivation Based on E ALPAYDIN 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2  Classification problem:

Map-Reduce for Machine Learning on Multicore C. Chu, S.K. Kim, Y. Lin, Y.Y. Yu, G. Bradski, A.Y. Ng, K. Olukotun (NIPS 2006) Shimin Chen Big Data Reading.

Clustering Gene Expression Data BMI/CS 576 Colin Dewey Fall 2010.

Made by: Maor Levy, Temple University Many data points, no labels 2.

HMM - Part 2 The EM algorithm Continuous density HMM.

Lecture 6 Spring 2010 Dr. Jianjun Hu CSCE883 Machine Learning.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

DATA MINING van data naar informatie Ronald Westra Dep. Mathematics Maastricht University.

Clustering Clustering is a technique for finding similarity groups in data, called clusters. I.e., it groups data instances that are similar to (near)

Mehdi Ghayoumi MSB rm 132 Ofc hr: Thur, a Machine Learning.

Radial Basis Function ANN, an alternative to back propagation, uses clustering of examples in the training set.

Clustering Instructor: Max Welling ICS 178 Machine Learning & Data Mining.

Flat clustering approaches

Chapter 13 (Prototype Methods and Nearest-Neighbors )

Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Adaptive FIR Neural Model for Centroid Learning in Self-Organizing.

DENCLUE 2.0: Fast Clustering based on Kernel Density Estimation Alexander Hinneburg Martin-Luther-University Halle-Wittenberg, Germany Hans-Henning Gabriel.

Rate Distortion Theory. Introduction The description of an arbitrary real number requires an infinite number of bits, so a finite representation of a.

Advanced Artificial Intelligence Lecture 8: Advance machine learning.

1 Cluster Analysis – 2 Approaches K-Means (traditional) Latent Class Analysis (new) by Jay Magidson, Statistical Innovations based in part on a presentation.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Mixture Densities Maximum Likelihood Estimates.

Machine Learning Expectation Maximization and Gaussian Mixtures CSE 473 Chapter 20.3.

Big Data Infrastructure Week 9: Data Mining (4/4) This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States.

Unsupervised Learning Part 2. Topics How to determine the K in K-means? Hierarchical clustering Soft clustering with Gaussian mixture models Expectation-Maximization.

Clustering (3) Center-based algorithms Fuzzy k-means

DataMining, Morgan Kaufmann, p Mining Lab. 김완섭 2004년 10월 27일

4.0 More about Hidden Markov Models

Foundation of Video Coding Part II: Scalar and Vector Quantization

INTRODUCTION TO Machine Learning

Clustering Techniques

Text Categorization Berlin Chen 2003 Reference:

A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International.

A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes International.

Presentation transcript:

Part 3 Vector Quantization and Mixture Density Model CSE717, SPRING 2008 CUBS, Univ at Buffalo

Vector Quantization Quantization Represents continuous range of values by a set of discrete values Example: floating-point representation of real numbers in computer Vector Quantization Represent a data space (vector space) by discrete set of vectors

Vector Quantizer A mapping from vector space onto a finite subset of the vector space Y = y 1,y 2,…,y N finite subset of IR k, referred to as the codebook of Q Q is usually determined by training data

Partition of Vector Quantizer The vector space is partitioned into N cells by the vector quantizer

Properties of Partition Vector quantizer Q defines a complete and disjoint partition of IR k into R 1,R 2,…,R N

Quantization Error Quantization Error for single vector x is a suitable distance measure Overall Quantization Error

Nearest-Neighbor Condition The minimum quantization error of a given codebook Y is given by partition x y1y1 y2y2 y3y3

Centroid Condition Centroid of a cell R i is minimized by choosing as the codebook For Euclidean distance d Centroid

Vector Quantizer Design – General Steps 1. Determine initial codebook Y 0 2. Adjust partition of sample data for the current codebook Y m using nearest-neighbor condition 3. Update the codebook Y m → Y m+1 using centroid condition 4. Check a certain condition of convergence. If it converges, return the current codebook Y m+1 ; otherwise go to step 2 Converge? Y N

Lloyd’s Algorithm

Lloyd’s Algorithm (Cont)

LBG Algorithm

LBG Algorithm (Cont)

k-Means Algorithm

k-Means Algorithm (Cont)

Mixture Density Model A mixture model of N random variables X 1,…,X N is defined as follows: is a random variables defined on N labels

Mixture Density Model Suppose the p.d.f.’s of X 1,…,X N are and then

Example: Gaussian Mixture Model of Two Components Histogram of samplesMixture Density

Estimation of Gaussian Mixture Model ML Estimation (Value X and label are given) Samples in the format of (-0.39, 0), (0.12, 0), (0.94, 1), (1.67, 0), (1.76, 1), … S 1 (Subset of ): (0.94, 1), (1.76, 1), … S 2 (Subset of ): (-0.39, 0), (0.12, 0), (1.67, 0), …

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown) 1. Choose initial values of 2. E-Step: For each sample x k, label is missing. But we can estimate using its expected value Samples in the format of ; is missing (-0.39), (0.12), (0.94), (1.67), (1.76), … x1x1 x2x2

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown) 3. M-Step: We can estimate again using the labels estimated in the E-Step:

Estimation of Gaussian Mixture Model EM Algorithm (Value X is given, label is unknown) 4. Termination: The log likelihood of n samples At the end of m-th iteration, if terminate; otherwise go to step 2 (the E-Step).