K-means method for Signal Compression: Vector Quantization

Slides:



Advertisements
Similar presentations
Applications of one-class classification
Advertisements

1 Classification using instance-based learning. 3 March, 2000Advanced Knowledge Management2 Introduction (lazy vs. eager learning) Notion of similarity.
Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?
Huffman code and ID3 Prof. Sin-Min Lee Department of Computer Science.
Data Mining Classification: Alternative Techniques
Data Mining Classification: Alternative Techniques
1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.
Instance Based Learning
1 Machine Learning: Lecture 7 Instance-Based Learning (IBL) (Based on Chapter 8 of Mitchell T.., Machine Learning, 1997)
Lazy vs. Eager Learning Lazy vs. eager learning
1er. Escuela Red ProTIC - Tandil, de Abril, Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.
. Markov Chains as a Learning Tool. 2 Weather: raining today40% rain tomorrow 60% no rain tomorrow not raining today20% rain tomorrow 80% no rain tomorrow.
Classification and Decision Boundaries
Nearest Neighbor. Predicting Bankruptcy Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return.
Content Based Image Clustering and Image Retrieval Using Multiple Instance Learning Using Multiple Instance Learning Xin Chen Advisor: Chengcui Zhang Department.
K nearest neighbor and Rocchio algorithm
CSE 589 Applied Algorithms Spring 1999 Image Compression Vector Quantization Nearest Neighbor Search.
MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 
Instance based learning K-Nearest Neighbor Locally weighted regression Radial basis functions.
Instance Based Learning
Spatial and Temporal Data Mining
Instance-Based Learning
These slides are based on Tom Mitchell’s book “Machine Learning” Lazy learning vs. eager learning Processing is delayed until a new instance must be classified.
Aprendizagem baseada em instâncias (K vizinhos mais próximos)
KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.
What is Cluster Analysis?
Instance Based Learning Bob Durrant School of Computer Science University of Birmingham (Slides: Dr Ata Kabán) 1.
INSTANCE-BASE LEARNING
Memory-Based Learning Instance-Based Learning K-Nearest Neighbor.
CS Instance Based Learning1 Instance Based Learning.
Radial Basis Function (RBF) Networks
Radial-Basis Function Networks
Module 04: Algorithms Topic 07: Instance-Based Learning
Methods in Medical Image Analysis Statistics of Pattern Recognition: Classification and Clustering Some content provided by Milos Hauskrecht, University.
K Nearest Neighborhood (KNNs)
DATA MINING LECTURE 10 Classification k-nearest neighbor classifier Naïve Bayes Logistic Regression Support Vector Machines.
COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.
1 Instance Based Learning Ata Kaban The University of Birmingham.
Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.7: Instance-Based Learning Rodney Nielsen.
Gang WangDerek HoiemDavid Forsyth. INTRODUCTION APROACH (implement detail) EXPERIMENTS CONCLUSION.
CpSc 881: Machine Learning Instance Based Learning.
CpSc 810: Machine Learning Instance Based Learning.
Vector Quantization Vector quantization is used in many applications such as image and voice compression, voice recognition (in general statistical pattern.
Vector Quantization CAP5015 Fall 2005.
Outline K-Nearest Neighbor algorithm Fuzzy Set theory Classifier Accuracy Measures.
Lazy Learners K-Nearest Neighbor algorithm Fuzzy Set theory Classifier Accuracy Measures.
KNN Classifier.  Handed an instance you wish to classify  Look around the nearby region to see what other classes are around  Whichever is most common—make.
Overview Data Mining - classification and clustering
Meta-learning for Algorithm Recommendation Meta-learning for Algorithm Recommendation Background on Local Learning Background on Algorithm Assessment Algorithm.
CS Machine Learning Instance Based Learning (Adapted from various sources)
K-Nearest Neighbor Learning.
Eick: kNN kNN: A Non-parametric Classification and Prediction Technique Goals of this set of transparencies: 1.Introduce kNN---a popular non-parameric.
Debrup Chakraborty Non Parametric Methods Pattern Recognition and Machine Learning.
Kansas State University Department of Computing and Information Sciences CIS 890: Special Topics in Intelligent Systems Wednesday, November 15, 2000 Cecil.
SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.
CS 8751 ML & KDDInstance Based Learning1 k-Nearest Neighbor Locally weighted regression Radial basis functions Case-based reasoning Lazy and eager learning.
1 Instance Based Learning Soongsil University Intelligent Systems Lab.
S.R.Subramanya1 Outline of Vector Quantization of Images.
Data Transformation: Normalization
Data Science Algorithms: The Basic Methods
Instance Based Learning
Instance Based Learning (Adapted from various sources)
K Nearest Neighbor Classification
Classification Nearest Neighbor
Nearest-Neighbor Classifiers
Instance Based Learning
COSC 4335: Other Classification Techniques
Chap 8. Instance Based Learning
Advanced Mathematics Hossein Malekinezhad.
Machine Learning: UNIT-4 CHAPTER-1
Presentation transcript:

K-means method for Signal Compression: Vector Quantization

Voronoi Region Blocks of signals: A sequence of audio. A block of image pixels. Formally: vector example: (0.2, 0.3, 0.5, 0.1) A vector quantizer maps k-dimensional vectors in the vector space Rk into a finite set of vectors Y = {yi: i = 1, 2, ..., N}.  Each vector yi is called a code vector or a codeword. and the set of all the codewords is called a codebook.  Associated with each codeword, yi, is a nearest neighbor region called Voronoi region, and it is defined by: The set of Voronoi regions partition the entire space Rk .

Two Dimensional Voronoi Diagram Codewords in 2-dimensional space.  Input vectors are marked with an x, codewords are marked with red circles, and the Voronoi regions are separated with boundary lines.

The Schematic of a Vector Quantizer (signal compression)

Compression Formula Amount of compression: Codebook size is K, input vector of dimension L In order to inform the decoder of which code vector is selected, we need to use bits. E.g. need 8 bits to represent 256 code vectors. Rate: each code vector contains the reconstruction value of L source output samples, the number of bits per vector component would be: . K is called “level of vector quantizer”.

Vector Quantizer Algorithm Determine the number of codewords, N,  or the size of the codebook. Select N codewords at random, and let that be the initial codebook.  The initial codewords can be randomly chosen from the set of input vectors. Using the Euclidean distance measure clusterize the vectors around each codeword.  This is done by taking each input vector and finding the Euclidean distance between it and each codeword.  The input vector belongs to the cluster of the codeword that yields the minimum distance.

Vector Quantizer Algorithm (contd.) 4. Compute the new set of codewords.  This is done by obtaining the average of each cluster.  Add the component of each vector and divide by the number of vectors in the cluster. where i is the component of each vector (x, y, z, ... directions), m is the number of vectors in the cluster. 5. Repeat steps 2 and 3 until the either the codewords don't change or the change in the codewords is small.

Other Algorithms Problem: k-means is a greedy algorithm, may fall into Local minimum. Four methods selecting initial vectors: Random Splitting (with perturbation vector) Animation Train with different subset PNN (pairwise nearest neighbor) Empty cell problem: No input corresponds to am output vector Solution: give to other clusters, e.g. most populate cluster.

VQ for image compression Taking blocks of images as vector L=NM. If K vectors in code book: need to use bits. Rate: The higher the value K, the better quality, but lower compression ratio. Overhead to transmit code book: Train with a set of images.

K-Nearest Neighbor Learning 22c:145 University of Iowa

Different Learning Methods Parametric Learning The target function is described by a set of parameters (examples are forgotten) E.g., structure and weights of a neural network Instance-based Learning Learning=storing all training instances Classification=assigning target function to a new instance Referred to as “Lazy” learning

Instance-based Learning Its very similar to a Desktop!!

General Idea of Instance-based Learning Learning: store all the data instances Performance: when a new query instance is encountered retrieve a similar set of related instances from memory use to classify the new query

Pros and Cons of Instance Based Learning Can construct a different approximation to the target function for each distinct query instance to be classified Can use more complex, symbolic representations Cons Cost of classification can be high Uses all attributes (do not learn which are most important)

Instance-based Learning K-Nearest Neighbor Algorithm Weighted Regression Case-based reasoning

k-nearest neighbor (knn) learning Most basic type of instance learning Assumes all instances are points in n-dimensional space A distance measure is needed to determine the “closeness” of instances Classify an instance by finding its nearest neighbors and picking the most popular class among the neighbors

1-Nearest Neighbor

3-Nearest Neighbor

Important Decisions Distance measure Value of k (usually odd) Voting mechanism Memory indexing

Euclidean Distance Typically used for real valued attributes Instance x (often called a feature vector) Distance between two instances xi and xj

Discrete Valued Target Function Training algorithm: For each training example <x, f(x)>, add the example to the list training_examples Classification algorithm: Given a query instance xq to be classified. Let x1…xk be the k training examples nearest to xq Return

Continuous valued target function Algorithm computes the mean value of the k nearest training examples rather than the most common value Replace fine line in previous algorithm with

Training dataset Customer ID Debt Income Marital Status Risk Abel High Married Good Ben Low Doubtful Candy Medium Very low Unmarried Poor Dale Very high Ellen Fred George Harry Igor Very Low Very High Jack

k-nn K = 3 Distance Score for an attribute is 1 for a match and 0 otherwise Distance is sum of scores for each attribute Voting scheme: proportionate voting in case of ties

Query: Zeb High Medium Married ? Customer ID Debt Income Marital Status Risk Abel High Married Good Ben Low Doubtful Candy Medium Very low Unmarried Poor Dale Very high Ellen Fred George Harry Igor Very Low Very High Jack

Query: Yong Low High Married ? Customer ID Debt Income Marital Status Risk Abel High Married Good Ben Low Doubtful Candy Medium Very low Unmarried Poor Dale Very high Ellen Fred George Harry Igor Very Low Very High Jack

Query: Vasco High Low Married ? Customer ID Debt Income Marital Status Risk Abel High Married Good Ben Low Doubtful Candy Medium Very low Unmarried Poor Dale Very high Ellen Fred George Harry Igor Very Low Very High Jack

Voronoi Diagram Decision surface formed by the training examples of two attributes

Examples of one attribute

Distance-Weighted Nearest Neighbor Algorithm Assign weights to the neighbors based on their ‘distance’ from the query point Weight ‘may’ be inverse square of the distances All training points may influence a particular instance Shepard’s method

Kernel function for Distance-Weighted Nearest Neighbor

Examples of one attribute

Remarks +Highly effective inductive inference method for noisy training data and complex target functions +Target function for a whole space may be described as a combination of less complex local approximations +Learning is very simple - Classification is time consuming

Curse of Dimensionality - When the dimensionality increases, the volume of the space increases so fast that the available data becomes sparse. This sparsity is problematic for any method that requires statistical significance. 

Curse of Dimensionality Suppose there are N data points of dimension n in the space [-1/2, 1/2]n. The k-neighborhood of a point is defined to be the smallest hypercube containing the k-nearest neighbor. Let l be the average side length of a k-neighborhood. Then the volume of an average hypercube is dn. So dn/1n = k/N, or d = (k/N)1/n

d = (k/N)1/n N k n d 1,000,000 10 2 0.003 3 0.02 17 0.5 200 0.94 When n is big, all the points are outliers.

- Curse of Dimensionality

- Curse of Dimensionality