Nearest-Neighbor Classifiers Sec 4.7. 5 minutes of math... Definition: a metric function is a function that obeys the following properties: Identity:

Slides:

Advertisements

Similar presentations

Principles of Density Estimation

Advertisements

Naïve-Bayes Classifiers Business Intelligence for Managers.

iDistance -- Indexing the Distance An Efficient Approach to KNN Indexing C. Yu, B. C. Ooi, K.-L. Tan, H.V. Jagadish. Indexing the distance:

INC 551 Artificial Intelligence Lecture 11 Machine Learning (Continue)

Linear Separators.

Indian Statistical Institute Kolkata

Distance and Similarity Measures

Lazy vs. Eager Learning Lazy vs. eager learning

Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)

Nearest Neighbor. Predicting Bankruptcy Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return.

Prof. Ramin Zabih (CS) Prof. Ashish Raj (Radiology) CS5540: Computational Techniques for Analyzing Clinical Data.

Learning from Observations Chapter 18 Section 1 – 4.

More Methodology; Nearest-Neighbor Classifiers Sec 4.7.

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Instance Based Learning

Linear methods: Regression & Discrimination Sec 4.6.

Instance Based Learning. Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return the answer associated.

1-NN Rule: Given an unknown sample X decide if for That is, assign X to category if the closest neighbor of X is from category i.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Supervised Distance Metric Learning Presented at CMU’s Computer Vision Misc-Read Reading Group May 9, 2007 by Tomasz Malisiewicz.

These slides are based on Tom Mitchell’s book “Machine Learning” Lazy learning vs. eager learning Processing is delayed until a new instance must be classified.

CES 514 – Data Mining Lec 9 April 14 Mid-term k nearest neighbor.

Aprendizagem baseada em instâncias (K vizinhos mais próximos)

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

Steep learning curves Reading: DH&S, Ch 4.6, 4.5.

INSTANCE-BASE LEARNING

Nearest Neighbor Classifiers other names: –instance-based learning –case-based learning (CBL) –non-parametric learning –model-free learning.

CS Instance Based Learning1 Instance Based Learning.

NN Cont’d. Administrivia No news today... Homework not back yet Working on it... Solution set out today, though.

Distance Measures Tan et al. From Chapter 2. Similarity and Dissimilarity Similarity –Numerical measure of how alike two data objects are. –Is higher.

Module 04: Algorithms Topic 07: Instance-Based Learning

Methods in Medical Image Analysis Statistics of Pattern Recognition: Classification and Clustering Some content provided by Milos Hauskrecht, University.

Supervised Learning and k Nearest Neighbors Business Intelligence for Managers.

K Nearest Neighbors Classifier & Decision Trees

1 Pattern Classification X. 2 Content General Method K Nearest Neighbors Decision Trees Nerual Networks.

Nearest Neighbor (NN) Rule & k-Nearest Neighbor (k-NN) Rule Non-parametric : Can be used with arbitrary distributions, No need to assume that the form.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Approximating a Square Root Approximate to the nearest integer. Example 2 The perfect square closest to, but less than, 51 is 49. The perfect square closest.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.7: Instance-Based Learning Rodney Nielsen.

Classification Problem GivenGiven Predict class label of a given queryPredict class label of a given query

METU Informatics Institute Min720 Pattern Classification with Bio-Medical Applications Part 6: Nearest and k-nearest Neighbor Classification.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Fast Similarity Metric Based Data Mining Techniques Using P-trees: k-Nearest Neighbor Classification  Distance metric based computation using P-trees.

Nearest Neighbor Classifier 1.K-NN Classifier 2.Multi-Class Classification.

Chapter 13 (Prototype Methods and Nearest-Neighbors )

Bayesian decision theory: A framework for making decisions when uncertainty exit 1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e.

KNN Classifier.  Handed an instance you wish to classify  Look around the nearby region to see what other classes are around  Whichever is most common—make.

Overview Data Mining - classification and clustering

Classify each triangle by its angle measures Simplify 4. If a = 6, b = 7, and c = 12, find a 2 + b 2 and find c 2. Which value is greater?

K nearest neighbors algorithm Parallelization on Cuda PROF. VELJKO MILUTINOVIĆ MAŠA KNEŽEVIĆ 3037/2015.

CS Machine Learning Instance Based Learning (Adapted from various sources)

Eick: kNN kNN: A Non-parametric Classification and Prediction Technique Goals of this set of transparencies: 1.Introduce kNN---a popular non-parameric.

1 Text Categorization  Assigning documents to a fixed set of categories  Applications:  Web pages  Recommending pages  Yahoo-like classification hierarchies.

Nonparametric Density Estimation – k-nearest neighbor (kNN) 02/20/17

Data Science Algorithms: The Basic Methods

MIRA, SVM, k-NN Lirong Xia. MIRA, SVM, k-NN Lirong Xia.

Lecture 05: K-nearest neighbors

K Nearest Neighbors and Instance-based methods

Metric Learning for Clustering

Similarity and Dissimilarity

Instance Based Learning (Adapted from various sources)

K Nearest Neighbor Classification

Nearest-Neighbor Classifiers

Data Mining extracting knowledge from a large amount of data

Nearest Neighbors CSC 576: Data Mining.

Lecture 03: K-nearest neighbors

Data Mining Classification: Alternative Techniques

Memory-Based Learning Instance-Based Learning K-Nearest Neighbor

MIRA, SVM, k-NN Lirong Xia. MIRA, SVM, k-NN Lirong Xia.

ECE – Pattern Recognition Lecture 10 – Nonparametric Density Estimation – k-nearest-neighbor (kNN) Hairong Qi, Gonzalez Family Professor Electrical.

Presentation transcript:

Nearest-Neighbor Classifiers Sec 4.7

5 minutes of math... Definition: a metric function is a function that obeys the following properties: Identity: Symmetry: Triangle inequality:

5 minutes of math... Examples: Euclidean distance * Note: omitting the square root still yields a metric and usually won’t change our results

5 minutes of math... Examples: Manhattan (taxicab) distance Distance travelled along a grid between two points No diagonals allowed

5 minutes of math... Examples: What if some attribute is categorical? Typical answer is 0/1 distance: For each attribute, add 1 if the instances differ in that attribute, else 0 (To make Daniel happy: for (i=0;i<xa.length;++i) { d+=(xa[i]!=xb[i]) ? 1 : 0; } )

Distances in classification Nearest neighbor: find the nearest instance to the query point in feature space, return the class of that instance Simplest possible distance-based classifier With more notation: Distance function is anything appropriate to your data

Properties of NN Training time of NN? Classification time? Geometry of model? d(, ) Closer to

Properties of NN Training time of NN? Classification time? Geometry of model?

Properties of NN Training time of NN? Classification time? Geometry of model?

Eventually...

NN miscellaney Slight generalization: k -Nearest neighbors ( k - NN) Find k training instances closest to query point Vote among them for label Q: How does this affect system? Q: Why does it work?

Geometry of k -NN d (7) Query point

Exercise Show that k-NN does something reasonable: Assume binary data Let X be query point, X’ be any k -neighbor of X Let p=Pr[Class(X’)==Class(X)] ( p>1/2 ) What is Pr[X receives correct label] ? What happens as k grows? But there are tradeoffs... Let V(k,N)=volume of sphere enclosing k neighbors of X, assuming N points in data set For fixed N, what happens to V(k,N) as k grows? For fixed k, what happens to V(k,N) as N grows? What about radius of V(k,N) ?

Excercise What is Pr[X receives correct label] ?

Excercise What happens as k →∞? Theorem: in the limit of large k, the binomial distribution is well approximated by the Gaussian:

Excercise So:

NN miscellaney Gotcha: unscaled dimensions What happens if one axis is measured in microns and one in lightyears? Usual trick is to scale each axis to [0,1] range (Sometimes [-1,1] is useful as well)