Presentation is loading. Please wait.

Presentation is loading. Please wait.

K Nearest Neighbors Saed Sayad 1www.ismartsoft.com.

Published bySamson Burns Modified over 9 years ago

Similar presentations

Presentation on theme: "K Nearest Neighbors Saed Sayad 1www.ismartsoft.com."— Presentation transcript:

1 K Nearest Neighbors Saed Sayad 1www.ismartsoft.com

2 KNN - Definition KNN is a simple algorithm that stores all available cases and classifies new cases based on a similarity measure 2www.ismartsoft.com

3 KNN – different names K-Nearest Neighbors Memory-Based Reasoning Example-Based Reasoning Instance-Based Learning Case-Based Reasoning Lazy Learning K-Nearest Neighbors Memory-Based Reasoning Example-Based Reasoning Instance-Based Learning Case-Based Reasoning Lazy Learning 3www.ismartsoft.com

4 KNN – Short History Nearest Neighbors have been used in statistical estimation and pattern recognition already in the beginning of 1970’s ( non-parametric techniques ). Dynamic Memory: A theory of Reminding and Learning in Computer and People (Schank, 1982). People reason by remembering and learn by doing. Thinking is reminding, making analogies. Examples = Concepts??? Nearest Neighbors have been used in statistical estimation and pattern recognition already in the beginning of 1970’s ( non-parametric techniques ). Dynamic Memory: A theory of Reminding and Learning in Computer and People (Schank, 1982). People reason by remembering and learn by doing. Thinking is reminding, making analogies. Examples = Concepts??? 4www.ismartsoft.com

5 KNN Classification Age Loan$ 5www.ismartsoft.com

6 KNN Classification – Distance AgeLoanDefaultDistance 25$40,000N102000 35$60,000N82000 45$80,000N62000 20$20,000N122000 35$120,000N22000 52$18,000N124000 23$95,000Y47000 40$62,000Y80000 60$100,000Y42000 48$220,000Y78000 33$150,000Y8000 48$142,000? Euclidean Distance 6www.ismartsoft.com

7 KNN Classification – Standardized Distance AgeLoanDefaultDistance 0.1250.11N0.7652 0.3750.21N0.5200 0.6250.31N0.3160 00.01N0.9245 0.3750.50N0.3428 0.80.00N0.6220 0.0750.38Y0.6669 0.50.22Y0.4437 10.41Y0.3650 0.71.00Y0.3861 0.3250.65Y0.3771 0.70.61? Standardized Variable 7www.ismartsoft.com

8 KNN Regression - Distance AgeLoanHouse Price IndexDistance 25$40,000135102000 35$60,00025682000 45$80,00023162000 20$20,000267122000 35$120,00013922000 52$18,000150124000 23$95,00012747000 40$62,00021680000 60$100,00013942000 48$220,00025078000 33$150,0002648000 48$142,000? 8www.ismartsoft.com

9 KNN Regression – Standardized Distance AgeLoanHouse Price IndexDistance 0.1250.111350.7652 0.3750.212560.5200 0.6250.312310.3160 00.012670.9245 0.3750.501390.3428 0.80.001500.6220 0.0750.381270.6669 0.50.222160.4437 10.411390.3650 0.71.002500.3861 0.3250.652640.3771 0.70.61? 9www.ismartsoft.com

10 KNN – Number of Neighbors If K=1, select the nearest neighbor If K>1, – For classification select the most frequent neighbor. – For regression calculate the average of K neighbors. If K=1, select the nearest neighbor If K>1, – For classification select the most frequent neighbor. – For regression calculate the average of K neighbors. 10www.ismartsoft.com

11 Distance – Categorical Variables XYDistance Male 0 Female1 11www.ismartsoft.com

12 Instance Based Reasoning IB1 is based on the standard KNN IB2 is incremental KNN learner that only incorporates misclassified instances into the classifier. IB3 discards instances that do not perform well by keeping success records. IB1 is based on the standard KNN IB2 is incremental KNN learner that only incorporates misclassified instances into the classifier. IB3 discards instances that do not perform well by keeping success records. 12www.ismartsoft.com

13 Case Based Reasoning 13www.ismartsoft.com

14 KNN - Applications Classification and Interpretation – legal, medical, news, banking Problem-solving – planning, pronunciation Function learning – dynamic control Teaching and aiding – help desk, user training Classification and Interpretation – legal, medical, news, banking Problem-solving – planning, pronunciation Function learning – dynamic control Teaching and aiding – help desk, user training 14www.ismartsoft.com

15 Summary KNN is conceptually simple, yet able to solve complex problems Can work with relatively little information Learning is simple (no learning at all!) Memory and CPU cost Feature selection problem Sensitive to representation KNN is conceptually simple, yet able to solve complex problems Can work with relatively little information Learning is simple (no learning at all!) Memory and CPU cost Feature selection problem Sensitive to representation 15www.ismartsoft.com

16 16www.ismartsoft.com Questions?

Download ppt "K Nearest Neighbors Saed Sayad 1www.ismartsoft.com."

Similar presentations

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.

1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.
R OBERTO B ATTITI, M AURO B RUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Feb 2014.

R OBERTO B ATTITI, M AURO B RUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Feb 2014.
INTRODUCTION TO Machine Learning 3rd Edition

INTRODUCTION TO Machine Learning 3rd Edition
1 Machine Learning: Lecture 7 Instance-Based Learning (IBL) (Based on Chapter 8 of Mitchell T.., Machine Learning, 1997)

1 Machine Learning: Lecture 7 Instance-Based Learning (IBL) (Based on Chapter 8 of Mitchell T.., Machine Learning, 1997)
Lazy vs. Eager Learning Lazy vs. eager learning

Lazy vs. Eager Learning Lazy vs. eager learning
1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006 4. Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.

1er. Escuela Red ProTIC - Tandil, de Abril, Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.
Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)

Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)
ETHEM ALPAYDIN © The MIT Press, 2010 Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
K nearest neighbor and Rocchio algorithm

K nearest neighbor and Rocchio algorithm
MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 
By Fernando Seoane, April 25 th, 2006 Demo for Non-Parametric Classification Euclidean Metric Classifier with Data Clustering.

By Fernando Seoane, April 25 th, 2006 Demo for Non-Parametric Classification Euclidean Metric Classifier with Data Clustering.
Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.
Instance Based Learning

Instance Based Learning
Instance-Based Learning

Instance-Based Learning
Machine Learning Group University College Dublin Nearest Neighbour Classifiers Lazy v’s Eager k-NN Condensed NN.

Machine Learning Group University College Dublin Nearest Neighbour Classifiers Lazy v’s Eager k-NN Condensed NN.
Instance Based Learning. Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return the answer associated.

Instance Based Learning. Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return the answer associated.
Comparison of Instance-Based Techniques for Learning to Predict Changes in Stock Prices iCML Conference December 10, 2003 Presented by: David LeRoux.

Comparison of Instance-Based Techniques for Learning to Predict Changes in Stock Prices iCML Conference December 10, 2003 Presented by: David LeRoux.
Case-based Reasoning System (CBR)

Case-based Reasoning System (CBR)

Similar presentations

Ads by Google