K Nearest Neighbors Saed Sayad 1www.ismartsoft.com.

Slides:

Advertisements

Similar presentations

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

Advertisements

Data Mining Classification: Alternative Techniques

1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.

R OBERTO B ATTITI, M AURO B RUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Feb 2014.

INTRODUCTION TO Machine Learning 3rd Edition

1 Machine Learning: Lecture 7 Instance-Based Learning (IBL) (Based on Chapter 8 of Mitchell T.., Machine Learning, 1997)

Lazy vs. Eager Learning Lazy vs. eager learning

1er. Escuela Red ProTIC - Tandil, de Abril, Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.

Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

K nearest neighbor and Rocchio algorithm

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

By Fernando Seoane, April 25 th, 2006 Demo for Non-Parametric Classification Euclidean Metric Classifier with Data Clustering.

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Instance Based Learning

Instance-Based Learning

Machine Learning Group University College Dublin Nearest Neighbour Classifiers Lazy v’s Eager k-NN Condensed NN.

Instance Based Learning. Nearest Neighbor Remember all your data When someone asks a question –Find the nearest old data point –Return the answer associated.

Comparison of Instance-Based Techniques for Learning to Predict Changes in Stock Prices iCML Conference December 10, 2003 Presented by: David LeRoux.

Case-based Reasoning System (CBR)

Data Mining Classification: Alternative Techniques

1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.

CES 514 – Data Mining Lec 9 April 14 Mid-term k nearest neighbor.

Aprendizagem baseada em instâncias (K vizinhos mais próximos)

Instance Based Learning Bob Durrant School of Computer Science University of Birmingham (Slides: Dr Ata Kabán) 1.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.

CLUSTERING (Segmentation)

INSTANCE-BASE LEARNING

Memory-Based Learning Instance-Based Learning K-Nearest Neighbor.

Nearest Neighbor Classifiers other names: –instance-based learning –case-based learning (CBL) –non-parametric learning –model-free learning.

CS Instance Based Learning1 Instance Based Learning.

12 -1 Lecture 12 User Modeling Topics –Basics –Example User Model –Construction of User Models –Updating of User Models –Applications.

Data Mining Techniques

Methods in Medical Image Analysis Statistics of Pattern Recognition: Classification and Clustering Some content provided by Milos Hauskrecht, University.

Copyright R. Weber Machine Learning, Data Mining ISYS370 Dr. R. Weber.

K Nearest Neighborhood (KNNs)

Introduction to Data Mining Group Members: Karim C. El-Khazen Pascal Suria Lin Gui Philsou Lee Xiaoting Niu.

COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.

1 Data Mining Lecture 5: KNN and Bayes Classifiers.

Introduction to machine learning and data mining 1 iCSC2014, Juan López González, University of Oviedo Introduction to machine learning Juan López González.

1 SUPPORT VECTOR MACHINES İsmail GÜNEŞ. 2 What is SVM? A new generation learning system. A new generation learning system. Based on recent advances in.

K Nearest Neighbors Classifier & Decision Trees

11/12/2012ISC471 / HCI571 Isabelle Bichindaritz 1 Prediction.

 2003, G.Tecuci, Learning Agents Laboratory 1 Learning Agents Laboratory Computer Science Department George Mason University Prof. Gheorghe Tecuci 9 Instance-Based.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Statistical Inference (By Michael Jordon) l Bayesian perspective –conditional perspective—inferences.

1 Instance Based Learning Ata Kaban The University of Birmingham.

Chapter 11 Statistical Techniques. Data Warehouse and Data Mining Chapter 11 2 Chapter Objectives  Understand when linear regression is an appropriate.

Nearest Neighbor Ling 572 Advanced Statistical Methods in NLP January 12, 2012.

CpSc 881: Machine Learning Instance Based Learning.

CpSc 810: Machine Learning Instance Based Learning.

COMP 2208 Dr. Long Tran-Thanh University of Southampton K-Nearest Neighbour.

KNN & Naïve Bayes Hongning Wang Today’s lecture Instance-based classifiers – k nearest neighbors – Non-parametric learning algorithm Model-based.

Outline K-Nearest Neighbor algorithm Fuzzy Set theory Classifier Accuracy Measures.

Lazy Learners K-Nearest Neighbor algorithm Fuzzy Set theory Classifier Accuracy Measures.

K nearest neighbors algorithm Parallelization on Cuda PROF. VELJKO MILUTINOVIĆ MAŠA KNEŽEVIĆ 3037/2015.

CS Machine Learning Instance Based Learning (Adapted from various sources)

K-Nearest Neighbor Learning.

Eick: kNN kNN: A Non-parametric Classification and Prediction Technique Goals of this set of transparencies: 1.Introduce kNN---a popular non-parameric.

Debrup Chakraborty Non Parametric Methods Pattern Recognition and Machine Learning.

Kansas State University Department of Computing and Information Sciences CIS 890: Special Topics in Intelligent Systems Wednesday, November 15, 2000 Cecil.

Instance-Based Learning Evgueni Smirnov. Overview Instance-Based Learning Comparison of Eager and Instance-Based Learning Instance Distances for Instance-Based.

1 Text Categorization  Assigning documents to a fixed set of categories  Applications:  Web pages  Recommending pages  Yahoo-like classification hierarchies.

1 Instance Based Learning Soongsil University Intelligent Systems Lab.

KNN & Naïve Bayes Hongning Wang

Machine Learning Usman Roshan Dept. of Computer Science NJIT.

K Nearest Neighbors and Instance-based methods

Outline Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no.

Nearest-Neighbor Classifiers

Advanced Mathematics Hossein Malekinezhad.

Presentation transcript:

K Nearest Neighbors Saed Sayad 1www.ismartsoft.com

KNN - Definition KNN is a simple algorithm that stores all available cases and classifies new cases based on a similarity measure 2www.ismartsoft.com

KNN – different names K-Nearest Neighbors Memory-Based Reasoning Example-Based Reasoning Instance-Based Learning Case-Based Reasoning Lazy Learning K-Nearest Neighbors Memory-Based Reasoning Example-Based Reasoning Instance-Based Learning Case-Based Reasoning Lazy Learning 3www.ismartsoft.com

KNN – Short History Nearest Neighbors have been used in statistical estimation and pattern recognition already in the beginning of 1970’s ( non-parametric techniques ). Dynamic Memory: A theory of Reminding and Learning in Computer and People (Schank, 1982). People reason by remembering and learn by doing. Thinking is reminding, making analogies. Examples = Concepts??? Nearest Neighbors have been used in statistical estimation and pattern recognition already in the beginning of 1970’s ( non-parametric techniques ). Dynamic Memory: A theory of Reminding and Learning in Computer and People (Schank, 1982). People reason by remembering and learn by doing. Thinking is reminding, making analogies. Examples = Concepts??? 4www.ismartsoft.com

KNN Classification Age Loan$ 5www.ismartsoft.com

KNN Classification – Distance AgeLoanDefaultDistance 25$40,000N $60,000N $80,000N $20,000N $120,000N $18,000N $95,000Y $62,000Y $100,000Y $220,000Y $150,000Y $142,000? Euclidean Distance 6www.ismartsoft.com

KNN Classification – Standardized Distance AgeLoanDefaultDistance N N N N N N Y Y Y Y Y ? Standardized Variable 7www.ismartsoft.com

KNN Regression - Distance AgeLoanHouse Price IndexDistance 25$40, $60, $80, $20, $120, $18, $95, $62, $100, $220, $150, $142,000? 8www.ismartsoft.com

KNN Regression – Standardized Distance AgeLoanHouse Price IndexDistance ? 9www.ismartsoft.com

KNN – Number of Neighbors If K=1, select the nearest neighbor If K>1, – For classification select the most frequent neighbor. – For regression calculate the average of K neighbors. If K=1, select the nearest neighbor If K>1, – For classification select the most frequent neighbor. – For regression calculate the average of K neighbors. 10www.ismartsoft.com

Distance – Categorical Variables XYDistance Male 0 Female1 11www.ismartsoft.com

Instance Based Reasoning IB1 is based on the standard KNN IB2 is incremental KNN learner that only incorporates misclassified instances into the classifier. IB3 discards instances that do not perform well by keeping success records. IB1 is based on the standard KNN IB2 is incremental KNN learner that only incorporates misclassified instances into the classifier. IB3 discards instances that do not perform well by keeping success records. 12www.ismartsoft.com

Case Based Reasoning 13www.ismartsoft.com

KNN - Applications Classification and Interpretation – legal, medical, news, banking Problem-solving – planning, pronunciation Function learning – dynamic control Teaching and aiding – help desk, user training Classification and Interpretation – legal, medical, news, banking Problem-solving – planning, pronunciation Function learning – dynamic control Teaching and aiding – help desk, user training 14www.ismartsoft.com

Summary KNN is conceptually simple, yet able to solve complex problems Can work with relatively little information Learning is simple (no learning at all!) Memory and CPU cost Feature selection problem Sensitive to representation KNN is conceptually simple, yet able to solve complex problems Can work with relatively little information Learning is simple (no learning at all!) Memory and CPU cost Feature selection problem Sensitive to representation 15www.ismartsoft.com

16www.ismartsoft.com Questions?