Presentation is loading. Please wait.

Presentation is loading. Please wait.

ECE 5424: Introduction to Machine Learning

Published byΞενία Δουμπιώτης Modified over 6 years ago

Similar presentations

Presentation on theme: "ECE 5424: Introduction to Machine Learning"— Presentation transcript:

1 ECE 5424: Introduction to Machine Learning
Topics: (Finish) Nearest Neighbor Readings: Barber 14 (kNN) Stefan Lee Virginia Tech

2 Administrative HW1 Out now on Scholar Due on Wednesday 09/14, 11:55pm
Please please please please please start early Implement K-NN Kaggle Competition Bonus points for best performing entries. Bonus points for beating the instructor/TA. (C) Dhruv Batra

3 Recap from last time (C) Dhruv Batra

4 Image Credit: Wikipedia
Nearest Neighbours (C) Dhruv Batra Image Credit: Wikipedia

5 Instance/Memory-based Learning
Four things make a memory based learner: A distance metric How many nearby neighbors to look at? A weighting function (optional) How to fit with the local points? (C) Dhruv Batra Slide Credit: Carlos Guestrin

6 1-Nearest Neighbour Four things make a memory based learner:
A distance metric Euclidean (and others) How many nearby neighbors to look at? 1 A weighting function (optional) unused How to fit with the local points? Just predict the same output as the nearest neighbour. (C) Dhruv Batra Slide Credit: Carlos Guestrin

7 1-NN for Regression y x Here, this is the closest datapoint
(C) Dhruv Batra Figure Credit: Carlos Guestrin

8 Multivariate distance metrics
Suppose the input vectors x1, x2, …xN are two dimensional: x1 = ( x11 , x12 ) , x2 = ( x21 , x22 ) , …xN = ( xN1 , xN2 ). One can draw the nearest-neighbor regions in input space. Dist(xi,xj) =(xi1 – xj1)2+(3xi2 – 3xj2)2 Dist(xi,xj) = (xi1 – xj1)2 + (xi2 – xj2)2 The relative scalings in the distance metric affect region shapes Slide Credit: Carlos Guestrin

9 Euclidean distance metric
Or equivalently, where A Slide Credit: Carlos Guestrin

10 Notable distance metrics (and their level sets)
Scaled Euclidian (L2) Mahalanobis (non-diagonal A) Slide Credit: Carlos Guestrin

11 Minkowski distance (C) Dhruv Batra
Image Credit: By Waldir (Based on File:MinkowskiCircles.svg) [CC BY-SA 3.0 ( via Wikimedia Commons

12 Notable distance metrics (and their level sets)
Scaled Euclidian (L2) L1 norm (absolute) Mahalanobis (non-diagonal A) Linf (max) norm Slide Credit: Carlos Guestrin

13 Plan for today (Finish) Nearest Neighbour
Kernel Classification/Regression Curse of Dimensionality (C) Dhruv Batra

14 Figure Credit: Andrew Moore
1-NN for Regression Often bumpy (overfits) (C) Dhruv Batra Figure Credit: Andrew Moore

15 Figure Credit: Andrew Moore
9-NN for Regression Often bumpy (overfits) (C) Dhruv Batra Figure Credit: Andrew Moore

16 Weighted k-NNs Neighbors are not all the same

17 Kernel Regression/Classification
Four things make a memory based learner: A distance metric Euclidean (and others) How many nearby neighbors to look at? All of them A weighting function (optional) wi = exp(-d(xi, query)2 / σ2) Nearby points to the query are weighted strongly, far points weakly. The σ parameter is the Kernel Width. Very important. How to fit with the local points? Predict the weighted average of the outputs predict = Σwiyi / Σwi (C) Dhruv Batra Slide Credit: Carlos Guestrin

18 Weighting/Kernel functions
wi = exp(-d(xi, query)2 / σ2) (Our examples use Gaussian) (C) Dhruv Batra Slide Credit: Carlos Guestrin

19 Image Credit: Ben Taskar
Effect of Kernel Width What happens as σinf? What happens as σ0? (C) Dhruv Batra Image Credit: Ben Taskar

20 Problems with Instance-Based Learning
Expensive No Learning: most real work done during testing For every test sample, must search through all dataset – very slow! Must use tricks like approximate nearest neighbour search Doesn’t work well when large number of irrelevant features Distances overwhelmed by noisy features Curse of Dimensionality Distances become meaningless in high dimensions (See proof in next lecture) (C) Dhruv Batra

21 Curse of Dimensionality
Consider: Sphere of radius 1 in d-dims Consider: an outer ε-shell in this sphere What is ? (C) Dhruv Batra

22 Curse of Dimensionality
(C) Dhruv Batra

23 What you need to know k-NN Kernel regression/classification
Simplest learning algorithm With sufficient data, very hard to beat “strawman” approach Picking k? Kernel regression/classification Set k to n (number of data points) and chose kernel width Smoother than k-NN Problems with k-NN Curse of dimensionality Irrelevant features often killers for instance-based approaches Slow NN search: Must remember (very large) dataset for prediction (C) Dhruv Batra

24 What you need to know Key Concepts (which we will meet again)
Supervised Learning Classification/Regression Loss Functions Statistical Estimation Training vs Testing Data Hypothesis Class Overfitting, Generalization (C) Dhruv Batra

Download ppt "ECE 5424: Introduction to Machine Learning"

Similar presentations

Nonparametric Methods: Nearest Neighbors

Nonparametric Methods: Nearest Neighbors
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) ETHEM ALPAYDIN © The MIT Press, 2010

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) ETHEM ALPAYDIN © The MIT Press, 2010
K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.
1er. Escuela Red ProTIC - Tandil, 18-28 de Abril, 2006 4. Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.

1er. Escuela Red ProTIC - Tandil, de Abril, Instance-Based Learning 4.1 Introduction Instance-Based Learning: Local approximation to the.
Instance Based Learning

Instance Based Learning
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –SVM –Multi-class SVMs –Neural Networks –Multi-layer Perceptron Readings:

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –SVM –Multi-class SVMs –Neural Networks –Multi-layer Perceptron Readings:
MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 
Instance based learning K-Nearest Neighbor Locally weighted regression Radial basis functions.

Instance based learning K-Nearest Neighbor Locally weighted regression Radial basis functions.
Learning from Observations Chapter 18 Section 1 – 4.

Learning from Observations Chapter 18 Section 1 – 4.
Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.
Instance Based Learning

Instance Based Learning
These slides are based on Tom Mitchell’s book “Machine Learning” Lazy learning vs. eager learning Processing is delayed until a new instance must be classified.

These slides are based on Tom Mitchell’s book “Machine Learning” Lazy learning vs. eager learning Processing is delayed until a new instance must be classified.
INSTANCE-BASE LEARNING

INSTANCE-BASE LEARNING
ECE 5984: Introduction to Machine Learning

ECE 5984: Introduction to Machine Learning
COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.

COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.
Overview of Supervised Learning. 2015-10-23Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Unsupervised Learning: Kmeans, GMM, EM Readings: Barber 20.1-20.3.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Unsupervised Learning: Kmeans, GMM, EM Readings: Barber
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Nearest Neighbour Readings: Barber 14 (kNN)

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Nearest Neighbour Readings: Barber 14 (kNN)
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Logistic Regression –NB & LR connections Readings: Barber.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Logistic Regression –NB & LR connections Readings: Barber.
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour.

Similar presentations

Ads by Google