Presentation is loading. Please wait.

Presentation is loading. Please wait.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour.

Published byEthel Whitehead Modified over 9 years ago

Similar presentations

Presentation on theme: "ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour."— Presentation transcript:

1 ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour Readings: Barber 14 (kNN)

2 Administrativia New class room –GBJ 102 More space –Force-adds approved Scholar –Anybody not have access? –Still have problems reading/submitting? Resolve ASAP. –Please post questions on Scholar Forum. –Please check scholar forums. You might not know you have a doubt. (C) Dhruv Batra2

3 Administrativia Reading/Material/Pointers –Slides on Scholar –Scanned handwritten notes on Scholar –Readings/Video pointers on Public Website (C) Dhruv Batra3

4 Administrativia Computer Vision & Machine Learning Reading Group –Meet: Fridays 5-6pm –Reading CV/ML conference papers –Whittemore 654 (C) Dhruv Batra4

5 Plan for today Supervised/Inductive Learning –Setup –Goal: Classification, Regression –Procedural View –Statistical Estimation View –Loss functions Your first classifier: k-Nearest Neighbour (C) Dhruv Batra5

6 Types of Learning Supervised learning –Training data includes desired outputs Unsupervised learning –Training data does not include desired outputs Weakly or Semi-supervised learning –Training data includes a few desired outputs Reinforcement learning –Rewards from sequence of actions (C) Dhruv Batra6

7 Supervised / Inductive Learning Given –examples of a function (x, f(x)) Predict function f(x) for new examples x –Discrete f(x): Classification –Continuous f(x): Regression –f(x) = Probability(x): Probability estimation (C) Dhruv Batra7

8 Slide Credit: Pedro Domingos, Tom Mitchel, Tom Dietterich

9 Supervised Learning Input: x (images, text, emails…) Output: y(spam or non-spam…) (Unknown) Target Function –f: X  Y(the “true” mapping / reality) Data –(x 1,y 1 ), (x 2,y 2 ), …, (x N,y N ) Model / Hypothesis Class –g: X  Y –y = g(x) = sign(w T x) Learning = Search in hypothesis space –Find best g in model class. (C) Dhruv Batra9

10 10 Slide Credit: Yaser Abu-Mostapha

11 Basic Steps of Supervised Learning Set up a supervised learning problem Data collection –Start with training data for which we know the correct outcome provided by a teacher or oracle. Representation –Choose how to represent the data. Modeling –Choose a hypothesis class: H = {g: X  Y} Learning/Estimation –Find best hypothesis you can in the chosen class. Model Selection –Try different models. Picks the best one. (More on this later) If happy stop –Else refine one or more of the above (C) Dhruv Batra11

12 Learning is hard! No assumptions = No learning (C) Dhruv Batra12

13 Klingon vs Mlingon Classification Training Data –Klingon: klix, kour, koop –Mlingon: moo, maa, mou Testing Data: kap Which language? Why? (C) Dhruv Batra13

14 Loss/Error Functions How do we measure performance? Regression: –L 2 error Classification: –#misclassifications –Weighted misclassification via a cost matrix –For 2-class classification: True Positive, False Positive, True Negative, False Negative –For k-class classification: Confusion Matrix (C) Dhruv Batra14

15 Training vs Testing What do we want? –Good performance (low loss) on training data? –No, Good performance on unseen test data! Training Data: –{ (x 1,y 1 ), (x 2,y 2 ), …, (x N,y N ) } –Given to us for learning f Testing Data –{ x 1, x 2, …, x M } –Used to see if we have learnt anything (C) Dhruv Batra15

16 Procedural View Training Stage: –Raw Data  x (Feature Extraction) –Training Data { (x,y) }  f (Learning) Testing Stage –Raw Data  x (Feature Extraction) –Test Data x  f(x) (Apply function, Evaluate error) (C) Dhruv Batra16

17 Statistical Estimation View Probabilities to rescue: –x and y are random variables –D = (x 1,y 1 ), (x 2,y 2 ), …, (x N,y N ) ~ P(X,Y) IID: Independent Identically Distributed –Both training & testing data sampled IID from P(X,Y) –Learn on training set –Have some hope of generalizing to test set (C) Dhruv Batra17

18 Concepts Capacity –Measure how large hypothesis class H is. –Are all functions allowed? Overfitting –f works well on training data –Works poorly on test data Generalization –The ability to achieve low error on new test data (C) Dhruv Batra18

19 Guarantees 20 years of research in Learning Theory oversimplified: If you have: –Enough training data D –and H is not too complex –then probably we can generalize to unseen test data (C) Dhruv Batra19

20 New Topic: Nearest Neighbours (C) Dhruv Batra20Image Credit: Wikipedia

21 Synonyms Nearest Neighbours k-Nearest Neighbours Member of following families: –Instance-based Learning –Memory-based Learning –Exemplar methods –Non-parametric methods (C) Dhruv Batra21

22 Nearest Neighbor is an example of…. Instance-based learning Has been around since about 1910. To make a prediction, search database for similar datapoints, and fit with the local points. Assumption: Nearby points behavior similarly wrt y x 1 y 1 x 2 y 2 x 3 y 3. x n y n (C) Dhruv Batra22

23 Instance/Memory-based Learning Four things make a memory based learner: A distance metric How many nearby neighbors to look at? A weighting function (optional) How to fit with the local points? Slide Credit: Carlos Guestrin(C) Dhruv Batra23

24 1-Nearest Neighbour Four things make a memory based learner: A distance metric –Euclidean (and others) How many nearby neighbors to look at? –1 A weighting function (optional) –unused How to fit with the local points? –Just predict the same output as the nearest neighbour. Slide Credit: Carlos Guestrin(C) Dhruv Batra24

25 k-Nearest Neighbour Four things make a memory based learner: A distance metric –Euclidean (and others) How many nearby neighbors to look at? –k A weighting function (optional) –unused How to fit with the local points? –Just predict the average output among the nearest neighbours. (C) Dhruv Batra25Slide Credit: Carlos Guestrin

26 1 vs k Nearest Neighbour (C) Dhruv Batra26Image Credit: Ying Wu

27 1 vs k Nearest Neighbour (C) Dhruv Batra27Image Credit: Ying Wu

28 Nearest Neighbour Demo 1 –http://cgm.cs.mcgill.ca/~soss/cs644/projects/perrier/Nearest. htmlhttp://cgm.cs.mcgill.ca/~soss/cs644/projects/perrier/Nearest. html Demo 2 –http://www.cs.technion.ac.il/~rani/LocBoost/http://www.cs.technion.ac.il/~rani/LocBoost/ (C) Dhruv Batra28

29 Spring 2013 Projects Gender Classification from body proportions –Igor Janjic & Daniel Friedman, Juniors (C) Dhruv Batra29

30 Scene Completion [Hayes & Efros, SIGGRAPH07] (C) Dhruv Batra30

31 Hays and Efros, SIGGRAPH 2007

32 … 200 total Hays and Efros, SIGGRAPH 2007

33 Context Matching Hays and Efros, SIGGRAPH 2007

34 Graph cut + Poisson blending Hays and Efros, SIGGRAPH 2007

35

36

37

38

39

40

Download ppt "ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Supervised Learning –General Setup, learning from data –Nearest Neighbour."

Similar presentations

CHAPTER 2: Supervised Learning. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Learning a Class from Examples.

CHAPTER 2: Supervised Learning. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Learning a Class from Examples.
Indian Statistical Institute Kolkata

Indian Statistical Institute Kolkata
Cos 429: Face Detection (Part 2) Viola-Jones and AdaBoost Guest Instructor: Andras Ferencz (Your Regular Instructor: Fei-Fei Li) Thanks to Fei-Fei Li,

Cos 429: Face Detection (Part 2) Viola-Jones and AdaBoost Guest Instructor: Andras Ferencz (Your Regular Instructor: Fei-Fei Li) Thanks to Fei-Fei Li,
ETHEM ALPAYDIN © The MIT Press, 2010 Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –SVM –Multi-class SVMs –Neural Networks –Multi-layer Perceptron Readings:

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –SVM –Multi-class SVMs –Neural Networks –Multi-layer Perceptron Readings:
MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 
1 Chapter 10 Introduction to Machine Learning. 2 Chapter 10 Contents (1) l Training l Rote Learning l Concept Learning l Hypotheses l General to Specific.

1 Chapter 10 Introduction to Machine Learning. 2 Chapter 10 Contents (1) l Training l Rote Learning l Concept Learning l Hypotheses l General to Specific.
Machine Learning CSE 473. © Daniel S. Weld 2 473 Topics Agency Problem Spaces Search Knowledge Representation Reinforcement Learning InferencePlanning.

Machine Learning CSE 473. © Daniel S. Weld Topics Agency Problem Spaces Search Knowledge Representation Reinforcement Learning InferencePlanning.
1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.

1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.
Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.

Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.
INSTANCE-BASE LEARNING

INSTANCE-BASE LEARNING
© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.
ECE 5984: Introduction to Machine Learning

ECE 5984: Introduction to Machine Learning
INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, 2004 Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
CS 478 - Instance Based Learning1 Instance Based Learning.

CS Instance Based Learning1 Instance Based Learning.
Part I: Classification and Bayesian Learning

Part I: Classification and Bayesian Learning
Crash Course on Machine Learning

Crash Course on Machine Learning
More Machine Learning Linear Regression Squared Error L1 and L2 Regularization Gradient Descent.

More Machine Learning Linear Regression Squared Error L1 and L2 Regularization Gradient Descent.
ECSE 6610 Pattern Recognition Professor Qiang Ji Spring, 2011.

ECSE 6610 Pattern Recognition Professor Qiang Ji Spring, 2011.
Data Mining Joyeeta Dutta-Moscato July 10, 2013. Wherever we have large amounts of data, we have the need for building systems capable of learning information.

Data Mining Joyeeta Dutta-Moscato July 10, Wherever we have large amounts of data, we have the need for building systems capable of learning information.

Similar presentations

Ads by Google