ECE 5424: Introduction to Machine Learning

Slides:

Advertisements

Similar presentations

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Regression –Model selection, Cross-validation –Error decomposition.

Advertisements

Parameter and Structure Learning Dhruv Batra, Recitation 10/02/2008.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO MACHINE LEARNING Bayesian Estimation.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Parameter Learning in Markov Nets Dhruv Batra, Recitation 11/13/2008.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –SVM –Multi-class SVMs –Neural Networks –Multi-layer Perceptron Readings:

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Classification and risk prediction

Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.

ECE 5984: Introduction to Machine Learning

Simple Bayesian Supervised Models Saskia Klein & Steffen Bollmann 1.

Bayes Classifier, Linear Regression 10701/15781 Recitation January 29, 2008 Parts of the slides are from previous years’ recitation and lecture notes,

Crash Course on Machine Learning

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Naïve Bayes Readings: Barber

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Probability Review –Statistical Estimation (MLE) Readings: Barber 8.1, 8.2.

A C B Small Model Middle Model Large Model Figure 1 Parameter Space The set of parameters of a small model is an analytic set with singularities. Rank.

ECE 6504: Deep Learning for Perception Dhruv Batra Virginia Tech Topics: –Neural Networks –Backprop –Modular Design.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Statistical Estimation (MLE, MAP, Bayesian) Readings: Barber 8.6, 8.7.

CSE 446: Point Estimation Winter 2012 Dan Weld Slides adapted from Carlos Guestrin (& Luke Zettlemoyer)

Empirical Research Methods in Computer Science Lecture 7 November 30, 2005 Noah Smith.

CSE 446 Logistic Regression Winter 2012 Dan Weld Some slides from Carlos Guestrin, Luke Zettlemoyer.

Ch 4. Linear Models for Classification (1/2) Pattern Recognition and Machine Learning, C. M. Bishop, Summarized and revised by Hee-Woong Lim.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Unsupervised Learning: Kmeans, GMM, EM Readings: Barber

Overview of the final test for CSC Overview PART A: 7 easy questions –You should answer 5 of them. If you answer more we will select 5 at random.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Nearest Neighbour Readings: Barber 14 (kNN)

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Logistic Regression –NB & LR connections Readings: Barber.

ECE 6504: Deep Learning for Perception Dhruv Batra Virginia Tech Topics: –Recurrent Neural Networks (RNNs) –BackProp Through Time (BPTT) –Vanishing / Exploding.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Regression Readings: Barber 17.1, 17.2.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Ensemble Methods: Bagging, Boosting Readings: Murphy 16.4; Hastie 16.

Machine Learning CUNY Graduate Center Lecture 6: Linear Regression II.

ECE 6504: Deep Learning for Perception Dhruv Batra Virginia Tech Topics: –Deconvolution.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Expectation Maximization –Principal Component Analysis (PCA) Readings:

A Study on Speaker Adaptation of Continuous Density HMM Parameters By Chin-Hui Lee, Chih-Heng Lin, and Biing-Hwang Juang Presented by: 陳亮宇 1990 ICASSP/IEEE.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Introduction to Machine Learning Nir Ailon Lecture 11: Probabilistic Models.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Model selection –Error decomposition –Bias-Variance Tradeoff –Classification:

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 1: INTRODUCTION.

Crash course in probability theory and statistics – part 2 Machine Learning, Wed Apr 16, 2008.

Matt Gormley Lecture 11 October 5, 2016

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

Dhruv Batra Georgia Tech

Probability Theory and Parameter Estimation I

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

Ch3: Model Building through Regression

CS 2750: Machine Learning Probability Review Density Estimation

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

Special Topics In Scientific Computing

ECE 5424: Introduction to Machine Learning

Probabilistic Models for Linear Regression

ECE 5424: Introduction to Machine Learning

ECE 5424: Introduction to Machine Learning

Pattern Recognition and Machine Learning

Support Vector Machine I

Parametric Methods Berlin Chen, 2005 References:

Dhruv Batra Georgia Tech

Presentation transcript:

ECE 5424: Introduction to Machine Learning Topics: Regression Readings: Barber 17.1, 17.2 Stefan Lee Virginia Tech

Administrativia HW1 Project Proposal HW2 39 Submissions 58 Students Enrolled 19 MIA (?????) Project Proposal Due: 09/21, 11:55 pm (LESS THAN A WEEK!) <= 2pages, NIPS format HW2 Out today Due on Wednesday 09/28, 11:55pm Please please please please please start early Implement Linear Regression, Naïve Bayes, Logistic Regression (C) Dhruv Batra

Recap of last time (C) Dhruv Batra

Learning a Gaussian Collect a bunch of data Learn parameters Hopefully, i.i.d. samples e.g., exam scores Learn parameters Mean Variance (C) Dhruv Batra

MLE for Gaussian Prob. of i.i.d. samples D={x1,…,xN}: Log-likelihood of data: (C) Dhruv Batra Slide Credit: Carlos Guestrin

Your second learning algorithm: MLE for mean of a Gaussian What’s MLE for mean? (C) Dhruv Batra Slide Credit: Carlos Guestrin

Learning Gaussian parameters MLE: (C) Dhruv Batra

Bayesian learning of Gaussian parameters Conjugate priors Mean: Gaussian prior Variance: Inverse Gamma or Wishart Distribution Prior for mean: (C) Dhruv Batra Slide Credit: Carlos Guestrin

MAP for mean of Gaussian (C) Dhruv Batra Slide Credit: Carlos Guestrin

New Topic: Regression (C) Dhruv Batra

Figure Credit: Andrew Moore 1-NN for Regression Often bumpy (overfits) (C) Dhruv Batra Figure Credit: Andrew Moore

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Linear Regression Demo http://hspm.sph.sc.edu/courses/J716/demos/LeastSquares/LeastSquaresDemo.html (C) Dhruv Batra

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Plan for Today Regression Linear Regression Recap Some matrix calculus review Connections with Gaussians The outlier problem  Robust Least Squares (C) Dhruv Batra

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

But, why? Why sum squared error??? Gaussians, Watson, Gaussians… (C) Dhruv Batra

Slide Credit: Greg Shakhnarovich (C) Dhruv Batra Slide Credit: Greg Shakhnarovich

MLE Under Gaussian Model On board (C) Dhruv Batra

Is OLS Robust? Demo http://www.calpoly.edu/~srein/StatDemo/All.html Bad things happen when the data does not come from your model! How do we fix this? (C) Dhruv Batra

Robust Linear Regression y ~ Lap(w’x, b) On board (C) Dhruv Batra