Ridge regression and Bayesian linear regression Kenneth D. Harris 6/5/15.

Slides:



Advertisements
Similar presentations
Ordinary Least-Squares
Advertisements

Overview of Techniques Case 1 Independent Variable is Groups, or Conditions Dependent Variable is continuous ( ) One sample: Z-test or t-test Two samples:
Geometric Representation of Regression. ‘Multipurpose’ Dataset from class website Attitude towards job –Higher scores indicate more unfavorable attitude.
1 1 Chapter 5: Multiple Regression 5.1 Fitting a Multiple Regression Model 5.2 Fitting a Multiple Regression Model with Interactions 5.3 Generating and.
Notes Sample vs distribution “m” vs “µ” and “s” vs “σ” Bias/Variance Bias: Measures how much the learnt model is wrong disregarding noise Variance: Measures.
Biointelligence Laboratory, Seoul National University
Pattern Recognition and Machine Learning
Lecture 3 HSPM J716. Efficiency in an estimator Efficiency = low bias and low variance Unbiased with high variance – not very useful Biased with low variance.
CS Statistical Machine learning Lecture 13 Yuan (Alan) Qi Purdue CS Oct
 Coefficient of Determination Section 4.3 Alan Craig
LINEAR REGRESSION: What it Is and How it Works Overview What is Bivariate Linear Regression? The Regression Equation How It’s Based on r.
The Kernel Trick Kenneth D. Harris 3/6/15.
Predictions 1. Multiple linear regression Kenneth D. Harris April 29, 2015.
Lasso, Support Vector Machines, Generalized linear models Kenneth D. Harris 20/5/15.
Today Linear Regression Logistic Regression Bayesians v. Frequentists
Statistics 350 Lecture 16. Today Last Day: Introduction to Multiple Linear Regression Model Today: More Chapter 6.
Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.
Arizona State University DMML Kernel Methods – Gaussian Processes Presented by Shankar Bhargav.
Modeling Gene Interactions in Disease CS 686 Bioinformatics.
Review of the fundamental concepts of probability Exploratory data analysis: quantitative and graphical data description Estimation techniques, hypothesis.
Discriminant Analysis Testing latent variables as predictors of groups.
Classification and Prediction: Regression Analysis
Statistical hypothesis testing – Inferential statistics II. Testing for associations.
Helsinki University of Technology Adaptive Informatics Research Centre Finland Variational Bayesian Approach for Nonlinear Identification and Control Matti.
PATTERN RECOGNITION AND MACHINE LEARNING
Machine Learning CUNY Graduate Center Lecture 3: Linear Regression.
Probabilities Probability Distribution Predictor Variables Prior Information New Data Prior and New Data Overview.
CSC321: 2011 Introduction to Neural Networks and Machine Learning Lecture 11: Bayesian learning continued Geoffrey Hinton.
Multiple Linear Regression. Purpose To analyze the relationship between a single dependent variable and several independent variables.
Simple Linear Regression. Deterministic Relationship If the value of y (dependent) is completely determined by the value of x (Independent variable) (Like.
Chapter 7 Relationships Among Variables What Correlational Research Investigates Understanding the Nature of Correlation Positive Correlation Negative.
Introduction to regression 3D. Interpretation, interpolation, and extrapolation.
Mathematics of PCR and CCA Simon Mason Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January.
ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: ML and Simple Regression Bias of the ML Estimate Variance of the ML Estimate.
PS 225 Lecture 20 Linear Regression Equation and Prediction.
College Prep Stats. x is the independent variable (predictor variable) ^ y = b 0 + b 1 x ^ y = mx + b b 0 = y - intercept b 1 = slope y is the dependent.
Syllabus. We covered Regression in Applied Stats. We will review Regression and cover Time Series and Principle Components Analysis. Reference Book.
–The shortest distance is the one that crosses at 90° the vector u Statistical Inference on correlation and regression.
Psychology 202a Advanced Psychological Statistics November 12, 2015.
Chapter1: Introduction Chapter2: Overview of Supervised Learning
Konstantina Christakopoulou Liang Zeng Group G21
Psychology 202a Advanced Psychological Statistics November 10, 2015.
Data Analytics CMIS Short Course part II Day 1 Part 1: Introduction Sam Buttrey December 2015.
CS Statistical Machine learning Lecture 12 Yuan (Alan) Qi Purdue CS Oct
The coefficient of determination, r 2, is The fraction of the variation in the value of y that is explained by the regression line and the explanatory.
Machine Learning CUNY Graduate Center Lecture 6: Linear Regression II.
Week of March 23 Partial correlations Semipartial correlations
Construction Engineering 221 Probability and Statistics.
CSC321: Lecture 8: The Bayesian way to fit models Geoffrey Hinton.
Analysis and Interpretation: Multiple Variables Simultaneously
Logistic Regression When and why do we use logistic regression?
Simple Linear Regression
Probability and Statistics for Computer Scientists Second Edition, By: Michael Baron Section 11.1: Least squares estimation CIS Computational.
Multiple Imputation using SOLAS for Missing Data Analysis
Linear regression project
Simple Linear Regression
Probabilistic Models for Linear Regression
Roberto Battiti, Mauro Brunato
Probability and Statistics for Computer Scientists Second Edition, By: Michael Baron Section 11.1: Least squares estimation CIS Computational.
Multiple Regression A curvilinear relationship between one variable and the values of two or more other independent variables. Y = intercept + (slope1.
Eco 6380 Predictive Analytics For Economists Spring 2016
Simple Linear Regression
The Multiple Regression Model
1.3 Vector Equations.
Contact: Machine Learning – (Linear) Regression Wilson Mckerrow (Fenyo lab postdoc) Contact:
Bayesian Linear Regression
Multiple Linear Regression
EQUATION 4.1 Relationship Between One Dependent and One Independent Variable: Simple Regression Analysis.
Cases. Simple Regression Linear Multiple Regression.
MGS 8110 Applied Regression/Forecasting
Presentation transcript:

Ridge regression and Bayesian linear regression Kenneth D. Harris 6/5/15

Multiple linear regression What are you predicting? Data typeContinuous Dimensionality1 What are you predicting it from? Data typeContinuous Dimensionalityp How many data points do you have?Enough What sort of prediction do you need?Single best guess What sort of relationship can you assume?Linear

Multiple linear regression What are you predicting? Data typeContinuous Dimensionality1 What are you predicting it from? Data typeContinuous Dimensionalityp How many data points do you have?Not enough What sort of prediction do you need?Single best guess What sort of relationship can you assume?Linear

Multiple predictors, one predicted variable

Too many predictors

Geometric interpretation Signal Noise

Geometric interpretation Signal Noise

Overfitting = large weight vectors

Example

Ridge regression introduces a bias

A quick trick to do ridge regression

Regression as a probability model What are you predicting? Data typeContinuous Dimensionality1 What are you predicting it from? Data typeContinuous Dimensionalityp How many data points do you have?Enough What sort of prediction do you need?Probability distribution What sort of relationship can you assume?Linear

Regression as a probability model

Bayesian linear regression

Bayesian predictions