1. with observed random variables
Lecture 4
Supervised Learning
with observed random variables
Linear Regression
Logistic Regression
Naive Bayes

2. Learning Supervised  unsupervised
continuous  discrete RVs --- regression  classification
generative  discriminative
with hidden variables  without hidden variables
This lecture is about supervised learning without hidden variables. We will look at both the generative and the discriminative approach.
Sometimes generative can inspire parameterizations discriminative.
plate notation

3. Linear Regression X  Y with Y continuous and X arbitrary.
discriminative approach: model p(Y|X) directly.
probability model: Gaussian with mean E[Y|X]=f(X)
Given data {Xn,Yn} what is the optimal setting of the parameters in the Maximum Likelihood framework
demo_LinReg
geometric interpretation

4. Classification (discriminative)
X  Y with Y discrete [0,1,2,..D] and X arbitrary.
Discriminative approach, binary Y: Logistic Regression.
Fit (regress) a logistic function to data where
E[Y|X] = logistic(X)  p(Y=1|X) = logistic(x).
Calculation of ML parameters.
demo_LogReg
softmax generalization for general discrete Y.

5. Classification (generative)
Generative approach: model P(X,Y) = P(X|Y) P(Y)
Naive Bayes assumption x_i indep. x_j given Y.
case 1: X = continuous: use Gaussians for P(x_i|Y)
case 2: X = discrete: use multinomial distribution.
classification: max_Y logP(X|Y) + logP(Y)
ML parameters settings have very natural interpretation in terms of frequencies, clusters means etc.