1. Chapter 4: Linear Models for Classification
Grit Hein & Susanne Leiberg

2. Goal
Our goal is to “classify” input vectors x into one of k classes. Similar to regression, but the output variable is discrete.
input space is divided into decision regions whose boundaries are called decision boundaries or decision surfaces
linear models for classification: decision boundaries are linear functions of input vector x
Decision
boundaries

3. Classifier seek an ‘optimal’ separation of classes (e.g., apples and
diameter
color
linear decision
boundary
weight vector
discriminant dimension
Classifier seek an ‘optimal’ separation of classes (e.g., apples and
oranges) by finding a set of weights for combining features
(e.g., color and diameter).

4. of posterior probabilities
Classifier
Discriminant
function
no computation
of posterior probabilities
(probability of certain class given the data)
directly
map each x onto
a class label
Probabilistic
Generative
Models
Discriminative
computation
of posterior probabilities
model class priors
(p(Ck)) & class-conditional
densities (p(x/Ck))
use to compute posterior
probabilities (p(Ck/x))
model posterior
directly
Logistic
Regression
Tools
Least Square
Classification
Fisher’s Linear
Discriminant
Tools
Tools
Bayes

5. Pros and Cons of the three approaches
Discriminant Functions are the most simple and intuitive approach to
classifying data, but do not allow to
compensate for class priors (e.g. class 1 is a very rare disease)
minimize risk (e.g. classifying sick person as healthy more costly than
classifying healthy person as sick)
implement reject option (e.g. person cannot be classified as sick or healthy
with a sufficiently high probability)
Probabilistic Generative and Discriminative models can do all that

6. Pros and Cons of the three approaches
Generative models provide a probabilistic model of all variables that allows to synthesize new data – but -
generating all this information is computationally expensive and complex and is not needed for a simple classification decision
Discriminative models provide a probabilistic model for the target variable
(classes) conditional on the observed variables
this is usually sufficient for making a well-informed classification decision
without the disadvantages of the simple Discriminant Functions

7. of posterior probabilities
Classifier
Discriminant
function
no computation
of posterior probabilities
(probability of certain class given the data)
directly
map each x onto
a class label
Least Square
Classification
Fisher’s Linear
Discriminant
Tools

8. Discriminant functions
are functions that are optimized to assign input x to one of k classes
y(x) = wTx + ω0
feature 2
Decision region 1
decision boundary
Decision region 2
w determines orientation
of decision boundary
ω0 determines location
feature 1

9. Discriminant functions - How to determine parameters?
Least Squares for Classification
General Principle: Minimize the squared distance (residual) between the observed data point and its prediction by a model function

10. Discriminant functions - How to determine parameters?
In the context of classification: find the parameters which minimize the squared distance (residual) between the data points and the decision boundary

11. Discriminant functions - How to determine parameters?
Problem: sensitive to outliers; also distance between the outliers and the discriminant function is minimized --> can shift function in a way that leads to misclassifications
least squares
logistic regression

12. Discriminant functions - How to determine parameters?
Fisher’s Linear Discriminant
General Principle: Maximize distance between means of different classes while minimizing the variance within each class
maximizing between-class variance & minimizing within-class variance
maximizing
between-class
variance

13. Probabilistic Generative Models
model class-conditional densities (p(x⎮Ck)) and class priors (p(Ck))
use them to compute posterior class probabilities (p(Ck⎮x)) according to Bayes theorem
posterior probabilities can be described as logistic sigmoid function
inverse of sigmoid function is the logit function which represents the
ratio of the posterior probabilities for the two classes
ln[p(C1⎮x)/p(C2⎮x)] --> log odds

14. Probabilistic Discriminative Models - Logistic Regression
you model the posterior probabilities directly assuming that they have a sigmoid-shaped distribution (without modeling class priors and class-conditional densities)
the sigmoid-shaped function (σ) is model function of logistic regressions
first non-linear transformation of inputs using a vector of basis functions ϕ(x) → suitable choices of basis functions can make the modeling of the posterior probabilities easier
p(C1/ϕ) = y(ϕ) = σ(wTϕ)
p(C2/ϕ) = 1-p(C1/ϕ)

15. Probabilistic Discriminative Models - Logistic Regression
Parameters of the logistic regression model determined by maximum likelihood estimation
maximum likelihood estimates are computed using iterative reweighted least squares → iterative procedure that minimizes error function using mathematical algorithms (Newton-Raphson iterative optimization scheme)
that means starting from some initial values the weights are changed until the likelihood is maximized

16. Normalizing posterior probabilities
To compare models and to use posterior probabilities in Bayesian Logistic Regression it is useful to have posterior probabilities in Gaussian form
LAPLACE APPROXIMATION is the tool to find a Gaussian approximation to a probability density defined over a set of continuous variables; here it is used to find a gaussian approximation of your posterior probabilities
Goal is to find Gaussian
approximation q(z) centered on
the mode of p(z)
Z = unknown
normalization constant
p(z) = 1/Z f(z)
q(z)
p(z)

17. How to find the best model? - Bayes Information Criterion (BIC)
the approximation of the normalization constant Z can be used to obtain an approximation for the model evidence
Consider data set D and models {Mi} having parameters {θi}
For each model define likelihood p(D|θi,Mi}
Introduce prior over parameters p(θi|Mi)
Need model evidence p(D|Mi) for various models
Z is approximation of model evidence p(D|Mi)

18. Making predictions
having obtained a Gaussian approximation of your posterior distribution (using Laplace approximation) you can make predictions for new data using BAYESIAN LOGISTIC REGRESSION
you use the normalized posterior distribution to arrive at a predictive distribution for the classes given new data
you marginalize with respect to the normalized posterior distribution

20. color diameter discriminant dimension weight vector linear decision
boundary
weight vector
discriminant dimension

21. Terminology Two classes
single target variable with binary representation
t ∈ {0,1}; t = 1 → class C1, t = 0 → class C2
K > 2 classes
1-of-K coding scheme; t is vector of length K
t = (0,1,0,0,0)T