1. 580.704 Mathematical Foundations of BME Reza Shadmehr
logistic regression, iterative re-weighted least squares

2. Logistic regression
In the last lecture we classified by computing a posterior probability. The posterior was calculated by modeling the likelihood and prior for each class.
To compute the posterior, we modeled the right side of the equation below by assuming that they were Gaussians and computed their parameters (or used a kernel estimate of the density).
In logistic regression, we want to directly model the posterior as a function of the variable x.
In practice, when there are k classes to classify, we model:

3. Classification by maximizing the posterior distribution
In this example we assume that the two distributions for the classes have equal variance. Suppose we want to classify a person as male or female based on height.
What we have:
What we want:
Probability of height, given that you have observed a female.
Height is normally distributed in the population of men and in the population of women, with different means, and similar variances. Let y be an indicator variable for being a female. Then the conditional distribution of x (the height becomes):

4. Posterior probability for classification when we have two classes:
Probability of the subject being female, given that you have observed height x. Note that here we are assuming that sigmas are equal for the two distributions. Only in this condition the x variable in the denominator appears linearly inside the exponential. If the two distributions have different sigmas, then the x-variable will be a squared term.

5. Computing the probability that the subject is female, given that we observed height x.
a logistic function
In the denominator, x appears linearly inside the exponential
120
140
160
180
200
220
0.2
0.4
0.6
0.8 1
So if we assume that the class membership densities p(x/y) are normal with equal variance, then the posterior probability will be a logistic function.
Posterior:

6. Logistic regression with assumption of equal variance among density of classes implies a linear decision boundary -4 -2 2 4 6
Class 0

7. Logistic regression: problem statement
Assumption of equal variance among the clusters
The goal is to find parameters w that maximize the log-likelihood.

8. Some useful properties of the logistic function

9. Online algorithm for logistic regression

10. Batch algorithm: Iteratively Re-weighted Least Squares

11. Iteratively Re-weighted Least Squares
IRLS
0.2
0.4
0.6
0.8 5 6 7 8 9 10 11
certain
certain
Sensitivity to error
uncertain

12. Iteratively Re-weighted Least Square: Example-6 -4 -2 2 4 6 -6 -4 -2 2 x1 4 x2
0.25
0.5
0.75 1

13. Modeling the posterior when the densities have unequal variance
(uni-variate case with two classes)

14. Logistic regression with basis functions
By using non-linear bases, we can deal with clusters having unequal variance.
Estimated posterior probability -8 -6 -4 -2 -1 1 2 3 4

15. Logistic function for multiple classes with equal variance
Rather than modeling the posterior directly, let us pick the posterior for one class as our reference and then model the ratio of the posterior for all other classes with respect to that class. Suppose we have k classes:

16. Logistic function for multiple classes with equal variance: soft-max
A “soft-max” function

17. Classification of multiple classes with equal variance
160
180
200
220
0.005
0.01
0.015
0.02
0.025
160
180
200
220
0.0025
0.005
0.0075
0.01
0.0125
0.015
Posterior probabilities
160
180
200
220
0.2
0.4
0.6
0.8 1
160
180
200
220
-15
-10 -5 5 10 15