1. Ch3: Model Building through Regression
3.1 Introduction
Regression: is to model or to determine through
statistical data analysis the explicit relationship
between a set of random variables,
mathematically, where
X : dependent variable (response)
independent variables (regressors)
Regression model:
: sample values of
expectational error for accounting for uncertainty

2. 3.2 Linear Regression Model
Linear function f  linear regression model
Nonlinear function f  nonlinear regression model
3.2 Linear Regression Model
Problem: The unknown stochastic environment is to
be probed using a set of examples (x, d).
Consider the linear regression model:

4. 3.3 ML and MAP Estimations of w
Problem (under stochastic environment):
Given the joint statistics of X, D, W , estimate w.
Methods of estimation:
i) maximum likelihood (ML),
ii) maximum a posterior (MAP)
3.3 ML and MAP Estimations of w
Refer to
(i) x bears no relation to w.
(ii) The information of w is contained in d.
Focus on the joint probability density function

5. From the conditional probability
(Bayesian eq.)
where : observation density of response
d due to regressor x, given parameter w and is
often reformulated as the likelihood function,
i.e.,

6. : prior density of w before any observation.
Let
: posterior density of w after observations.
: evidence of the information contained in d.
Bayesian eq. becomes

7. Maximum likelihood (ML) estimate of w
Maximum a posteriori (MAP) estimate of w or
ML ignores the prior
How to come up with an approximate
Considering a Gaussian environment
Let be the training sample.

8. Assumptions:
(1) are iid.
(2) is described by a Gaussian of zero mean and
common variance , i.e.,
of w are iid. Each element is governed
by a Gaussian density function of zero mean
and common variance

9. The likelihood function measures the similarity between and in turn their difference
From Assumption (2),

10. From Assumption (1), the overall likelihood function

11. From Assumption (3),
where

12. Substitute Eqs. (B) and (C) into Eq. (A),
Substitute into

13. Let
Obtain
where

14. If is large, the prior distribution of each element
of w is close to be uniform and is close to zero,
the MAP estimate reduces to
the ML estimate
The ML estimator is unbiased, i.e.,
while the MAP estimator is biased.

15. 3.4 Relationship between Regularized LS and MAP Estimations of w
Least Squares (LS) Estimation
Define the cost function as
Solve for w by minimizing
Obtain
which is the same solution as
the ML one.
Modify the cost function as
structural
regularizer

16. Solve for w by minimizing
Obtain
which is the same
solution as the MAP one.