1. Regression Models - Introduction
In regression models, two types of variables that are studied:
A dependent variable, Y, also called response variable. It is
modeled as random.
An independent variable, X, also called predictor variable or
explanatory variable. It is sometimes modeled as random and
sometimes it has fixed value for each observation.
In regression models we are fitting a statistical model to data.
We generally use regression to be able to predict the value of one
variable given the value of others.
STA302/1001 week 1

2. Simple Linear Regression - Introduction
Simple linear regression studies the relationship between a
quantitative response variable Y, and a single explanatory variable X.
Idea of statistical model: Actual observed value of Y = …
Box (a well know statistician) claim: “All models are wrong, some
are useful”. ‘Useful’ means that they describe the data well and can
be used for predictions and inferences.
Recall: parameters are constants in a statistical model which we usually don’t know but will use data to estimate.
STA302/1001 week 1

3. Simple Linear Regression Models
The statistical model for simple linear regression is a straight line
model of the form where…
For particular points,
We expect that different values of X will produce different mean
response. In particular we have that for each value of X, the possible
values of Y follow a distribution whose mean is
Formally it means that ….
STA302/1001 week 1

4. Estimation – Least Square Method
Estimates of the unknown parameters β0 and β1 based on our
observed data are usually denoted by b0 and b1.
For each observed value xi of X the fitted value of Y is
This is an equation of a straight line.
The deviations from the line in vertical direction are the errors in
prediction of Y and are called “residuals”. They are defined as
The estimates b0 and b1 are found by the Method of Lease Squares
which is based on minimizing sum of squares of residuals.
Note, the least-squares estimates are found without making any statistical assumptions about the data.
STA302/1001 week 1

5. Derivation of Least-Squares Estimates
Let
We want to find b0 and b1 that minimize RSS.
Use calculus….
STA302/1001 week 1

6. Properties of Fitted Line
Note: you need to know how to prove the above properties.
STA302/1001 week 1

7. Statistical Assumptions for SLR
Recall, the simple linear regression model is Yi = β0 + β1Xi + εi
where i = 1, …, n.
The assumptions for the simple linear regression model are:
1) E(εi)=0
2) Var(εi) = σ2
3) εi’s are uncorrelated.
These assumptions are also called Gauss-Markov conditions.
The above assumptions can be stated in terms of Y’s…
STA302/1001 week 1

8. Gauss-Markov Theorem
The least-squares estimates are BLUE (Best Linear, Unbiased Estimators).
The least-squares estimates are linear in y’s…
Of all the possible linear, unbiased estimators of β0 and β1 the least squares estimates have the smallest variance.
STA302/1001 week 1