1. Linear Regression Models
Andy Wang
CIS
Computer Systems
Performance Analysis

2. Linear Regression Models
What is a (good) model?
Estimating model parameters
Allocating variation
Confidence intervals for regressions
Verifying assumptions visually

3. What Is a (Good) Model?
For correlated data, model predicts response given an input
Model should be equation that fits data
Standard definition of “fits” is least-squares
Minimize squared error
Keep mean error zero
Minimizes variance of errors

4. Least-Squared Error If then error in estimate for xi is
Minimize Sum of Squared Errors (SSE)
Subject to the constraint

5. Estimating Model Parameters
Best regression parameters are
where
Note error in book!

6. Parameter Estimation Example
Execution time of a script for various loop counts:
= 6.8, = 2.32, xy = 88.7, x2 = 264
b0 = 2.32  (0.30)(6.8) = 0.28

7. Graph of Parameter Estimation Example

8. Allocating Variation If no regression, best guess of y is
Observed values of y differ from , giving rise to errors (variance)
Regression gives better guess, but there are still errors
We can evaluate quality of regression by allocating sources of errors

9. The Total Sum of Squares
Without regression, squared error is

10. The Sum of Squares from Regression
Recall that regression error is
Error without regression is SST
So regression explains SSR = SST - SSE
Regression quality measured by coefficient of determination

11. Evaluating Coefficient of Determination
Compute
where R = R(x,y) = correlation(x,y)

12. Example of Coefficient of Determination
For previous regression example
y = 11.60, y2 = 29.88, xy = 88.7,
SSE = (0.28)(11.60)-(0.30)(88.7) = 0.028
SST = = 2.97
SSR = = 2.94
R2 = ( )/2.97 = 0.99

13. Standard Deviation of Errors
Variance of errors is SSE divided by degrees of freedom
DOF is n2 because we’ve calculated 2 regression parameters from the data
So variance (mean squared error, MSE) is SSE/(n2)
Standard dev. of errors is square root: (minor error in book)

14. Checking Degrees of Freedom
Degrees of freedom always equate:
SS0 has 1 (computed from )
SST has n1 (computed from data and , which uses up 1)
SSE has n2 (needs 2 regression parameters) So
We’ve talked about DOF for all except SSR. The above equation defines the DOF for SSR (indirectly).

15. Example of Standard Deviation of Errors
For regression example, SSE was 0.03, so MSE is 0.03/3 = 0.01 and se = 0.10
Note high quality of our regression:
R2 = 0.99
se = 0.10
Our samples were generated by timing a shell script that looped for different x values.

16. Confidence Intervals for Regressions
Regression is done from a single population sample (size n)
Different sample might give different results
True model is y = 0 + 1x
Parameters b0 and b1 are really means taken from a population sample

17. Calculating Intervals for Regression Parameters
Standard deviations of parameters:
Confidence intervals are where t has n - 2 degrees of freedom
Not divided by sqrt(n)

18. Example of Regression Confidence Intervals
Recall se = 0.13, n = 5, x2 = 264, = 6.8 So
Using 90% confidence level, t0.95;3 = 2.353

19. Regression Confidence Example, cont’d
Thus, b0 interval is
Not significant at 90%
And b1 is
Significant at 90% (and would survive even 99.9% test)

20. Confidence Intervals for Predictions
Previous confidence intervals are for parameters
How certain can we be that the parameters are correct?
Purpose of regression is prediction
How accurate are the predictions?
Regression gives mean of predicted response, based on sample we took

21. Predicting m Samples
Standard deviation for mean of future sample of m observations at xp is
Note deviation drops as m 
Variance minimal at x =
Use t-quantiles with n–2 DOF for interval

22. Example of Confidence of Predictions
Using previous equation, what is predicted time for a single run of 8 loops?
Time = (8) = 2.68
Standard deviation of errors se = 0.10
90% interval is then

23. Prediction Confidence
Red dot is mean; blue lines indicate confidence intervals for predictions. Prediction is most accurate at mean. As you get farther from the mean, confidence drops. If you try to predict very far outside the sampled data, confidence is very poor.

24. Verifying Assumptions Visually
Regressions are based on assumptions:
Linear relationship between response y and predictor x
Or nonlinear relationship used in fitting
Predictor x nonstochastic and error-free
Model errors statistically independent
With distribution N(0,c) for constant c
If assumptions violated, model misleading or invalid

25. Testing Linearity Scatter plot x vs. y to see basic curve type Linear
Piecewise Linear
Outlier
Nonlinear (Power)

26. Testing Independence of Errors
Scatter-plot i versus
Should be no visible trend
Example from our curve fit:

27. More on Testing Independence
May be useful to plot error residuals versus experiment number
In previous example, this gives same plot except for x scaling
No foolproof tests
“Independence” test really disproves particular dependence
Maybe next test will show different dependence

28. Testing for Normal Errors
Prepare quantile-quantile plot of errors
Example for our regression:

29. Testing for Constant Standard Deviation
Tongue-twister: homoscedasticity
Return to independence plot
Look for trend in spread
Example:

30. Linear Regression Can Be Misleading
Regression throws away some information about the data
To allow more compact summarization
Sometimes vital characteristics are thrown away
Often, looking at data plots can tell you whether you will have a problem

31. Example of Misleading Regression
I II III IV
x y x y x y x y

32. What Does Regression Tell Us About These Data Sets?
Exactly the same thing for each!
N = 11
Mean of y = 7.5
Y = X
Standard error of regression is 0.118
All the sums of squares are the same
Correlation coefficient = .82
R2 = .67

33. Now Look at the Data PlotsI II
III IV

34. White Slide