1. Chapter 13: Linear Correlation and Regression Analysis

2. Chapter Goals
More detailed look at linear correlation and regression analysis.
Develop a hypothesis test to determine the strength of a linear relationship.
Consider the line of best fit. Use this to make confidence interval estimations.

3. 13.1: Linear Correlation Analysis
The coefficient of linear correlation, r, is a measure of the strength of a linear relationship.
Consider another measure of dependence: covariance.
Recall: bivariate data - ordered pairs of numerical values.

4. Derivation of the covariance:
Goal: a measure of the linear relationship between two variables.
Consider the following set of bivariate data:
{(8, 22), (5, 28), (8, 18), (4, 16), (13, 27), (15, 23), (17, 17), (12, 13)}
Consider a graph of the data:
1. The point is the centroid of the data.
2. A vertical and horizontal line through the centroid divides the graph into four sections.

5. Graph of the data, with centroid.

6. Note:
1. Each point (x, y) lies a certain distance from each of the two lines.
: the horizontal distance from (x, y) to the vertical line passing through the centroid.
: the vertical distance from (x, y) to the horizontal line passing through the centroid.
4. The distances may be positive, negative, or zero.
5. Consider the product:
a. If the graph has lots of points to the upper right and lower left of the centroid (positive linear relationship), most products will be positive.
b. If the graph has lots of points to the upper left and lower right of the centroid (negative linear relationship), most products will be negative.

7. Covariance:
The covariance of x and y is defined as the sum of the products of the distances of all values x and y from the centroid divided by n - 1.
Note:

8. Calculations for finding covar(x, y):

9. Data and Covariance:
Positive covariance

10. Negative covariance

11. Covariance near 0

12. Problem:
1. The covariance does not have a standardized unit of measure.
2. Suppose we multiply each data point in the example in this section by 15.
The covariance of the new data set is
3. The amount of the dependency between x and y seems stronger. But the relationship is really the same.
4. We must find a way to eliminate the effect of the spread of the data when we measure the strength of a linear relationship.

13. Solution:
1. Standardize x and y:
2. Compute the covariance of x’ and y’.
3. This covariance is not affected by the spread of the data.
4. This is exactly what is accomplished by the coefficient of linear correlation:

14. Note:
1. The coefficient of linear correlation standardizes the measure of dependency and allows us to compare the relative strengths of dependency of different sets of data.
2. Also commonly called Pearson’s product moment, r.
Calculation of r (for the data presented in this section):

15. Alternative (Computational) Formula for r:
1. This formula avoids the separate calculations of the means, standard deviations, and the deviations from the means.
2. This formula is easier and more accurate: minimizes round-off error.

16. 13.2 Inferences About the Linear Correlation Coefficient
Use the calculated value of the coefficient of linear correlation, r*, to make an inference about the population correlation coefficient, r.
Consider a confidence interval for r and a hypothesis test concerning r.

17. Assumptions for inferences about linear correlation coefficient:
The set of (x, y) ordered pairs forms a random sample and the y-values at each x have a normal distribution. Inferences use the t-distribution with n - 2 degrees of freedom.
Caution:
The inferences about the linear correlation coefficient are about the pattern of behavior of the two variables involved and the usefulness of one variable in predicting the other. Significance of the linear correlation coefficient does not mean there is a direct cause-and-effect relationship.

18. Confidence Interval Procedure:
1. A confidence interval may be used to estimate the value of the population correlation coefficient, r.
2. Use a table showing confidence belts.
3. Table 10, Appendix B: confidence belts for 95% confidence intervals.
4. Table 10 utilizes n, the sample size.

19. Example: A random sample of 25 ordered pairs of data have a calculated value of r = Find a 95% confidence interval for r, the population linear correlation coefficient.
Solution:
1. Population Parameter of Concern:
The linear correlation coefficient for the population, r.
2. The Confidence Interval Criteria:
a. Assumptions: The ordered pairs form a random sample, and for each x, the y-values have a mounded distribution.
b. Test statistic: The calculated value of r.
c. Confidence level: 1 - a = 0.95
3. Sample Evidence:
n = 25 and r = -0.45

20. 4. The Confidence Interval:
The confidence interval is read from Table 10, Appendix B.
Find r = at the bottom of Table 10.
Visualize a vertical line through that point.
Find the two points where the belts marked for the correct sample size cross the vertical line.
Draw a horizontal line through each point to the vertical scale on the left and read the confidence interval.
The values are and -0.12
5. The Results:
-0.68 to is the 95% confidence interval for r.

21. Hypothesis-Testing Procedure:
1. Null hypothesis: the two variables are linearly unrelated, r = 0.
2. Alternative hypothesis: one- or two-tailed, usually r 0
3. Test statistic: calculated value of r.
4. Probability bounds or critical values for r: Table 11, Appendix B.
5. Number of degrees of freedom for the r-statistic: n - 2.

22. Example: In a study of 32 randomly selected ordered pairs, r = 0. 421
Example: In a study of 32 randomly selected ordered pairs, r = Is there any evidence to suggest the linear correlation coefficient is different from 0 at the 0.05 level of significance?
Solution:
1. The Set-up:
a. Population parameter of concern: The linear correlation coefficient for the population, r.
b. The null and the alternative hypothesis:
H0: r = 0
Ha: r 0

23. 2. The Hypothesis Test Criteria:
a. Assumptions: The ordered pairs form a random sample and we will assume that the y-values at each x have a mounded distribution.
b. Test statistic:
r* (calculated value of r) with df = = 30
c. Level of significance: a = 0.05
3. The Sample Evidence:
n = 32 and r = r* = 0.421

24. 4. The Probability Distribution (Classical Approach):
a. Critical Value: The critical value is found at the intersection of the df = 30 row and the two-tailed column of Table 11: 0.349
b. r* is in the critical region.
4. The Probability Distribution (p-Value Approach):
a. The p-value: Use Table 11: < P < 0.02
b. The p-value is smaller than the level of significance, a.
5. The Results:
a. Decision: Reject H0.
b. Conclusion: At the 0.05 level of significance, there is evidence to suggest x and y are correlated.

25. 13.3: Linear Regression Analysis
Line of best fit results from an analysis of two (or more) related variables.
Try to predict the value of the dependent, or output, variable.
The variable we control is the independent, or input, variable.

26. Method of Least Squares:
The line of best fit:
The slope:
The y-intercept:
Note:
1. A scatter plot may suggest curvilinear regression.
2. If two or more input variables are used: multiple regression.

27. The Linear Model:
This equation represents the linear relationship between the two variables in a population.
b0: The y-intercept, estimated by b0.
b1: The slope, estimated by b1.
e: Experimental error, estimated by
The random variable e is called the residual.
e is the difference between the observed value of y and the predicted value of y at a given x.
The sum of the residuals is exactly zero.
Mean value of experimental error is zero: me = 0
Variance of experimental error:

28. Estimating the Variance of the Experimental Error:
Assumption: The distribution of y’s is approximately normal and the variances of the distributions of y at all values of x are the same (The standard deviation of the distribution of y about y is the same for all values of x.).
Consider the sample variance:
1. The variance of y involves an additional complication: there is a different mean for y at each value of x.
2. Each “mean” is actually the predicted value, y
3. Variance of the error e estimated by:
Degrees of freedom: n - 2

29. Rewriting
SSE = sum of squares for error

30. Example: A recent study was conducted to determine the relation between advertising expenditures and sales of statistics texts (for the first year in print). The data is given below (in thousands). Find the line of best fit and the variance of y about the line of best fit.

31. Solution:

32. The equation for the line of best fit:
The variance of y about the regression line:
Note: Extra decimal places are often needed for this type of calculation.

33. Scatter plot, regression line, and random errors as line segments:

34. Minitab Output: Regression Analysis The regression equation is
C2 = C1
Predictor Coef StDev T P
Constant C
S = R-Sq = 74.6% R-Sq(adj) = 71.4%
Analysis of Variance
Source DF SS MS F P
Regression
Residual Error
Total

35. 13.4: Inferences Concerning the Slope of the Regression Line
Null hypothesis: the equation of the line of best fit is of no value in predicting y given x (b1 = 0).
Use a t test.

36. Sampling Distribution of the Slope b1:
Assume: Random samples of size n are repeatedly taken from a bivariate population.
1. b1 has a sampling distribution that is approximately normal.
2. The mean of b1 is b1.
3. The variance of b1 is
provided there is no lack of fit.

37. Estimator for
The standard error of regression (slope) is
and is estimated by
Example (continued): For the advertising costs and sales data:

38. Assumptions for inferences about the slope parameter b1:
The set of (x, y) ordered pairs forms a random sample and the y-values at each x have a normal distribution. Since the population standard deviation is unknown and replaced with the sample standard deviation, the t-distribution will be used with n - 2 degrees of freedom.
Confidence Interval Procedure:
The 1 - a confidence interval for b1 is given by

39. Example: Find the 95% confidence interval for the population slope b1 for the advertising costs and sales example.
Solution:
1. Population parameter of Interest:
The slope, b1, for the line of best fit for the population.
2. The Confidence Interval Criteria:
a. Assumptions: The ordered pairs form a random sample and we will assume the y-values (sales) at each x (advertising costs) have a mounded distribution.
b. Test statistic: t with df = = 8
c. Confidence level: 1 - a = 0.95
3. Sample Evidence:
Sample information:

40. 4. The Confidence Interval:
a. Confidence coefficients:
t(df, a/2) = t(8, 0.025) = 2.31
b. Interval:
5. The Results:
The slope of the line of best fit of the population from which the sample was drawn is between and with 95% confidence.

41. Hypothesis-Testing Procedure:
1. Null hypothesis is always H0: b1 = 0
2. Use the Students t distribution with df = n - 2.
3. The test statistic:

42. Example: In the previous example, is the slope for the line of best fit significant enough to show that advertising cost is useful in predicting the first year sales? Use a = 0.05
Solution:
1. The Set-up:
a. Population parameter of concern: The parameter of concern is b1, the slope of the line of best fit for the population.
b. The null and alternative hypothesis:
H0: b1 = 0 (x is of no use in predicting y)
Ha: b1 > 0 (we expect sales to increase as costs increase)

43. 2. The Hypothesis Test Criteria:
a. Assumptions: The ordered pairs form a random sample and we will assume the y-values (sales) at each x (advertising costs) have a mounded distribution.
b. Test statistic: t* with df = n - 2 = 8
c. Level of significance: a = 0.05
3. The Sample Evidence:
a. Sample information:
b. Calculate the value of the test statistic:

44. 4. The Probability Distribution (Classical Approach):
a. Critical value: t(8, 0.05) = 1.86
b. t* is in the critical region.
4. The Probability Distribution (p-Value Approach):
a. The p-value: P = P(t* > , with df = 8) < 0.001
b. The p-value is smaller than the level of significance, a.
5. The Results:
a. Decision: Reject H0.
b. Conclusion: At the 0.05 level of significance, there is evidence to suggest the slope of the line of best fit is greater than zero. The evidence indicates there is a linear relationship and that advertising cost (x) is useful in predicting the first year sales (y).

45. 13.5: Confidence Interval Estimates for Regression
Use the line of best fit to make predictions.
Predict the population y-value at a given x.
Predict the individual y-value selected at random that will occur at a given value of x.
The best point estimate, or prediction, for both is y

46. Notation:
1. Mean of the population y-values at a given value of x:
2. The individual y-value selected at random for a given value of x:
Background:
1. Recall: the development of confidence intervals for the population mean m when the variance was known and when the variance was estimated.
2. The confidence interval for and the prediction interval for are constructed in a similar fashion.
3. y replaces as the point estimate.
4. The sampling distribution of y is normal.

47. 5. The standard deviation in both cases is computed by
5. The standard deviation in both cases is computed by multiplying the square root of the variance of the error by an appropriate correction factor.
6. The line of best fit passes through the centroid:
Consider a confidence interval for the slope b1.
If we draw lines with slopes equal to the extremes of that confidence interval through the centroid, the value for y fluctuates considerably for different values of x (See the Figure on the next slide.).
It is reasonable to expect a wider confidence interval as we consider values of x further from .
We need a correction factor to adjust for the distance between x0 and .
This factor must also adjust for the variation of the y- values about y.

48. Lines Representing the Confidence Interval for Slope:
Slope is 8.676
Slope is 5.707

49. Confidence interval for the mean value of y at a given value of x,
Note:
1. The numerator of the second term under the radical sign is the square of the distance of x0 from
2. The denominator is closely related to the variance of x and has a standardizing effect on this term.
standard error of y

50. Example: It is believed that the amount of nitrogen fertilizer used per acre has a direct effect on the amount of wheat produced. The data below shows the amount of nitrogen fertilizer used per test plot and the amount of wheat harvested per test plot.
a. Find the line of best fit.
b. Construct a 95% confidence interval for the mean amount of wheat harvested for 45 pounds of fertilizer.

51. Solution:
Using Minitab, the line of best fit:
Confidence Interval:
1. Population Parameter of Interest:
The mean amount of wheat produced for 45 pounds of fertilizer,
2. The Confidence Interval Criteria:
a. Assumptions: The ordered pairs form a random sample and the y-values at each x have a mounded distribution.
b. Test statistic: t with df = = 14
c. Confidence level: 1 - a = 0.95
3. Sample Information:

52. 4. The Confidence Interval:

53. Confidence interval: green vertical line.
Confidence interval belt: upper and lower boundaries of all 95% confidence intervals.
Line of best fit
Upper boundary for
Lower boundary for

54. Prediction interval of the value of a single randomly selected y:
Example: Find the 95% prediction interval for the amount of wheat harvested for 45 pounds of fertilizer.
Solution:
1. Population Parameter of Interest:
yx=45, the amount of wheat harvested for 45 pounds of fertilizer

55. 2. The Confidence Interval Criteria:
a. Assumptions: The ordered pairs form a random sample and the y-values at each x have a mounded distribution.
b. Test statistic: t with df = = 14
c. Confidence level: 1 - a = 0.95
3. Sample Information:

56. 4. The Confidence Interval:

57. Prediction belts for Line of best fit
Upper boundary on individual y-values
Lower boundary for 95% prediction interval on individual y-values at any x
x0 = 45

58. Precautions:
1. The regression equation is meaningful only in the domain of the x variable studied. Estimation outside this domain is risky; it assumes the relationship between x and y is the same outside the domain of the sample data.
2. The results of one sample should not be used to make inferences about a population other than the one from which the sample was drawn.
3. Correlation (or association) does not imply causation. A significant regression does not imply x causes y to change. Most common problem: missing, or third, variable effect.

59. 13.6: Understanding the Relationship Between Correlation and Regression
We have considered correlation and regression analysis.
When do we use these techniques?
Is there any duplication of work?

60. Remarks:
1. The primary use of the linear correlation coefficient is in answering the question “Are these two variables related?”
2. The linear correlation coefficient may be used to indicate the usefulness of x as a predictor of y (if the linear model is appropriate).
The test concerning the slope of the regression line
(H0: b1 = 0) tests the same basic concept.
3. Lack-of-fit test: Is the linear model appropriate?
Consider the scatter diagram.

61. Conclusions:
1. Linear correlation and regression measure different characteristics. It is possible to have a strong linear correlation and have the wrong model.
2. Regression analysis should be used to answer questions about the relationship between to variables.
a. What is the relationship?
b. How are the two variables related?