Regression & Correlation (1) A relationship between 2 variables X and Y The relationship seen as a straight line Two problems How can we tell if our regression line is useful? Test of hypothesis about the slope, β1 Correlation Useful features of r Test of hypothesis about ρ Examples

A relationship between two variables X & Y We often have pairs of scores for a given set of cases. For example, we might have: # of years of education and annual income, or IQ and GPA income and # of books in the household More generally, we have any X and Y, and our question is, does knowing something about X tell us anything about Y?

A relationship between two variables X & Y Does knowing something about X tell us anything about Y? For example, knowing how many years of education a person has, could you usefully estimate their annual income, or the number of cigarettes they smoke in a year?

A relationship between two variables X & Y Often, the answer to that question is, Yes – there is a relationship between the X and Y scores you have measured. On average, as number of years of education goes up (across a set of people), number of cigarettes smoked per year goes down.

A relationship between two variables X & Y In the graph on the next slide, we see two things: X goes down as Y goes up. At each value of X, there is some variability in Y – but substantially less than there is in Y overall.

Note that the range of the Y values for this value of X is small, compared to the whole range of Y in the data set. Y = Cigarettes per year X = Years of education

The relationship seen as a straight line The relationship between an X and a Y can be described using the equation for a straight line. Y = β0 + β1X + ε Y-intercept Slope Error Note: this is the (theoretical) population equation relating Y to X Note: this equation exists only in theory – it’s the equation we would obtain if we measured X and Y for all cases in the population

Two problems Y = β0 + β1X + ε In principle, this equation would let us predict the value of Y for a given X without error IF A. X were the only variable that influenced Y Usually, it isn’t B. We knew the population values of β0 + β1 Usually, we don’t Note that we can never have the true equation Y = β0 + β1X + ε – that is, we can never know what β0 & β1 are. We can only obtain estimates of these values from our sample data. And most of the time, we’ll be working with criterion variables Y which are influenced by many X variables

Be sure to distinguish between Actual values of Y in the population. Two problems Be sure to distinguish between Actual values of Y in the population. Values of Y we would predict using Y = β0 + β1X + ε if we had the population values for β0 + β1. C. Values of Y we predict on the basis of the X-Y relationship in our sample data: Y = β0 + β1X ^ ^ Why no ε here?

Two problems When we predict Y on the basis of X for a given case, two things can cause the predicted values to be different from the values we would find if we actually measured Y for that case: 1. We don’t know the population values of β0 and β1 – only the sample values β0 and β1. Note that if we did know β0 & β1, this source of error would disappear. ^ ^

Two problems 2. In the population, Y is not uniquely determined by X. As a result, for each value of X, there is a distribution of Y values. relative to our predicted Y for a given value of X, the observed values of Y will sometimes be higher and sometimes be lower. these “errors” are random – over the long term, they will cancel each other out but even if we knew β0 and β1, this source of error would still exist.

Two problems In other words We don’t have population values for the slope and the intercept of the line relating X to Y. That’s one problem. Even if we had population values for the slope and the intercept, the equation relating X to Y would still not perfectly predict Y. That’s the other problem.

How can we tell if our regression line is useful? The line is useful if the predicted values of Y are close to the observed values of Y (in the sample). We use our sample X and Y values to compute the regression line, Y = β0 + β1X. We then use this line to predict the same Y values, and compare our predicted values with the observed values in the sample data. If the prediction is good, we can then use the regression line to predict Y for values of X not in our sample. ^ ^

How can we tell if our regression line is useful? ^ ^ ^ ^ ^ ^ (Yi – Yi) = Yi – (β0 + β1Xi) (since Yi = β0 + β1Xi) Therefore, the sum of the squared deviations of predicted Y values from actual Y values is: SSE = Σ[Yi – (β0 + β1Xi)]2 Now β0 and β1 are the “least squares estimators” of β0 + β1 – giving smaller SSE than any other values of β0 and β1 would. ^ ^ Note that SSE is the sum of squared deviations – squared so that the sum is not zero. ^ ^ ^ ^

Y When there is no relation between X and Y, the best estimator of the Y value for any case is the mean, Y. Notice that the slope of this line is zero! X

How can we tell if our regression line is useful? If X is completely unrelated to Y, the best estimate we could make of Y would be the mean, Y, for any value of X. We find out whether our regression line is useful by asking whether its slope is different from 0. H0: β1 = 0 ^ [Why not β1?]

How can we tell if our regression line is useful? ^ To test that null hypothesis, we use the fact that β1 is one slope taken from the sampling distribution of β1. β1 = SSXY β0 = Y - β1X SSXX Where SSXY = Σ(Xi – X) (Yi –Y) = ΣXiYi – ΣXi ΣYi n ^ ^ ^ ^ Note: If we had drawn a different sample, we would have gotten a different β1-hat from that sampling distribution

How can we tell if our regression line is useful? SSXX = Σ(Xi – X)2 = ΣX2 – (ΣX)2 n (n = sample size) For the sampling distribution of β1: The mean = β1 β1 =  √SSXX ^ ^

How can we tell if our regression line is useful? We estimate β1 by sβ1 = s √SSXX Where s = SSE n-2 ^ ^ √

Test of hypothesis about the slope, β1 Since  is unknown, we use t to test H0: H0: β1 = 0 H0: β1 = 0 HA: β1 < 0 HA: β1 ≠ 0 or β1 > 0 Test statistic: t = β1 – 0 Sβ1 ^ ^

Test of hypothesis about the slope, β1 Rejection region: tobt < t │tobt│ > t/2 tobt > t tcrit is based on n-2 degrees of freedom.

Correlation The Pearson Correlation coefficient r is a numerical, descriptive measure of the strength and direction of relationship between two variables X and Y. r = SSXY SSXXSSYY r gives much the same information as β1. However r is “scale-less” and (-1 ≤ r ≤1) √ Those two qualities of r are not true of β1-with-a-hat ^

Useful features of r r indexes the X-Y relationship: r > 0 means Y increases as X increases r < 0 means Y decreases as X increases r = 0 means there is no relationship between X & Y r is the sample correlation coefficient. We can use it to estimate rho (ρ), the population correlation coefficient, and use r to test H0: ρ = 0 This test of hypothesis about ρ is equivalent to a test of H0: β1 = 0.

Test of hypothesis about ρ H0: ρ = 0 H0: ρ = 0 HA: ρ < 0 HA: ρ ≠ 0 or ρ > 0 Test statistic: t = r – ρ 1 – r2 n – 2 tcrit has n-2 degrees of freedom. √ Note: this test is strongly equivalent to the test of the hypothesis that the slope of the regression line, β1 = 0.

√ Example 1 H0: ρ = 0 HA: ρ ≠ 0 Test statistic: t = r – ρ 1 – r2 n – 2 tcrit = t(5, α/2 = .025) = 2.571. √

Example 1 – Sum formulas First, calculations involving X: ΣX = 74 (ΣX)2 = 5476 ΣX2 = 922 Then, analogous calculations involving Y: ΣY = 82 (ΣY)2 = 6724 ΣY2 = 1076 Then, calculations involving X and Y: ΣXY = 976

Example 1 – Sums of squares formulas SSXY = Σ(Xi – X) (Yi –Y) = ΣXiYi – ΣXi ΣYi n SSXX = Σ(Xi – X)2 = ΣX2 – (ΣX)2 SSYY = Σ(Yi – Y)2 = ΣY2 – (ΣY)2

√ Example 1 – calculate r SSXY = 109.143 SSXX = 139.71 SSYY = 115.429 r = SSXY r = .859 SSXXSSYY √

√ √ Example 1 – do t-test t = r – ρ 1 – r2 n – 2 1 - .738 .229 5 Reject H0: A significant correlation exists. √ √

√ Example 2 H0: ρ = 0 HA: ρ > 0 Test statistic: t = r – ρ 1 – r2 Note – these are the Greek letter rho, NOT the English letter P H0: ρ = 0 HA: ρ > 0 Test statistic: t = r – ρ 1 – r2 n – 2 tcrit = t(7-2 = 5, α = .05) = 2.015 √

Example 2 – Sum formulas First, calculations involving X: ΣX = 4.2 (ΣX)2 = 17.64 ΣX2 = 2.86 Then, analogous calculations involving Y: ΣY = 32 (ΣY)2 = 1024 ΣY2 = 161.5 Then, calculations involving X and Y: ΣXY = 21.35

Example 2 – calculate r SSXY = 21.35 – (4.2)(32) = 2.15 7 SSXX = 2.86 – 17.64 = .34

√ Example 2 – calculate r SSYY = 161.5 – 1024 = 15.2143 7 r = SSXY SSXXSSYY r = .945 √

√ √ Example 2 – do t-test t = r – ρ 1 – r2 n – 2 1 - .893 .146 5 Reject H0: A significant correlation exists. √ √