1. Chapter 5: Regression1 Chapter 5 Relationships: Regression

2. 2 Objectives (BPS chapter 5) Regression u Regression lines u The least-squares regression line u Facts about least-squares regression u Residuals u Influential observations u Cautions about correlation and regression u Association does not imply causation

3. 3 Correlation tells us about strength (scatter) and direction of the linear relationship between two quantitative variables. We would like to have a numerical description of how the variables vary together. We would also like to make predictions based on the observed association between those two variables. We wish to find the straight line that best fits our data. But which line best describes our data?

4. 4 A regression line A regression line is a straight line that describes how a response variable (y) changes as an explanatory variable (x) changes. We often use a regression line to predict the value of y for a given value of x.

5. 5Chapter 5: Regression5 Linear Regression u We wish to quantify the linear relationship between an explanatory variable and a response variable. We can then predict the average response for all subjects with a given value of the explanatory variable. u Regression equation: y = a + bx –x is the value of the explanatory variable –y is the average value of the response variable –note that a and b are just the y-intercept and slope of a straight line

6. 6Chapter 5: Regression6 Thought Question 1 How would you draw a line through the points? How do you determine which line ‘fits best’?

7. 7Chapter 5: Regression7 Linear Equations High School Teacher

8. 8Chapter 5: Regression8 The Linear Model u Remember from Algebra that a straight line can be written as: u In Statistics we use a slightly different notation: u We write to emphasize that the points that satisfy this equation are just our predicted values, not the actual data values. = a + bx

9. 9Chapter 5: Regression9 Example: Fat Versus Protein u The following is a scatterplot of total fat versus protein for 30 items on the Burger King menu: We wish to fit a straight line through the data.

10. 10Chapter 5: Regression10 Residuals u The model won’t be perfect, regardless of the line we draw. u Some points will be above the line and some will be below. u The estimate made from a model is the predicted value (denoted as ).

11. 11Chapter 5: Regression11 Residuals (cont.) u The difference between the observed value and its associated predicted value is called the residual. u To find the residuals, we always subtract the predicted value from the observed one:

12. 12Chapter 5: Regression12 Residuals (cont.) u A negative residual means the predicted value is too big (an overestimate). u A positive residual means the predicted value is too small (an underestimate).

13. 13Chapter 5: Regression13 “Best Fit” Means Least Squares u Some residuals are positive, others are negative, and, on average, they cancel each other out. u So, we can’t assess how well the line fits by adding up all the residuals. u Similar to what we did with the standard deviation, we square the residuals and add the squares. u The smaller the sum, the better the fit. u The line of best fit is the line for which the sum of the squared residuals is smallest.

14. 14Chapter 5: Regression14 Least Squares u Used to determine the “best” line u We want the line to be as close as possible to the data points in the vertical (y) direction (since that is what we are trying to predict) u Least Squares: use the line that minimizes the sum of the squares of the vertical distances of the data points from the line

15. 15Chapter 5: Regression15 The Linear Model (cont.)  We write b and a for the slope and intercept of the line. The b and a are called the coefficients of the linear model.  The coefficient b is the slope, which tells us how rapidly changes with respect to x. The coefficient a is the intercept, which tells where the line hits (intercepts) the y -axis.

16. 16 First we calculate the slope of the line, b. We already know how to calculate r, s x and s y. r is the correlation (the slope has the same sign as r) s y is the standard deviation of the response variable y s x is the the standard deviation of the explanatory variable x Once we know b, the slope, we can calculate a, the y-intercept: where x and y are the sample means of the x and y variables How to: This means that we don’t have to calculate a lot of squared distances to find the least- squares regression line for a data set. We can instead rely on the equation. Some calculators can calculate r, a and b.

17. 17Chapter 5: Regression17 Example Fill in the missing information in the table below:

18. 18 Facts about least-squares regression 1. The distinction between explanatory and response variables is essential in regression. 2. The correlation coefficient (r) and the slope (b) of the least-squares line have the same sign. The direction of the association determines the sign of the slope of the regression line. 3. The least-squares regression line always passes through the point. 4. The correlation r describes the strength of a straight-line relationship. The square of the correlation, r 2, is the fraction of the variation in the values of y that is explained by the least-squares regression of y on x. The square of the correlation is called Coefficient of Determination.

19. 19 The distinction between explanatory and response variables is crucial in regression. If you exchange y for x in calculating the regression line, you will get the wrong line. Regression examines the distance of all points from the line in the y direction only. Data from the Hubble telescope about galaxies moving away from Earth: These two lines are the two regression lines calculated either correctly (x = distance, y = velocity, solid line) or incorrectly (x = velocity, y = distance, dotted line).

20. 20Chapter 5: Regression20 Interpretation of the Slope and Intercept u The slope indicates the amount by which changes when x changes by one unit.  The intercept is the value of when x = 0. u It is not always meaningful.

21. 21Chapter 5: Regression21 Example The regression line for the Burger King data is Interpret the slope and the intercept. Slope: For every one gram increase in protein, the fat content increases by 0.97g. Intercept: A BK meal that has 0g of protein contains 6.8g of fat.

22. 22Chapter 5: Regression22 In predicting a value of y based on some given value of x... 1. If there is no linear correlation, the best predicted y-value is y. Predictions 2.If there is a linear correlation, the best predicted y-value is found by substituting the x-value into the regression equation.

23. 23Chapter 5: Regression23 Example: Fat Versus Protein The regression line for the Burger King data fits the data well: –The equation is –The predicted fat content for a BK Broiler chicken sandwich that contains 30g of protein is 6.8 + 0.97(30) = 35.9 grams of fat.

24. 24Chapter 5: Regression24 Prediction via Regression Line Hand, et al., A Handbook of Small Data Sets, London: Chapman and Hall u The regression equation is y = 3.6 + 0.97x –y is the average age of all husbands who have wives of age x u Suppose we know that an individual wife’s age is 30. What would we predict her husband’s age to be? u For all women aged 30, we predict the average husband age to be 32.7 years: 3.6 + (0.97)(30) = 32.7 years Husband and Wife: Ages ^

25. 25 Extrapolation is the use of a regression line for predictions outside the range of x values used to obtain the line. This can be misleading, as seen here. Caution! Beware of Extrapolation

26. 26Chapter 5: Regression26 Caution ! Beware of Extrapolation u Sarah’s height was plotted against her age u Can you predict her height at age 42 months? u Can you predict her height at age 30 years (360 months)?

27. 27Chapter 5: Regression27 A Caution Beware of Extrapolation u Regression line: = 71.95 +.383 x u height at age 42 months? = 88 cm. u height at age 30 years? = 209.8 cm. –She is predicted to be 6' 10.5" at age 30.

28. 28Chapter 5: Regression28 Residuals Revisited u Residuals help us to see whether the model makes sense. u When a regression model is appropriate, nothing interesting should be left behind. u After we fit a regression model, we usually plot the residuals in the hope of finding no apparent pattern.

29. 29Chapter 5: Regression29 Residual Plot Analysis A residual plot is a scatterplot of the regression residuals against the explanatory variable. If a residual plot does not reveal any pattern, the regression equation is a good representation of the association between the two variables. If a residual plot reveals some systematic pattern, the regression equation is not a good representation of the association between the two variables.

30. 30Chapter 5: Regression30 Residuals Revisited (cont.) u The residuals for the BK menu regression look appropriately boring: Plot

31. 31 Residuals are randomly scattered—good! A curved pattern—means the relationship you are looking at is not linear. A change in variability across plot is a warning sign. You need to find out what it is and remember that predictions made in areas of larger variability will not be as good.

32. 32Chapter 5: Regression32 Coefficient of Determination (R 2 ) u Measures usefulness of regression prediction u R 2 (or r 2, the square of the correlation): measures the percentage of the variation in the values of the response variable (y) that is explained by the regression line v r=1: R 2 =1:regression line explains all (100%) of the variation in y v r=.7: R 2 =.49:regression line explains almost half (50%) of the variation in y

33. 33Chapter 5: Regression33 u Along with the slope and intercept for a regression, you should always report R 2 so that readers can judge for themselves how successful the regression is at fitting the data. u Statistics is about variation, and R 2 measures the success of the regression model in terms of the fraction of the variation of y accounted for by the regression. R 2 (cont)

34. 34 r = −1 r 2 = 1 Changes in x explain 100% of the variations in y. y can be entirely predicted for any given value of x. r = 0 r 2 = 0 Changes in x explain 0% of the variations in y. The value(s) y takes is (are) entirely independent of what value x takes. Here the change in x only explains 76% of the change in y. The rest of the change in y (the vertical scatter, shown as red arrows) must be explained by something other than x. r = 0.87 r 2 = 0.76

35. 35Chapter 5: Regression35 Caution with regression u Since regression and correlation are closely related, we need to check the same conditions for regression as we did for correlation: –Quantitative Variables Condition –Straight Enough Condition –Outlier Condition

36. 36 Caution with regression  Do not use a regression on inappropriate data. Pattern in the residuals Presence of large outliers Use residual plots for help. Clumped data falsely appearing linear  Recognize when the correlation/regression is performed on averages.  A relationship, however strong, does not imply causation.  Beware of lurking variables.  Avoid extrapolating (predicting outside the observed x data range).

37. 37Chapter 5: Regression37 1. If there is no linear correlation, don’t use the regression equation to make predictions. 2. When using the regression equation for predictions, stay within the scope of the available sample data. 3. A regression equation based on old data is not necessarily valid now. 4. Don’t make predictions about a population that is different from the population from which the sample data were drawn. Guidelines for Using The Regression Equation

38. 38Chapter 5: Regression38 Vocabulary  Marginal Change – refers to the slope; the amount the response variable changes when the explanatory variable changes by one unit.  Outlier - A point lying far away from the other data points.  Influential Point - An outlier that that has the potential to change the regression line. - Points that are outliers in either the x or y direction of a scatterplot are often influential for the correlation. - Points that outliers in the x direction are often influential for the least-squares regression line. Try

39. 39 All data Without child 18 Without child 19 Outlier in y-direction Influential Are these points influential?

40. 40 Vocabulary: Lurking vs. Confounding LURKING VARIABLE u A lurking variable is a variable that is not among the explanatory or response variables in a study and yet may influence the interpretation of relationships among those variables. CONFOUNDING u Two variables are confounded when their effects on a response variable cannot be distinguished from each other. The confounded variables may be either explanatory variables or lurking variables.

41. 41 Lurking variables Lurking variables can falsely suggest a relationship. What is the lurking variable in these examples? How could you answer if you didn’t know anything about the topic?  Strong positive association between the number firefighters at a fire site and the amount of damage a fire does  Negative association between moderate amounts of wine drinking and death rates from heart disease in developed nations

42. 42Chapter 5: Regression42 Correlation Does Not Imply Causation Even very strong correlations may not correspond to a real causal relationship.

43. 43Chapter 5: Regression43 Evidence of Causation u A properly conducted experiment establishes the connection u Other considerations: –A reasonable explanation for a cause and effect exists –The connection happens in repeated trials –The connection happens under varying conditions –Potential confounding factors are ruled out –Alleged cause precedes the effect in time

44. 44Chapter 5: Regression44 Evidence of Causation u An observed relationship can be used for prediction without worrying about causation as long as the patterns found in past data continue to hold true. u We must make sure that the prediction makes sense. u We must be very careful of extreme extrapolation.

45. 45Chapter 5: Regression45 Reasons Two Variables May Be Related (Correlated) u Explanatory variable causes change in response variable u Response variable causes change in explanatory variable u Explanatory may have some cause, but is not the sole cause of changes in the response variable u Confounding variables may exist u Both variables may result from a common cause –such as, both variables changing over time u The correlation may be merely a coincidence

46. 46Chapter 5: Regression46 Common Response (both variables change due to common cause) u Both may result from an unhappy marriage. u Explanatory: Divorce among men u Response: Percent abusing alcohol

47. 47Chapter 5: Regression47 Both Variables are Changing Over Time u Both divorces and suicides have increased dramatically since 1900. u Are divorces causing suicides? u Are suicides causing divorces??? u The population has increased dramatically since 1900 (causing both to increase). u Better to investigate: Has the rate of divorce or the rate of suicide changed over time?

48. 48Chapter 5: Regression48 The Relationship May Be Just a Coincidence Sometimes we see some strong correlations (or apparent associations) just by chance, even when the variables are not related in the population.

49. 49Chapter 5: Regression49 u A required whooping cough vaccine was blamed for seizures that caused brain damage –led to reduced production of vaccine (due to lawsuits) u Study of 38,000 children found no evidence for the accusations (reported in New York Times) –“people confused association with cause-and-effect” –“virtually every kid received the vaccine…it was inevitable that, by chance, brain damage caused by other factors would occasionally occur in a recently vaccinated child” Coincidence (?) Vaccines and Brain Damage

50. 50 Quiz 1. The least-squares regression line is A) the line that best splits the data in half, with half of the points above the line and half below the line. B) the line that makes the square of the correlation in the data as large as possible. C) the line that makes the sum of the squares of the vertical distances of the data points from the line as small as possible. D) all of the above.

51. 51 Quiz 2. The fraction of the variation in the values of a response y that is explained by the least-squares regression of y on x is A) the intercept of the least-squares regression line B) the correlation coefficient. C) the slope of the least-squares regression line. D) the square of the correlation coefficient.

52. 52 Quiz 3. A researcher obtained the average SAT scores of all students in each of the 50 states, and the average teacher salaries in each of the 50 states of the US. He found a negative correlation between these variables. The researcher concluded that a lurking variable must be present. By lurking variable he means A) the true cause of a response. B) the true variable, which is explained by the explanatory variable. C) a variable that is not among the variables studied but which affects the response variable. D) any variable that produces a large residual..

53. 53 1.Find the least-squares regression line of John’s height on age based on the data. 2.Interpret the slope of the regression line. 3.If John’s height at 54 months has a residual of 1.2 in, what was his actual height? 4.What percentage of the variation in John’s height is explained by the regression model? Example:

54. 54Chapter 5: Regression54 Key Concepts u Least Squares Regression Equation u Interpretation of the slope and intercept u Prediction – avoid extrapolations u Residual Plot Analysis uR2uR2 u Correlation does not imply causation u Reasons variables may be correlated