7.1 - Motivation Motivation Correlation / Simple Linear Regression Correlation / Simple Linear Regression Extensions of Simple Linear Regression Extensions of Simple Linear Regression CHAPTER 7 Linear Correlation & Regression Methods CHAPTER 7 Linear Correlation & Regression Methods

Parameter Estimation via SAMPLE DATA … Testing for association between two POPULATION variables X and Y… Categorical variables Numerical variables Chi-squared Test  Chi-squared Test ???????  ??????? Examples: X = Disease status (D+, D–) Y = Exposure status (E+, E–) X = # children in household (0, 1-2, 3-4, 5+) Y = Income level (Low, Middle, High) PARAMETERS  Means:  Variances:  Covariance:

Parameter Estimation via SAMPLE DATA … PARAMETERS  Means:  Variances: Numerical variables ???????  ???????  Means:  Variances: PARAMETERS STATISTICS (can be +, –, or 0)  Covariance:

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn PARAMETERS  Means:  Variances:  Covariance: Numerical variables ???????  ???????  Means:  Variances: PARAMETERS STATISTICS (can be +, –, or 0) X Y JAMA. 2003;290: Scatterplot ( n data points)  Covariance:

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn PARAMETERS  Means:  Variances:  Covariance: Numerical variables ???????  ???????  Means:  Variances: PARAMETERS STATISTICS (can be +, –, or 0) X Y JAMA. 2003;290: Scatterplot Does this suggest a linear trend between X and Y? If so, how do we measure it? ( n data points)  Covariance:

Testing for association between two population variables X and Y… Numerical variables ???????  ??????? PARAMETERS  Means:  Variances:  Covariance:  Linear Correlation Coefficient: Always between –1 and +1 LINEAR ^

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn PARAMETERS  Means:  Variances:  Covariance: Numerical variables ???????  ???????  Means:  Variances:  Covariance: PARAMETERS STATISTICS (can be +, –, or 0) X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 ( n data points)

Parameter Estimation via SAMPLE DATA … JAMA. 2003;290: x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn PARAMETERS  Means:  Variances:  Covariance: Numerical variables ???????  ???????  Means:  Variances:  Covariance: PARAMETERS STATISTICS (can be +, –, or 0) X Y Scatterplot ( n data points) Example in R (reformatted for brevity): > c(mean(x), mean(y)) > var(x) > var(y) > cov(x, y)  Linear Correlation Coefficient: Always between –1 and +1 > cor(x, y) > pop = seq(0, 20, 0.1) > x = sort(sample(pop, 10)) > y = sample(pop, 10) n = 10 plot(x, y, pch = 19)

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Numerical variables X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 r measures the strength of linear association ( n data points)

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Numerical variables X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 – positive linear correlation negative linear correlation r r measures the strength of linear association ( n data points)

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Numerical variables X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 – positive linear correlation negative linear correlation r r measures the strength of linear association ( n data points)

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Numerical variables X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 – positive linear correlation negative linear correlation r r measures the strength of linear association ( n data points) linear r measures the strength of linear association

Parameter Estimation via SAMPLE DATA … x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Numerical variables X Y JAMA. 2003;290: Scatterplot  Linear Correlation Coefficient: Always between –1 and +1 – positive linear correlation negative linear correlation r ( n data points) > cor(x, y) > cor(x, y) linear r measures the strength of linear association

linear Testing for linear association between two numerical population variables X and Y…  Linear Correlation Coefficient Now that we have r, we can conduct HYPOTHESIS TESTING on  Test Statistic for p-value p-value =.0189 <.05 2 * pt(-2.935, 8)

Parameter Estimation via SAMPLE DATA … If such an association between X and Y exists, then it follows that for any intercept  0 and slope  1, we have…  Linear Correlation Coefficient: linear r measures the strength of linear association “Response = Model + Error” Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals in what sense???

Parameter Estimation via SAMPLE DATA … in what sense??? If such an association between X and Y exists, then it follows that for any intercept  0 and slope  1, we have…  Linear Correlation Coefficient: linear r measures the strength of linear association “Response = Model + Error” Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes “Least Squares Regression Line” SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES

If such an association between X and Y exists, then it follows that for any intercept  0 and slope  1, we have…  Linear Correlation Coefficient: linear r measures the strength of linear association “Response = Model + Error” Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES Check 

Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes X Y SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response

Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes X Y SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response

Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes X Y ~EXERCISE~ SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response

Find estimates and for the “best” line > cor(x, y) > cor(x, y) Residuals i.e., that minimizes SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response residuals X Y ~EXERCISE~

Find estimates and for the “best” line > cor(x, y) > cor(x, y) i.e., that minimizes SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response residuals X Y ~EXERCISE~ ~EXERCISE~ Residuals

linear Testing for linear association between two numerical population variables X and Y…  Linear Regression Coefficients Test Statistic for p-value? “Response = Model + Error” Now that we have these, we can conduct HYPOTHESIS TESTING on  0 and  1

Find estimates and for the “best” line > cor(x, y) > cor(x, y) i.e., that minimizes SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response residuals X Y ~EXERCISE~ ~EXERCISE~ Residuals

linear Testing for linear association between two numerical population variables X and Y…  Linear Regression Coefficients Now that we have these, we can conduct HYPOTHESIS TESTING on  0 and  1 Test Statistic for p-value “Response = Model + Error” Same t-score as H 0 :  = 0! p-value =.0189

> plot(x, y, pch = 19) > lsreg = lm(y ~ x) # or lsfit(x,y) > abline(lsreg) > summary(lsreg) Call: lm(formula = y ~ x) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) *** x * --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 8 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 8 DF, p-value: BUT WHY HAVE TWO METHODS FOR THE SAME PROBLEM??? Because this second method generalizes…

SourcedfSSMSF-ratiop-value Treatment Error Total – ANOVA Table

SourcedfSSMSF-ratiop-value Regression Error Total – ANOVA Table

SourcedfSSMSF-ratiop-value Regression 1 Error Total – ANOVA Table

linear Testing for linear association between two numerical population variables X and Y…  Linear Regression Coefficients Now that we have these, we can conduct HYPOTHESIS TESTING on  0 and  1 Test Statistic for p-value “Response = Model + Error” Same t-score as H 0 :  = 0! p-value =.0189

SourcedfSSMSF-ratiop-value Regression 1 Error 8 Total – ANOVA Table

Parameter Estimation via SAMPLE DATA …  Means:  Variances: STATISTICS JAMA. 2003;290: x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Scatterplot ( n data points)

Parameter Estimation via SAMPLE DATA …  Means:  Variances: STATISTICS JAMA. 2003;290: x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Scatterplot ( n data points) SS Tot is a measure of the total amount of variability in the observed responses (i.e., before any model-fitting).

Parameter Estimation via SAMPLE DATA … JAMA. 2003;290: x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Scatterplot ( n data points)  Means:  Variances: STATISTICS SS Reg is a measure of the total amount of variability in the fitted responses (i.e., after model-fitting.)

Parameter Estimation via SAMPLE DATA …  Means:  Variances: STATISTICS JAMA. 2003;290: x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn Scatterplot ( n data points) SS Err is a measure of the total amount of variability in the resulting residuals (i.e., after model-fitting).

> cor(x, y) > cor(x, y) SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response residuals X Y ~EXERCISE~ ~EXERCISE~ Residuals = = = 9 ( ) = 204.2

SIMPLE LINEAR REGRESSION via the METHOD OF LEAST SQUARES predictor observed response fitted response residuals X Y ~EXERCISE~ ~EXERCISE~ Residuals = = = SS Tot = SS Reg + SS Err minimum > cor(x, y) > cor(x, y) Tot Err Reg

SourcedfSSMSF-ratiop-value Regression MS Reg F k – 1, n – k 0 < p < 1 Error MS Err Total – ANOVA Table

SourcedfSSMSF-ratiop-value Regression Error Total – ANOVA Table Same as before!

SourcedfSSMSF-ratiop-value Regression Error Total – > summary(aov(lsreg)) Df Sum Sq Mean Sq F value Pr(>F) x * Residuals

SourcedfSSMSF-ratiop-value Regression Error Total – Moreover, The least squares regression line accounts for 51.85% of the total variability in the observed response, with 48.15% remaining. Coefficient of Determination

> cor(x, y) > cor(x, y) Coefficient of Determination Moreover, The least squares regression line accounts for 51.85% of the total variability in the observed response, with 48.15% remaining.

> plot(x, y, pch = 19) > lsreg = lm(y ~ x) > abline(lsreg) > summary(lsreg) Call: lm(formula = y ~ x) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) *** x * --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 8 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 8 DF, p-value: Coefficient of Determination The least squares regression line accounts for 51.85% of the total variability in the observed response, with 48.15% remaining.

Given:  Linear Correlation Coefficient  Least Squares Regression Line minimizes SS Err = Summary of Linear Correlation and Simple Linear Regression x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn MeansVariancesCovariance JAMA. 2003;290: X Y XYXY = SS Tot – SS Reg –1  r  +1 linear measures the strength of linear association (ANOVA) All point estimates can be upgraded to CIs for hypothesis testing, etc.

Given:  Linear Correlation Coefficient –1  r  +1 linear measures the strength of linear association  Least Squares Regression Line minimizes SS Err = (ANOVA) Summary of Linear Correlation and Simple Linear Regression x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn MeansVariancesCovariance JAMA. 2003;290: X Y XYXY = SS Tot – SS Reg upper 95% confidence band 95% Confidence Intervals lower 95% confidence band All point estimates can be upgraded to CIs for hypothesis testing, etc. (see notes for “95% prediction intervals”)

Given:  Linear Correlation Coefficient –1  r  +1 linear measures the strength of linear association  Least Squares Regression Line minimizes SS Err = (ANOVA) Summary of Linear Correlation and Simple Linear Regression x1x1 x2x2 x3x3 x4x4 …xnxn y1y1 y2y2 y3y3 y4y4 …ynyn MeansVariancesCovariance JAMA. 2003;290: X Y XYXY  Coefficient of Determination = SS Tot – SS Reg All point estimates can be upgraded to CIs for hypothesis testing, etc. proportion of total variability modeled by the regression line’s variability.

linear Testing for linear association between a population response variable Y and multiple predictor variables X 1, X 2, X 3, … etc. “Response = Model + Error” Multilinear Regression “main effects” For now, assume the “additive model,” i.e., main effects only.

Multilinear Regression Fitted response  Residual True response y i X1X1 X2X2 0 Y (x 1i, x 2i ) Predictors Once calculated, how do we then test the null hypothesis? Least Squares calculation of regression coefficients is computer-intensive. Formulas require Linear Algebra (matrices)! ANOVA

linear Testing for linear association between a population response variable Y and multiple predictor variables X 1, X 2, X 3, … etc. “Response = Model + Error” Multilinear Regression “main effects” R code example: lsreg = lm(y ~ x1+x2+x3)

R code example: lsreg = lm(y ~ x+x^2+x^3) linear Testing for linear association between a population response variable Y and multiple predictor variables X 1, X 2, X 3, … etc. “Response = Model + Error” Multilinear Regression “main effects” quadratic terms, etc. (“polynomial regression”) quadratic terms, etc. (“polynomial regression”)

R code example: lsreg = lm(y ~ x+x^2+x^3) R code example: lsreg = lm(y ~ x1+x2+x1:x2) R code example: lsreg = lm(y ~ x1*x2) linear Testing for linear association between a population response variable Y and multiple predictor variables X 1, X 2, X 3, … etc. “Response = Model + Error” Multilinear Regression “main effects” quadratic terms, etc. (“polynomial regression”) quadratic terms, etc. (“polynomial regression”) “ interactions ”

Recall… Example in R (reformatted for brevity): > I = c(1,1,1,1,1,0,0,0,0,0) > lsreg = lm(y ~ x*I) > summary(lsreg) Coefficients: Estimate (Intercept) x I x:I Suppose these are actually two subgroups, requiring two distinct linear regressions! Multiple Linear Reg with interaction with an indicator (“dummy”) variable: I = 1 I = 0

ANOVA Table (revisited) From sample of n data points…. Note that if true, then it would follow that But how are these regression coefficients calculated in general? Normal equations “Normal equations” solved via computer (intensive). Note that if true, then it would follow that

SourcedfSS MS Fp-value Regression Error Total ANOVA Table (revisited) (based on n data points). *** How are only the statistically significant variables determined? ***

p-values: p 1 <.05 p 2 <.05 p 4 <.05 “MODEL SELECTION”(BE) Step 0.Conduct an overall F-test of significance (via ANOVA) of the full model. X1X …… X2X2 X3X3 X4X4 …… Step 1. t-tests: Reject H 0 Reject H 0 Accept H 0 Reject H 0 …… Step 2. Are all coefficients significant at level  ? If not…. If significant, then…

p-values: p 1 <.05 p 2 <.05 p 4 <.05 “MODEL SELECTION”(BE) Step 0.Conduct an overall F-test of significance (via ANOVA) of the full model. X1X …… X2X2 X3X3 X4X4 …… Step 1. t-tests: Reject H 0 Reject H 0 Accept H 0 Reject H 0 …… Step 2. Are all coefficients significant at level  ? If not…. X1X …… X2X2 X4X4 X3X3 delete that term, If significant, then…

p-values: p 1 <.05 p 2 <.05 p 4 <.05 “MODEL SELECTION”(BE) Step 0.Conduct an overall F-test of significance (via ANOVA) of the full model. X1X …… X2X2 X3X3 X4X4 …… Step 1. t-tests: Reject H 0 Reject H 0 Accept H 0 Reject H 0 …… Step 2. Are all coefficients significant at level  ? If not…. X1X …… X2X2 X4X4 Step 3. Repeat 1-2 as necessary until all coefficients are significant → reduced model delete that term, and recompute new coefficients! If significant, then… X1X1 X2X2 X4X4 +++ ……

11 22 kk = = = H0:H0: k  2 independent, equivariant, normally-distributed “treatment groups” Recall ~

Re-plot data on a “log-log” scale.

Re-plot data on a “log” scale (of Y only)..

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No)

“MAXIMUM LIKELIHOOD ESTIMATION” “log-odds” (“logit”) = example of a general “link function” (Note: Not based on LS implies “pseudo-R 2,” etc.)

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No) Suppose one of the predictor variables is binary… “log-odds” (“logit”) SUBTRACT !

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No) Suppose one of the predictor variables is binary… “log-odds” (“logit”) SUBTRACT !

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No) Suppose one of the predictor variables is binary… “log-odds” (“logit”)

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No) Suppose one of the predictor variables is binary… “log-odds” (“logit”)

Binary outcome, e.g., “Have you ever had surgery?” (Yes / No) Suppose one of the predictor variables is binary… “log-odds” (“logit”) ………….. implies …………..

in population dynamics Unrestricted population growth (e.g., bacteria) Population size y obeys the following law with constant a > 0. With initial condition Restricted population growth (disease, predation, starvation, etc.) Population size y obeys the following law, constant a > 0, and “carrying capacity” M. Exponential growth Let survival probability  = Logistic growth