BIOL 582 Lecture Set 19 Matrices, Matrix calculations, Linear models using linear algebra

Compact method of expressing mathematical operations (including statistics) Makes linear models easier to compute BIOL 582Matrix operations Scalar: a number Vector: an ordered list (array) of scalars (n rows x 1 cols ) Matrix: a rectangular array of scalars (n rows x p cols ) Nomenclature: elements (or variables) are italicized, matrices are bold. Lowercase = vector; Capital = matrix Many variants in scientific literature

Reverse rows and columns Represent by A t or A′ Vector transpose works identically BIOL 582Matrix operations: transpose

Matrices must have same dimensions Add/subtract element-wise Vector addition/subtraction works identically Addition Subtraction BIOL 582Matrix operations: addition and subtraction

inner Scalar multiplication: Multiply scalar by each element in matrix or vector Matrix/vector multiplication is a summed multiplication Inner dimensions allow multiplication Outer dimensions determine size of result Order of matrices makes a difference: AB ≠ BA AB n 1 × p 1 * n 2 × p 2 BIOL 582Matrix operations: multiplication Inner dimension must agree or multiplication cannot take place

Scalar multiplication: Matrix multiplication: BIOL 582Matrix operations

Inner (scalar) product: vector multiplication resulting in a scalar (weighted linear combination) Outer (matrix) product: vector multiplication resulting in a matrix Inner Product Outer Product Inner dimensions MUST AGREE!!! BIOL 582Matrix operations

BIOL 582Special matrices I: Identity matrix (equivalent to ‘1’ for matrices) 1: A matrix of all ones 0: A matrix of all zeros Diagonal: diagonal contains non-zero elements Square: n = p Symmetric: off-diagonal elements same:

Orthogonal: square matrix with property: VERY useful for statistics and other fields (e.g, morphometrics) Orthonormal Example: BIOL 582Special Matrices

Cannot divide matrices, so calculate the inverse (reciprocal) of denominator and multiply Inverses have property that: Inverses are tedious to calculate, so in practice we use a computer Only works for square matrices whose determinant ≠ 0 (singular) Determinant: combination of diagonal and off-diagonal elements BIOL 582Matrix operations: division

For the 2 x 2 case: Example: Confirm: BIOL 582Matrix operations: invserse

The linear equation Can be written in matrix form as where BIOL 582Linear Model using matrix operations

Why is it so simple? Consider just this part for a simple example of four subjects and two independent variables: BIOL 582Linear Model using matrix operations

The linear model is: The estimated coefficients (parameter estimates) are solved as: How/why? Try to solve for Cannot divide both sides by X Cannot multiply by inverse of X, unless X is square- symmetric BIOL 582Linear Model using matrix operations

Making X symmetric: This matrix can be inverted: So, multiplying both sides of by will assist inverting the necessary part Note the dimensions so far: (k x n)(n x 1) = (k x n)(n x k)(k x 1)  (k x 1) = (k x 1) Now multiply both sides by inverse above Which has dimensions: (k x n)(n x k) (k x 1) = (k x n)(n x k) (k x 1)  (k x 1) = (k x 1) BIOL 582Linear Model using matrix operations

The equation Simplifies to And the dimensions of each side remain (k x 1) One problem is that the predicted values of the response are unknown without knowing the parameter estimates. However, the best estimates of the response values are the values themselves, so the equation is written as What this means is that one does not have to calculate SS for x and y and solve each coefficient independently! BIOL 582Linear Model using matrix operations

Done for a simple linear model of head size as a function of log SVL BIOL 582Example in R using Snake data > snake<-read.csv("snake.data.csv") > attach(snake) > # number of responses > n<-length(HS) > X<-matrix(c(rep(1,n),log(SVL)), nrow=n, ncol=2) > X[1:10,] [,1] [,2] [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,] > dim(X) [1] 40 2

Done for a simple linear model of head size as a function of log SVL BIOL 582Example in R using Snake data > y<-matrix(HS, nrow=n, ncol=1) > y[1:10,];dim(y) [,1] [1,] [2,] [3,] 7.16 [4,] [5,] [6,] 8.25 [7,] 9.74 [8,] [9,] [10,] [1] 40 1

Done for a simple linear model of head size as a function of log SVL BIOL 582Example in R using Snake data > B<-solve(t(X)%*%X)%*%t(X)%*%y > B [,1] [1,] [2,] > > # compare to canned function > lm.snake<-lm(HS~log(SVL),x=T) > lm.snake Call: lm(formula = HS ~ log(SVL), x = T) Coefficients: (Intercept) log(SVL)

Done for a simple linear model of head size as a function of log SVL BIOL 582Example in R using Snake data > # Predictions (fitted values) > y.hat<-X%*%B > y.hat[1:7,] [1] > > # Residuals > e<-y-y.hat > e[1:7,] [1] > > # Compare to > predict(lm.snake)[1:7] > resid(lm.snake)[1:7] >

After solving How does one determine if any or all coefficients are significant? Do the same thing for a reduced model and compare SSE First, how does one find SSE? First: Then Thus BIOL 582Analysis of variance using matrix operations

How is ? Using the snake example… BIOL 582Analysis of variance using matrix operations > SSE.f<-t(e)%*%e > SSE.f [,1] [1,]

ANOVA step by step for the snake data BIOL 582Analysis of variance using matrix operations > # ANOVA by hand, with matrix operations > > X.f<-matrix(c(rep(1,n),log(SVL)),nrow=n,ncol=2) > X.r<-matrix(rep(1,n),nrow=n,ncol=1) > y<-matrix(HS, nrow=n, ncol=1) > B.f<-solve(t(X.f)%*%X.f)%*%t(X.f)%*%y > B.r<-solve(t(X.r)%*%X.r)%*%t(X.r)%*%y > e.f<-y-X.f%*%B.f > e.r<-y-X.r%*%B.r > SSE.f<-t(e.f)%*%e.f > SSE.r<-t(e.r)%*%e.r > > SSE.f [,1] [1,] > SSE.r [,1] [1,] > k.f<-ncol(X.f);k.r<-ncol(X.r) > F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f)) > F.snake [,1] [1,] > P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f)) > P.value [,1] [1,] e-08 > R2<-(SSE.r-SSE.f)/(SSE.r) # only because X.r includes only an intercept > R2 [,1] [1,]

ANOVA for the snake data, this time relying on lm functions BIOL 582Analysis of variance using matrix operations > # ANOVA first using lm, then matrix operations > > lm.f<-lm(HS~log(SVL),x=T) > lm.r<-lm(HS~1,x=T) > e.f<-resid(lm.f) > e.r<-resid(lm.r) > SSE.f<-t(e.f)%*%e.f > SSE.r<-t(e.r)%*%e.r > > SSE.f [,1] [1,] > SSE.r [,1] [1,] > > k.f<-ncol(X.f);k.r<-ncol(X.r) > F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f)) > F.snake [,1] [1,] > P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f)) > P.value [,1] [1,] e-08 > > R2<-(SSE.r-SSE.f)/(SSE.r) > R2 [,1] [1,]

ANOVA for the snake data, what R does should be clear now BIOL 582Analysis of variance using matrix operations > # ANOVA via model comparison method > > lm.f<-lm(HS~log(SVL),x=T) > lm.r<-lm(HS~1,x=T) > > anova(lm.r,lm.f) Analysis of Variance Table Model 1: HS ~ 1 Model 2: HS ~ log(SVL) Res.Df RSS Df Sum of Sq F Pr(>F) e-08 *** --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 >

ANOVA for the snake data, what R does should be clear now BIOL 582Analysis of variance using matrix operations > # or just a model summary > summary(lm.f) Call: lm(formula = HS ~ log(SVL), x = T) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) *** log(SVL) e-08 *** --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 38 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: on 1 and 38 DF, p-value: 4.488e-08

It is worth looking at design matrices… BIOL 582Analysis of variance using matrix operations > X.f [,1] [,2] [1,] [2,] [3,] [4,] [5,] [6,] [7,] [8,] [9,] [10,] [11,] [12,] [13,] [14,] [15,] [16,] [17,] [18,] [19,] [20,] [21,] [22,] [23,] [24,] [25,] [26,] [27,] [28,] [29,] [30,] > lm.f$x (Intercept) log(SVL) > X.r [,1] [1,] 1 [2,] 1 [3,] 1 [4,] 1 [5,] 1 [6,] 1 [7,] 1 [8,] 1 [9,] 1 [10,] 1 [11,] 1 [12,] 1 [13,] 1 [14,] 1 [15,] 1 [16,] 1 [17,] 1 [18,] 1 [19,] 1 [20,] 1 [21,] 1 [22,] 1 [23,] 1 [24,] 1 [25,] 1 [26,] 1 [27,] 1 [28,] 1 [29,] 1 [30,] 1 > lm.r$x (Intercept)

Now for an example for a single factor ANOVA BIOL 582Analysis of variance using matrix operations > # Single factor Anova example, relying more so on lm commands > lm.f<-lm(HS~Sex,x=T) > lm.f$x (Intercept) SexM > lm.f<-lm(HS~Sex,x=T) > X.f<-lm.f$x > lm.r<-lm(HS~1,x=T) > X.r<-lm.r$x > y<-HS > B.f<-solve(t(X.f)%*%X.f)%*%t(X.f)%*%y > B.r<-solve(t(X.r)%*%X.r)%*%t(X.r)%*%y > e.f<-y-X.f%*%B.f > e.r<-y-X.r%*%B.r > SSE.f<-t(e.f)%*%e.f > SSE.r<-t(e.r)%*%e.r > > SSE.f [,1] [1,] > SSE.r [,1] [1,] > k.f<-ncol(X.f);k.r<-ncol(X.r) > F.snake<-((SSE.r-SSE.f)/(k.f-k.r))/(SSE.f/(n-k.f)) > F.snake [,1] [1,] > P.value<-1-pf(F.snake,(k.f-k.r),(n-k.f)) > P.value [,1] [1,] > > R2<-(SSE.r-SSE.f)/(SSE.r) > R2 [,1] [1,]

Now for an example for a single factor ANOVA BIOL 582Analysis of variance using matrix operations > summary(lm.f) Call: lm(formula = HS ~ Sex, x = T) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) e-13 *** SexM Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 38 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: on 1 and 38 DF, p-value: > B.f [,1] (Intercept) SexM >