2. Lectures prepared by: Elchanan Mossel Yelena Shvets Introduction to probability Stat 134 FAll 2005 Berkeley Follows Jim Pitman’s book: Probability Sections 6.4

3. Do taller people make more money? Question: How can this be measured? wage at 19 height at 16 National Longitudinal Survey of Youth 1997 (NLSY97) - Ave (wage) Ave (height)

4. Definition of Covariance Cov (X,Y)= E[(X-  X )(Y –  Y )] Alternative Formula Cov (X,Y)= E(XY) – E(X)E(Y) Variance of a Sum Var (X+Y)= Var (X) + Var (Y)+2 Cov (X,Y) Claim: Covariance is Bilinear

5. What does the sign of covariance mean? Look at Y = aX + b. Then: Cov(X,Y) = Cov(X,aX + b) = aVar(X). If a > 0, above the average in X goes with above the ave in Y. If a < 0, above the average n X goes with below the ave in Y. Cov(X,Y) = 0 means that there is no linear trend which connects X and Y. x y x y a>0 a<0 Ave(Y) Ave(X)

6. Meaning of the value of Covariance Back to the National Survey of Youth study : the actual covariance was 3028 where height is inches and the wages in dollars. Question: Suppose we measured all the heights in centimeters, instead. There are 2.54 cm/inch? Question: What will happen to the covariance? Solution: So let H I be height in inches and H C be the height in centimeters, with W – the wages. Cov(H C,W) = Cov(2.54 H I,W) = 2.54 Cov (H I,W). So the value depends on the units and is not very informative!

7. Covariance and Correlation Define the correlation coefficient: Using the linearity of Expectation we get: Notice that  (aX+b, cY+d) =  (X,Y). This new quantity is independent of the change in scale and it’s value is quite informative.

8. Covariance and Correlation Properties of correlation:

9. Covariance and Correlation Claim: The correlation is always between –1 and +1  = 1 iff Y = aX + b.

10. Correlation and Independence X & Y are uncorrelated iff any of the following hold Cov(X,Y) = 0, Corr(X,Y) = 0 E(XY) = E(X) E(Y). In particular, if X and Y are independent they are uncorrelated. Example: Let X » N(0,1) and Y = X 2, then Cov(XY) =E(XY) – E(X)E(Y) = E(X 3 ) = 0, since the density is symmetric. X X2X2

11. Roll a die N times. Let X be #1’s, Y be #2’s. Question: What is the correlation between X and Y? Solution: To compute the correlation directly from the multinomial distribution would be difficult. Let’s use a trick: Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y). Since X+Y is just the number of 1’s or 2’s, X+Y » Binom(p 1 +p 2,N). Var(X+Y) = (p 1 +p 2 )(1 - p 1 +p 2 ) N. And X » Binom(p 1,N), Y » Binom(p 2,N), so Var(X) =p 1 (1-p 1 )N; Var(Y) = p 2 (1-p 2 )N.

12. Correlations in the Multinomial Distribution Hence Cov(X,Y) = (Var(X+Y) – Var(X) – Var(Y))/2 Cov(X,Y) = N((p 1 +p 2 )(1 - p 1 -p 2 ) - p 1 (1-p 1 ) -p 2 (1-p 2 ))/2 = -N p 1 p 2 In our case p 1 = p 2 = 1/6, so  = 1/5. The formula holds for a general multinomial distribution. -

13. Variance of the Sum of N Variables Var(  i X i ) =  i Var(X i ) + 2  j<i Cov(X i X j ) Proof: Var(  i X i ) = E[  i X i – E(  j X i ) ] 2 [  i X i – E(  j X i ) ] 2 = [  i (X i –  i ) ] 2 =  i (X i –  i ) 2 + 2  j<i (X i –  i ) (X j –  j ). Now take expectations and we have the result.

14. Variance of the Sample Average Let the population be a list of N numbers x(1), …, x(N). Then are the population mean and population variance. Let X 1, X 2,…, X n be a sample of size n drawn from this population. Then each X k has the same distribution as the entire population and Let be the sample average. x = (  i=1 N x(i))/N -  2 = (  i=1 N (x(i)-x) 2 )/N -

15. Variance of the Sample Average When X 1, X 2,…, X n are drawn with replacement, they are independent and each X k has variance  2. Then By linearity of expectation, both for a sample drawn with and without replacement.

16. Variance of the Sample Average In particular, 0 =Var(S N ) N  2 + N(n-N) Cov(X 1,X 2 ) Therefore: Cov(X 1 X 2 ) = -  2 /(N-1). And hence Var(S n ) =  2 n(1- (n-1)/(N-1)). Question: What is the SD for sampling without replacement? Solution: Let S n = X 1 + X 2 + … + X n. Then. Var(S n ) =  i Var(X i ) + 2  j<i Cov(X i X j ) By symmetry: Cov(X i,X j ) = Cov(X 1,X 2 ), so Var(S n ) = n  2 + n(n-1) Cov(X 1 X 2 ).