Sums of Random Variables and Long-Term Averages Sums of R.V. ‘s S n = X 1 + X X n of course

Thus Var ( X + Y) = Var (X) + Var (Y) only if Cov ( X,Y) = 0 If X k are i.i.d ( Independent,Identically distributed) ~ ,  2 E (S n ) = n  Var (S n ) = n  2

PDF of sums of independent R.V.’s: because of independence

Sample Mean: If X k are iid measurements of a R.V. X, the sample mean M n is a random variable. M n is an unbiased estimator of .

Clearly In fact, this holds even when  2 does not exist. Weak Law of Large Numbers: If X k are i.i.d samples of X, E(X) = ,  is finite then, for all  > 0 Strong Law of Large Numbers: If X k are i.i.d samples of X, E(X) =  and Var(X) =  2 ,  2 finite limit of probability  1

The Weak law states that : for a large enough n, M n is close to  with high probability. The Strong law states that : with probability 1, a sequence of M n ‘s calculated using n samples converges to  as n   Note that the weak law does not say anything about any particular sequence of M n ‘s converging as a function of n. It only gives the probability of being close to  for any fixed n, and this probability approaches 1 as n  .

Central Limit Theorem Let X 1,X 2,..... Be a sequence of i.i.d R.V. ‘s with mean  and variance  2 If  and  2 exist. i.e. The sum of “enough” iid R.V.’s, properly normalized, is approximately normally distributed.

More general version: If X k are independent ( not necessarily identically distributed), and where  s and  2 s are mean and variance of S n. If X k are iid with mean  and variance  2  s = n   2 s = n  2

if i.e. for at least some  > 2, the  th moments exist for all X k. 1) is true if  k >  > 0  k 2) is true if all are 0 outside the interval [-c,c] and in many other situations.

In the discrete case, if S n takes values k, e.g. If X k are 0/1 outcomes of a Bernoulli trials, S n has a binomial distribution.  s = n p,  2 s = n pq ( q = 1-p ) By the CLT which is called as de Moivre - Laplace Theorem.

Convergence of R.V. Sequences How does a sequence of R.V.’s, X 1,X 2,.... Converge to R.V X ? X n  X as n   Sequence of R.V.’s: A function that assigns a countably infinite number of real values, X k, to each outcome, , from a sample space, S. sequence = { X n (  ) } or { X n } e.g. S = (0,1) ( i.e.   (0,1) ) X n (  ) =  ( 1- 1/n ) So X n (  )  

Convergence: X n  X if, for any  > 0,  integer N such that | X n – X | N Cauchy Criterion: X n  X iff, for any  > 0,  integer N such that | X n – X m | N XnXn nN X }2 

Types of Convergence Sure Convergence: {X n (  ) }  X (  ) surely If X n (  )  X (  ) as n      S. i.e. The sample sequence for each  converges, though possible to different values for different  ’s. Almost Sure Convergence: almost {X n (  ) }  X (  ) surely If P(  : X n (  )  X (  ) as n   ) = 1 So there might be some outcomes for which X n (  ) does not converge, but they have probability 0. e.g. strong LOLN.

Mean-square Convergence: MS {X n (  ) }  X (  ) If E [ ( X n (  ) - X (  ) ) 2 ]  0 as n   [ In Cauchy criterion terms E [ ( X n (  ) - X m (  ) ) 2 ]  0 as m, n   Convergence in probability: prob {X n (  ) }  X (  ) if for any  > 0 P( | X n (  ) - X (  ) | >  )  0 as n   i.e. The probability of being within 2  of X (  ) converges, not X n (  ) themselves. ( Weak law of large numbers for X n = M n, X =  )

Convergence in distribution: Sequence {X n } with cdf’s {F n (x) } converges to X with cdf F(x) if F n (x)  F(x) as n    x at which F(x) is continuous. e.g. Central Limit Theorem. distprob a.s s m.s. Relationships between convergence modes

Example :

But Y n has a well defined distribution So Y n converges in distribution as n  