1. Multinomial Distribution
The Binomial distribution can be extended to describe number of
outcomes in a series of independent trials each having more than 2
possible outcomes.
If a given trail can result in the k outcomes E1, E2, …, Ek with
probabilities p1, p2, …, pk, then the probability distribution of the
random variables X1, X2, …, Xk, representing the number of
occurrences for E1, E2, …, Ek in n independent trials is
with , and
STA347 - week 8

2. Properties of Expectations Involving Joint Distributions
For random variables X, Y and constants
E(aX + bY) = aE(X) + bE(Y)
Proof:
For independent random variables X, Y
E(XY) = E(X)E(Y)
whenever these expectations exist.
STA347 - week 8

3. Covariance Recall: Var(X+Y) = Var(X) + Var(Y) +2 E[(X-E(X))(Y-E(Y))]
Definition
For random variables X, Y with E(X), E(Y) < ∞, the covariance of X and Y is
Covariance measures whether or not X-E(X) and Y-E(Y) have the same sign.
Claim:
Proof:
Note: If X, Y independent then E(XY) =E(X)E(Y), and Cov(X,Y) = 0.
STA347 - week 8

4. Example
Suppose X, Y are discrete random variables with probability function given by
Find Cov(X,Y). Are X,Y independent?
y x -1 1
pX(x)
1/8
pY(y)
STA347 - week 8

5. Important Facts
Independence of X, Y implies Cov(X,Y) = 0 but NOT vice versa.
If X, Y independent then Var(X+Y) = Var(X) + Var(Y).
If X, Y are NOT independent then
Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y).
Cov(X,X) = Var(X).
STA347 - week 8

6. Properties of Covariance
For random variables X, Y, Z and constants
Cov(aX+b, cY+d) = acCov(X,Y)
Cov(X+Y, Z) = Cov(X,Z) + Cov(Y,Z)
Cov(X,Y) = Cov(Y, X)
STA347 - week 8

7. Correlation Definition
For X, Y random variables the correlation of X and Y is
whenever V(X), V(Y) ≠ 0 and all these quantities exists.
Claim:
ρ(aX+b,cY+d) = ρ(X,Y)
Proof:
This claim means that the correlation is scale invariant.
STA347 - week 8

8. Theorem
For X, Y random variables, whenever the correlation ρ(X,Y) exists it must satisfy
-1 ≤ ρ(X,Y) ≤ 1
Proof:
STA347 - week 8

9. Interpretation of Correlation ρ
ρ(X,Y) is a measure of the strength and direction of the linear relationship between X, Y.
If X, Y have non-zero variance, then
If X, Y independent, then ρ(X,Y) = 0. Note, it is not the only time when ρ(X,Y) = 0 !!!
Y is a linearly increasing function of X if and only if ρ(X,Y) = 1.
Y is a linearly decreasing function of X if and only if ρ(X,Y) = -1.
STA347 - week 8

10. Example
Find Var(X - Y) and ρ(X,Y) if X, Y have the following joint density
STA347 - week 8

11. Generating Functions
For a sequence of real numbers {aj} = a0, a1, a2 ,…, the generating function of {aj} is
if this converges for |t| < t0 for some t0 > 0.
STA347 - week 8

12. Probability Generating Functions
Suppose X is a random variable taking the values 0, 1, 2, … (or a subset of
the non-negative integers).
Let pj = P(X = j) , j = 0, 1, 2, …. This is in fact a sequence p0, p1, p2, …
Definition: The probability generating function of X is
Since if |t| < 1 and the pgf converges absolutely at least
for |t| < 1.
In general, πX(1) = p0 + p1 + p2 +… = 1.
The pgf of X is expressible as an expectation:
STA347 - week 8

13. Examples X ~ Binomial(n, p), converges for all real t.
X ~ Geometric(p),
converges for |qt| < 1 i.e.
Note: in this case pj = pqj for j = 1, 2, …
STA347 - week 8

14. PGF for sums of independent random variables
If X, Y are independent and Z = X+Y then,
Example
Let Y ~ Binomial(n, p). Then we can write Y = X1+X2+…+ Xn . Where Xi’s are i.i.d Bernoulli(p). The pgf of Xi is
The pgf of Y is then
STA347 - week 8

15. Use of PGF to find probabilities
Theorem
Let X be a discrete random variable, whose possible values are the nonnegative integers. Assume πX(t0) < ∞ for some t0 > 0. Then
πX(0) = P(X = 0),
etc. In general,
where is the kth derivative of πX with respect to t.
Proof:
STA347 - week 8

16. Example Suppose X ~ Poisson(λ). The pgf of X is given by
Using this pgf we have that
STA347 - week 8

17. Finding Moments from PGFs
Theorem
Let X be a discrete random variable, whose possible values are the
nonnegative integers. If πX(t) < ∞ for |t| < t0 for some t0 > 1. Then
etc. In general,
Where is the kth derivative of πX with respect to t.
Note: E(X(X-1)∙∙∙(X-k+1)) is called the kth factorial moment of X.
Proof:
STA347 - week 8

18. Example Suppose X ~ Binomial(n, p). The pgf of X is πX(t) = (pt+q)n.
Find the mean and the variance of X using its pgf.
STA347 - week 8

19. Uniqueness Theorem for PGF
Suppose X, Y have probability generating function πX and πY respectively. Then πX(t) = πY(t) if and only if P(X = k) = P(Y = k) for k = 0,1,2,…
Proof:
Follow immediately from calculus theorem:
If a function is expressible as a power series at x=a, then there is only one such series.
A pgf is a power series about the origin which we know exists with radius of convergence of at least 1.
STA347 - week 8

20. Moment Generating Functions
The moment generating function of a random variable X is
mX(t) exists if mX(t) < ∞ for |t| < t0 >0
If X is discrete
If X is continuous
Note: mX(t) = πX(et).
STA347 - week 8

21. Examples X ~ Exponential(λ). The mgf of X is
X ~ Uniform(0,1). The mgf of X is
STA347 - week 8

22. Generating Moments from MGFs
Theorem
Let X be any random variable. If mX(t) < ∞ for |t| < t0 for some t0 > 0. Then
mX(0) = 1
etc. In general,
Where is the kth derivative of mX with respect to t.
Proof:
STA347 - week 8

23. Example
Suppose X ~ Exponential(λ). Find the mean and variance of X using its
moment generating function.
STA347 - week 8

24. Example
Suppose X ~ N(0,1). Find the mean and variance of X using its moment
generating function.
STA347 - week 8

25. Properties of Moment Generating Functions
mX(0) = 1.
If Y=a+bX, then the mgf of Y is given by
If X,Y independent and Z = X+Y then,
STA347 - week 8

26. Uniqueness Theorem
If a moment generating function mX(t) exists for t in an open interval
containing 0, it uniquely determines the probability distribution.
STA347 - week 8

27. Example
Find the mgf of X ~ N(μ,σ2) using the mgf of the standard normal random variable.
Suppose, , independent.
Find the distribution of X1+X2 using mgf approach.
STA347 - week 8

28. Characteristic Function
The characteristic function, cX, of a random variable X is defined by:
The definition of the characteristics function is just like the definition of the
mgf, except for the introduction of the imaginary number
Using properties of complex numbers, we see that the characteristic
function can also be written as
The characteristics function unlike the mgf is always finite…
STA347 - week 8

29. Properties of Characteristic Function
The characteristics function has many nice properties similar to the mgf.
In particular, it can be used to generate moments.
Let X be any random variable with its first k moments finite. Then
cX(0) = 1
etc. In general,
Where is the kth derivative of cX with respect to s.
Let X and Y be independent random variables. Then
cX + Y (s) = cX(s) cY(s)
STA347 - week 8

30. Examples
STA347 - week 8