1. Probability and Statistics
Brief Review
Probability and Statistics

2. Probability distributions
Continuous distributions

3. Defn (density function)
Let x denote a continuous random variable then f(x) is called the density function of x
1) f(x) ≥ 0 2) 3)

4. Defn (Joint density function)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables then
f(x) = f(x1 ,x2 ,x3 , ... , xn)
is called the joint density function of x = (x1 ,x2 ,x3 , ... , xn) if
1) f(x) ≥ 0 2) 3)

5. Note:

6. Defn (Marginal density function)
The marginal density of x1 = (x1 ,x2 ,x3 , ... , xp) (p < n) is defined by:
f1(x1) = =
where x2 = (xp+1 ,xp+2 ,xp+3 , ... , xn)
The marginal density of x2 = (xp+1 ,xp+2 ,xp+3 , ... , xn) is defined by:
f2(x2) = =
where x1 = (x1 ,x2 ,x3 , ... , xp)

7. Defn (Conditional density function)
The conditional density of x1 given x2 (defined in previous slide) (p < n) is defined by:
f1|2(x1 |x2) =
conditional density of x2 given x1 is defined by:
f2|1(x2 |x1) =

8. Marginal densities describe how the subvector xi behaves ignoring xj
Conditional densities describe how the subvector xi behaves when the subvector xj is held fixed

9. Defn (Independence)
The two sub-vectors (x1 and x2) are called independent if:
f(x) = f(x1, x2) = f1(x1)f2(x2)
= product of marginals or
the conditional density of xi given xj :
fi|j(xi |xj) = fi(xi) = marginal density of xi

10. Example (p-variate Normal)
The random vector x (p × 1) is said to have the
p-variate Normal distribution with
mean vector m (p × 1) and
covariance matrix S (p × p)
(written x ~ Np(m,S)) if:

11. Example (bivariate Normal)
The random vector is said to have the bivariate
Normal distribution with mean vector
and
covariance matrix

15. Theorem (Transformations)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables with joint density function f(x1 ,x2 ,x3 , ... , xn) = f(x). Let
y1 =f1(x1 ,x2 ,x3 , ... , xn)
y2 =f2(x1 ,x2 ,x3 , ... , xn)
...
yn =fn(x1 ,x2 ,x3 , ... , xn)
define a 1-1 transformation of x into y.

16. Then the joint density of y is g(y) given by: g(y) = f(x)|J| where
= the Jacobian of the transformation

17. Corollary (Linear Transformations)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables with joint density function f(x1 ,x2 ,x3 , ... , xn) = f(x). Let
y1 = a11x1 + a12x2 + a13x3 , a1nxn
y2 = a21x1 + a22x2 + a23x3 , a2nxn
...
yn = an1x1 + an2x2 + an3x3 , annxn
define a 1-1 transformation of x into y.

18. Then the joint density of y is g(y) given by:

19. Corollary (Linear Transformations for Normal Random variables)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables having an n-variate Normal distribution with mean vector m and covariance matrix S.
i.e. x ~ Nn(m, S)
Let
y1 = a11x1 + a12x2 + a13x3 , a1nxn
y2 = a21x1 + a22x2 + a23x3 , a2nxn
...
yn = an1x1 + an2x2 + an3x3 , annxn
define a 1-1 transformation of x into y.
Then y = (y1 ,y2 ,y3 , ... , yn) ~ Nn(Am,ASA')

20. Defn (Expectation)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables with joint density function
f(x) = f(x1 ,x2 ,x3 , ... , xn).
Let U = h(x) = h(x1 ,x2 ,x3 , ... , xn)
Then

21. Defn (Conditional Expectation)
Let x = (x1 ,x2 ,x3 , ... , xn) = (x1 , x2 ) denote a vector of continuous random variables with joint density function
f(x) = f(x1 ,x2 ,x3 , ... , xn) = f(x1 , x2 ).
Let U = h(x1) = h(x1 ,x2 ,x3 , ... , xp)
Then the conditional expectation of U given x2

22. Defn (Variance)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables with joint density function
f(x) = f(x1 ,x2 ,x3 , ... , xn).
Let U = h(x) = h(x1 ,x2 ,x3 , ... , xn)
Then

23. Defn (Conditional Variance)
Let x = (x1 ,x2 ,x3 , ... , xn) = (x1 , x2 ) denote a vector of continuous random variables with joint density function
f(x) = f(x1 ,x2 ,x3 , ... , xn) = f(x1 , x2 ).
Let U = h(x1) = h(x1 ,x2 ,x3 , ... , xp)
Then the conditional variance of U given x2

24. Defn (Covariance, Correlation)
Let x = (x1 ,x2 ,x3 , ... , xn) denote a vector of continuous random variables with joint density function
f(x) = f(x1 ,x2 ,x3 , ... , xn).
Let U = h(x) = h(x1 ,x2 ,x3 , ... , xn) and
V = g(x) =g(x1 ,x2 ,x3 , ... , xn)
Then the covariance of U and V.

25. Properties
Expectation
Variance
Covariance
Correlation

26. E[a1x1 + a2x2 + a3x anxn]
= a1E[x1] + a2E[x2] + a3E[x3] anE[xn]
or E[a'x] = a'E[x]

27. E[UV] = E[h(x1)g(x2)]
= E[U]E[V] = E[h(x1)]E[g(x2)]
if x1 and x2 are independent

28. Var[a1x1 + a2x2 + a3x anxn]
or Var[a'x] = a′S a

29. Cov[a1x1 + a2x anxn ,
b1x1 + b2x bnxn]
or Cov[a'x, b'x] = a′S b

31. Statistical Inference
Making decisions from data

32. There are two main areas of Statistical Inference
Estimation – deciding on the value of a parameter
Point estimation
Confidence Interval, Confidence region Estimation
Hypothesis testing
Deciding if a statement (hypotheisis) about a parameter is True or False

33. The general statistical model Most data fits this situation

34. Defn (The Classical Statistical Model)
The data vector
x = (x1 ,x2 ,x3 , ... , xn)
The model
Let f(x| q) = f(x1 ,x2 , ... , xn | q1 , q2 ,... , qp) denote the joint density of the data vector x = (x1 ,x2 ,x3 , ... , xn) of observations where the unknown parameter vector q  W (a subset of p-dimensional space).

35. An Example The data vector
x = (x1 ,x2 ,x3 , ... , xn) a sample from the normal distribution with mean m and variance s2
The model
Then f(x| m , s2) = f(x1 ,x2 , ... , xn | m , s2), the joint density of x = (x1 ,x2 ,x3 , ... , xn) takes on the form:
where the unknown parameter vector q = (m , s2)  W ={(x,y)|-∞ < x < ∞ , 0 ≤ y < ∞}.

36. Defn (Sufficient Statistics)
Let x have joint density f(x| q) where the unknown parameter vector q  W.
Then S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is called a set of sufficient statistics for the parameter vector q if the conditional distribution of x given S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is not functionally dependent on the parameter vector q.
A set of sufficient statistics contains all of the information concerning the unknown parameter vector

37. A Simple Example illustrating Sufficiency
Suppose that we observe a Success-Failure experiment n = 3 times. Let q denote the probability of Success. Suppose that the data that is collected is x1, x2, x3 where xi takes on the value 1 is the ith trial is a Success and 0 if the ith trial is a Failure.

38. The following table gives possible values of (x1, x2, x3).
The data can be generated in two equivalent ways:
Generating (x1, x2, x3) directly from f (x1, x2, x3|q) or
Generating S from g(S|q) then generating (x1, x2, x3) from f (x1, x2, x3|S). Since the second step does involve q, no additional information will be obtained by knowing (x1, x2, x3) once S is determined

39. The Sufficiency Principle
Any decision regarding the parameter q should be based on a set of Sufficient statistics S1(x), S2(x), ...,Sk(x) and not otherwise on the value of x.

40. A useful approach in developing a statistical procedure
Find sufficient statistics
Develop estimators , tests of hypotheses etc. using only these statistics

41. Defn (Minimal Sufficient Statistics)
Let x have joint density f(x| q) where the unknown parameter vector q  W.
Then S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is a set of Minimal Sufficient statistics for the parameter vector q if S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is a set of Sufficient statistics and can be calculated from any other set of Sufficient statistics.

42. Theorem (The Factorization Criterion)
Let x have joint density f(x| q) where the unknown parameter vector q  W.
Then S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is a set of Sufficient statistics for the parameter vector q if
f(x| q) = h(x)g(S, q)
= h(x)g(S1(x) ,S2(x) ,S3(x) , ... , Sk(x), q).
This is useful for finding Sufficient statistics
i.e. If you can factor out q-dependence with a set of statistics then these statistics are a set of Sufficient statistics

43. Defn (Completeness)
Let x have joint density f(x| q) where the unknown parameter vector q  W.
Then S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is a set of Complete Sufficient statistics for the parameter vector q if S = (S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) is a set of Sufficient statistics and whenever
E[f(S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) ] = 0
then
P[f(S1(x) ,S2(x) ,S3(x) , ... , Sk(x)) = 0] = 1

44. Defn (The Exponential Family)
Let x have joint density f(x| q)| where the unknown parameter vector q  W. Then f(x| q) is said to be a member of the exponential family of distributions if:
q W,where

45. - ∞ < ai < bi < ∞ are not dependent on q.
2) W contains a nondegenerate k-dimensional rectangle.
3) g(q), ai ,bi and pi(q) are not dependent on x.
4) h(x), ai ,bi and Si(x) are not dependent on q.

46. If in addition.
5) The Si(x) are functionally independent for i = 1, 2,..., k.
6) [Si(x)]/ xj exists and is continuous for all i = 1, 2,..., k j = 1, 2,..., n.
7) pi(q) is a continuous function of q for all i = 1, 2,..., k.
8) R = {[p1(q),p2(q), ...,pK(q)] | q  W,} contains nondegenerate k-dimensional rectangle.
Then
the set of statistics S1(x), S2(x), ...,Sk(x) form a Minimal Complete set of Sufficient statistics.

47. Defn (The Likelihood function)
Let x have joint density f(x|q) where the unkown parameter vector q W. Then for a
given value of the observation vector x ,the Likelihood function, Lx(q), is defined by:
Lx(q) = f(x|q) with q W
The log Likelihood function lx(q) is defined by:
lx(q) =lnLx(q) = lnf(x|q) with q W

48. The Likelihood Principle
Any decision regarding the parameter q should be based on the likelihood function Lx(q) and not otherwise on the value of x.
If two data sets result in the same likelihood function the decision regarding q should be the same.

49. Some statisticians find it useful to plot the likelihood function Lx(q) given the value of x.
It summarizes the information contained in x regarding the parameter vector q.

50. An Example The data vector
x = (x1 ,x2 ,x3 , ... , xn) a sample from the normal distribution with mean m and variance s2
The joint distribution of x
Then f(x| m , s2) = f(x1 ,x2 , ... , xn | m , s2), the joint density of x = (x1 ,x2 ,x3 , ... , xn) takes on the form:
where the unknown parameter vector q = (m , s2)  W ={(x,y)|-∞ < x < ∞ , 0 ≤ y < ∞}.

51. The Likelihood function
Assume data vector is known
x = (x1 ,x2 ,x3 , ... , xn)
The Likelihood function
Then L(m , s)= f(x| m , s) = f(x1 ,x2 , ... , xn | m , s2),

52. or

53. hence
Now consider the following data: (n = 10)

54. 70 m s 50 20

55. 70 m 50 20 s

56. Now consider the following data: (n = 100)

57. 70 m s 50 20

58. 70 m 50 20 s

59. The Sufficiency Principle
Any decision regarding the parameter q should be based on a set of Sufficient statistics S1(x), S2(x), ...,Sk(x) and not otherwise on the value of x.
If two data sets result in the same values for the set of Sufficient statistics the decision regarding q should be the same.

60. Theorem (Birnbaum - Equivalency of the Likelihood Principle and Sufficiency Principle)
Lx1(q)  Lx2(q)
if and only if
S1(x1) = S1(x2),..., and Sk(x1) = Sk(x2)

61. The following table gives possible values of (x1, x2, x3).
The Likelihood function

62. Estimation Theory
Point Estimation

63. Defn (Estimator)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q W.
Then an estimator of the parameter f(q) = f(q1 ,q2 , ... , qk) is any function T(x)=T(x1 ,x2 ,x3 , ... , xn) of the observation vector.

64. Defn (Mean Square Error)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W.
Let T(x) be an estimator of the parameter
f(q). Then the Mean Square Error of T(x) is defined to be:

65. Defn (Uniformly Better)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W.
Let T(x) and T*(x) be estimators of the parameter f(q). Then T(x) is said to be uniformly better than T*(x) if:

66. Defn (Unbiased )
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W.
Let T(x) be an estimator of the parameter f(q). Then T(x) is said to be an unbiased estimator of the parameter f(q) if:

67. Theorem (Cramer Rao Lower bound)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W. Suppose that:
i) exists for all x and for all
ii)
iii)
iv)

68. Let M denote the p x p matrix with ijth element.
Then V = M-1 is the lower bound for the covariance matrix of unbiased estimators of q.
That is, var(c' ) = c'var( )c ≥ c'M-1c = c'Vc where is a vector of unbiased estimators of q.

69. Defn (Uniformly Minimum Variance Unbiased Estimator)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W. Then T*(x) is said to be the UMVU (Uniformly minimum variance unbiased) estimator of f(q) if:
1) E[T*(x)] = f(q) for all q  W.
2) Var[T*(x)] ≤ Var[T(x)] for all q  W whenever E[T(x)] = f(q).

70. Theorem (Rao-Blackwell)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W.
Let S1(x), S2(x), ...,SK(x) denote a set of sufficient statistics.
Let T(x) be any unbiased estimator of f(q).
Then T*[S1(x), S2(x), ...,Sk (x)] = E[T(x)|S1(x), S2(x), ...,Sk (x)] is an unbiased estimator of f(q) such that:
Var[T*(S1(x), S2(x), ...,Sk(x))] ≤ Var[T(x)]
for all q  W.

71. Theorem (Lehmann-Scheffe')
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W.
Let S1(x), S2(x), ...,SK(x) denote a set of complete sufficient statistics.
Let T*[S1(x), S2(x), ...,Sk (x)] be an unbiased estimator of f(q). Then:
T*(S1(x), S2(x), ...,Sk(x)) )] is the UMVU estimator of f(q).

72. Defn (Consistency)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W. Let Tn(x) be an estimator of f(q). Then Tn(x) is called a consistent estimator of f(q) if for any e > 0:

73. Defn (M. S. E. Consistency)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x|q) where the unknown parameter vector q  W. Let Tn(x) be an estimator of f(q). Then Tn(x) is called a M. S. E. consistent estimator of f(q) if for any e > 0:

74. Methods for Finding Estimators
The Method of Moments
Maximum Likelihood Estimation

75. Methods for finding estimators
Method of Moments
Maximum Likelihood Estimation

76. Method of Moments
Let x1, … , xn denote a sample from the density function
f(x; q1, … , qp) = f(x; q)
The kth moment of the distribution being sampled is defined to be:

77. The kth sample moment is defined to be:
To find the method of moments estimator of q1, … , qp we set up the equations:

78. We then solve the equations
for q1, … , qp.
The solutions
are called the method of moments estimators

79. The Method of Maximum Likelihood
Suppose that the data x1, … , xn has joint density function
f(x1, … , xn ; q1, … , qp)
where q = (q1, … , qp) are unknown parameters assumed to lie in W (a subset of p-dimensional space).
We want to estimate the parametersq1, … , qp

80. Definition: Maximum Likelihood Estimation
Suppose that the data x1, … , xn has joint density function
f(x1, … , xn ; q1, … , qp)
Then the Likelihood function is defined to be
L(q) = L(q1, … , qp)
= f(x1, … , xn ; q1, … , qp)
the Maximum Likelihood estimators of the parameters q1, … , qp are the values that maximize

81. the Maximum Likelihood estimators of the parameters q1, … , qp are the values
Such that
Note:
is equivalent to maximizing
the log-likelihood function

82. The General Linear Model
Application
The General Linear Model

83. Consider the random variable Y with 1. E[Y] = g(U1 ,U2 , ... , Uk)
= b1f1(U1 ,U2 , ... , Uk) + b2f2(U1 ,U2 , ... , Uk) bpfp(U1 ,U2 , ... , Uk) =
and
2. var(Y) = s2
where b1, b2 , ... ,bp are unknown parameters
and f1 ,f2 , ... , fp are known functions of the nonrandom variables U1 ,U2 , ... , Uk.
Assume further that Y is normally distributed.

84. Thus the density of Y is:
f(Y|b1, b2 , ... ,bp, s2) = f(Y| b, s2)
i = 1,2, … , p

85. Now suppose that n independent observations of Y,
(y1, y2, ..., yn) are made
corresponding to n sets of values of (U1 ,U2 , ... , Uk) - (u11 ,u12 , ... , u1k),
(u11 ,u12 , ... , u1k),
...
(u11 ,u12 , ... , u1k).
Let xij = fj(ui1 ,ui2 , ... , uik) j =1, 2, ..., p; i =1, 2, ..., n.
Then the joint density of y = (y1, y2, ... yn) is:
f(y1, y2, ..., yn|b1, b2 , ... ,bp, s2) = f(y|b, s2)

87. Thus f(y|b,s2) is a member of the exponential family of distributions
and S = (y'y, X'y) is a Minimal Complete set of Sufficient Statistics.

88. The General Linear Model
Estimation
The General Linear Model

89. The Maximum Likelihood estimates of b and s2 are the values
that maximize
or equivalently

90. yields the system of linear equations (The Normal Equations)
while
yields the equation:

91. If [X'X]-1 exists then the normal equations have solution:
and

92. Properties of The Maximum Likelihood Estimates
Unbiasedness, Minimum Variance

93. Thus is an unbiased estimator of . Since
Note:
and
Thus is an unbiased estimator of Since
is also a function of the set of complete minimal sufficient statistics, it is the UMVU estimator of (Lehman-Scheffe)

94. Note:
where
In general

95. Thus:
where

96. Thus:

97. Let
Then
Thus s2 is an unbiased estimator of s2.
Since s2 is also a function of the set of complete minimal sufficient statistics, it is the UMVU estimator of s2.

98. Hypothesis Testing

99. Defn (Test of size a)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Let w be any subset of W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.

100. Let A denote the acceptance region for the test
Let A denote the acceptance region for the test. (all values x = (x1 ,x2 ,x3 , ... , xn) of such that the decision to accept H0 is made.)
and let C denote the critical region for the test (all values x = (x1 ,x2 ,x3 , ... , xn) of such that the decision to reject H0 is made.).
Then the test is said to be of size a if

101. Defn (Power)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.
where w is any subset of W. Then the Power of the test for q  w is defined to be:

102. Defn (Uniformly Most Powerful (UMP) test of size a)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.
where w is any subset of W.
Let C denote the critical region for the test . Then the test is called the UMP test of size a if:

103. Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.
where w is any subset of W.
Let C denote the critical region for the test . Then the test is called the UMP test of size a if:

104. and for any other critical region C* such that:
then

105. Theorem (Neymann-Pearson Lemma)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W = (q0, q1).
Consider testing the the Null Hypothesis
H0: q = q0
against the alternative hypothesis
H1: q = q1.
Then the UMP test of size a has critical region:
where K is chosen so that

106. Defn (Likelihood Ratio Test of size a)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.
where w is any subset of W
Then the Likelihood Ratio (LR) test of size a has critical region:
where K is chosen so that

107. Theorem (Asymptotic distribution of Likelihood ratio test criterion)
Let x = (x1 ,x2 ,x3 , ... , xn) denote the vector of observations having joint density f(x| q) where the unknown parameter vector q  W.
Consider testing the the Null Hypothesis
H0: q  w
against the alternative hypothesis
H1: q  w.
where w is any subset of W
Then under proper regularity conditions on U = -2lnl(x) possesses an asymptotic Chi-square distribution with degrees of freedom equal to the difference between the number of independent parameters in W and w.