1. Time Series Analysis

2. Definition
A Time Series {xt : t  T} is a collection of random variables usually parameterized by
1) the real line T = R= (-∞, ∞)
2) the non-negative real line T = R+ = [0, ∞)
3) the integers T = Z = {…,-2, -1, 0, 1, 2, …}
4) the non-negative integers T = Z+ = {0, 1, 2, …}

3. If xt is a vector, the collection of random vectors
{xt : t  T}
is a multivariate time series or multi-channel time series.
If t is a vector, the collection of random variables
{xt : t  T} is a multidimensional “time” series or spatial series.
(with T = Rk= k-dimensional Euclidean space or a k-dimensional lattice.)

4. Example of spatial time series

5. The project Buoys are located in a grid across the Pacific ocean
Measuring
Surface temperature
Wind speed (two components)
Other measurements
The data is being collected almost continuously
The purpose is to study El Nino

6. Technical Note:
The probability measure of a time series is defined by specifying the joint distribution (in a consistent manner) of all finite subsets of {xt : t  T}.
i.e. marginal distributions of subsets of random variables computed from the joint density of a complete set of variables should agree with the distribution assigned to the subset of variables.

7. The time series is Normal if all finite subsets of
{xt : t  T} have a multivariate normal distribution.
Similar statements are true for multi-channel time series and multidimensional time series.

8. Definition: m(t) = mean value function of {xt : t T} = E[xt]
for t  T.
s(t,s) = covariance function of {xt : t  T}
= E[(xt - m(t))(xs - m(s))] for t,s  T.

9. For multichannel time series
m(t) = mean vector function of {xt : t  T} = E[xt]
for t  T and
S(t,s) = covariance matrix function of {xt : t  T}
= E[(xt - m(t))(xs - m(s))′] for t,s  T.
The ith element of the k × 1 vector m(t)
mi(t) =E[xit]
is the mean value function of the time series {xit : t  T}
The i,jth element of the k × k matrix S(t,s)
sij(t,s) =E[(xit - mi(t))(xjs - mj(s))]
is called the cross-covariance function of the two time series
{xit : t  T} and {xjt : t  T}

10. Definition:
The time series {xt : t  T} is stationary if the joint distribution of xt1, xt2, ... , xtk is the same as the joint distribution of xt1+h ,xt2+h , ... ,xtk+h for all finite subsets t1, t2, ... , tk of T and all choices of h.

11. Definition:
The multi-channel time series {xt : t  T} is stationary if the joint distribution of xt1, xt2, ... , xtk is the same as the joint distribution of xt1+h , xt2+h , ... , xtk+h for all finite subsets t1, t2, ... , tk of T and all choices of h.

12. Definition:
The multidimensional time series {xt : t  T} is stationary if the joint distribution of xt1, xt2, ... , xtk is the same as the joint distribution of xt1+h ,xt2+h , ... ,xtk+h for all finite subsets t1, t2, ... , tk of T and all choices of h.

13. Stationarity The distribution of observations at these points in time
same as
Time
Stationarity

14. Some Implication of Stationarity
If {xt : t  T} is stationary then:
The distribution of xt is the same for all t  T.
The joint distribution of xt, xt + h is the same as the joint distribution of xs, xs + h .

15. Implication of Stationarity for the mean value function and the covariance function
If {xt : t  T} is stationary then for t  T.
m(t) = E[xt] = m
and for t,s  T.
s(t,s) = E[(xt - m)(xs - m)]
= E[(xt+h - m)(xs+h - m)]
= E[(xt-s - m)(x0 - m)] with h = -s
= s(t-s)

16. If the multi-channel time series{xt : t  T} is stationary then for t  T.
m(t) = E[xt] = m
and for t,s T
S(t,s) = S(t-s)
Thus for stationary time series the mean value function is constant and the covariance function is only a function of the distance in time (t – s)

17. If the multidimensional time series {xt : t T} is stationary then for t  T.
m(t) = E[xt] = m
and for t,s  T.
s(t,s) = E[(xt - m)(xs - m)]
= s(t-s) (called the Covariogram)
Variogram
V(t,s) = V(t - s) = Var[(xt - xs)] = E[(xt - xs)2]
= Var[xt] + Var[xs] –2Cov[xt,xs]
= 2[s(0) - s(t-s)]

18. Definition: r(t,s) = autocorrelation function of {xt : t T}
= correlation between xt and xs.
for t,s  T.
If {xt : t  T} is stationary then
r(h) = autocorrelation function of {xt : t  T}
= correlation between xt and xt+h.

19. Definition: The time series {xt : t  T} is weakly stationary if:
m(t) = E[xt] = m for all t  T.
and
s(t,s) = s(t-s) for all t,s  T. or
r(t,s) = r(t-s) for all t,s  T.

20. Stationary time series
Examples
Stationary time series

21. Let X denote a single random variable with mean m and standard deviation s. In addition X may also be Normal (this condition is not necessary)
Let xt = X for all t  T = { …,, -2, -1, 0, 1, 2, …}
Then E[xt] = m = E[X] for t  T and
s(h) = E[(xt+h - m)(xt - m)]
= Cov(xt+h,xt )
= E[(X - m)(X - m)] = Var(X)
= s2 for all h.

22. Excel file illustrating this time series

23. Suppose {xt : t  T} are identically distributed and uncorrelated (independent).
Then E[xt] = m for t  T and
s(h) = E[(xt+h - m)(xt - m)]
= Cov(xt+h,xt )

24. The auto correlation function:
Comment:
If m = 0 then the time series {xt : t  T} is called a white noise time series.
Thus a white noise time series consist of independent identically distributed random variables with mean 0 and common variance s2

25. Excel file illustrating this time series

26. Suppose X1, X2, … , Xk and Y1, Y2, … , Yk are independent independent random variables with
Let l1, l2, … lk denote k values in (0,p)
For any t T = { …,, -2, -1, 0, 1, 2, …}

27. Excel file illustrating this time series

28. Then

29. Hence

30. Hence using
cos(A – B) = cos(A) cos(B) + sin(A) sin(B)
and

31. The Moving Average Time series of order q, MA(q)
Let a0 =1, a1, a2, … aq denote q + 1 numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
Then {xt|t  T} is called a Moving Average time series of order q. MA(q)

32. Excel file illustrating this time series

33. The mean
The auto covariance function

35. The autocovariance function for an MA(q) time series
The autocorrelation function for an MA(q) time series

36. The Autoregressive Time series of order p, AR(p)
Let b1, b2, … bp denote p numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
Then {xt|t  T} is called a Autoregressive time series of order p. AR(p)

37. Excel file illustrating this time series

38. Comment:
An Autoregressive time series is not necessarily stationary.
Suppose {xt|t  T} is an AR(1) time series satisfying the equation:
where {ut|t  T} is a white noise time series with variance s2. i.e. b1 = 1 and d = 0.

39. but
and is not constant.
A time series {xt|t  T} satisfying the equation:
is called a Random Walk.

40. We use extensively the rules of expectation
Derivation of the mean, autocovariance function and autocorrelation function of a stationary Autoregressive time series
We use extensively the rules of expectation

41. Assume that the autoregressive time series {xt|t T} be defined by the equation:
is stationary.
Let m = E(xt). Then

42. The Autocovariance function, s(h)
The Autocovariance function, s(h), of a stationary autoregressive time series {xt|t  T}can be determined by using the equation:
Thus

43. Hence
where

44. Now

45. The equations for the autocovariance function of an AR(p) time series
etc

46. Or using s(-h) = s(h)
and
for h > p

47. Use the first p + 1 equations to find s(0), s(1) and s(p)
Then use
To compute s(h)
for h > p

48. The Autoregressive Time series of order p, AR(p)
Let b1, b2, … bp denote p numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
Then {xt|t  T} is called a Autoregressive time series of order p. AR(p)

49. If the autoregressive time series {xt|t  T} be defined by the equation:
is stationary.
Then

50. The Autocovariance function, s(h), of a stationary autoregressive time series {xt|t  T} be defined by the equation:
Satisfy the equations:

51. The autocovariance function for an AR(p) time series
The mean
The autocovariance function for an AR(p) time series
Yule Walker Equations
and
for h > p

52. Use the first p + 1 equations (the Yole-Walker Equations) to find s(0), s(1) and s(p)
Then use
To compute s(h)
for h > p

53. The Autocorrelation function, r(h), of a stationary autoregressive time series {xt|t  T}:
The Yule walker Equations become:

54. and
for h > p

55. To find r(h) and s(0): solve for r(1), …, r(p)
Then
for h > p

56. Suppose X1, X2, … , Xk and Y1, Y2, … , Yk are independent independent random variables with
Let l1, l2, … lk denote k values in (0,p)
For any t T = { …,, -2, -1, 0, 1, 2, …}

57. Three important Models of Stationary Time Series
Sin-Cosine series with k frequencies
Moving Average Time series of order q – MA(q)
Autoregressive time series of order q - AR(q)
Comment: Any non-trivial stationary time series can be approximated by either
Sin-Cos series of order k
MA(q) time series, or
AR(p) time series, or

58. Sin-Cos series of order k
for any t T = { …,, -2, -1, 0, 1, 2, …}
where l1, l2, … lk denote k fixed values in (0,p)
and X1, X2, … , Xk and Y1, Y2, … , Yk are independent random variables with

59. The Moving Average Time series of order q, MA(q)
Let a0 =1, a1, a2, … aq denote q + 1 numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
Then {xt|t  T} is called a Moving Average time series of order q. MA(q)

60. The Autoregressive Time series of order p, AR(p)
Let b1, b2, … bp denote p numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
Then {xt|t  T} is called a Autoregressive time series of order p. AR(p)

61. Mean Value, autocovariance, autocorrelation function for a Sin-Cos(k) time series (of order k)
The autocovariance function for an MA(q) time series
The autocorrelation function for an MA(q) time series

62. Mean Value, autocovariance, autocorrelation function for an MA(q) time series
The autocovariance function for an MA(q) time series
The autocorrelation function for an MA(q) time series

63. The autocovariance function for an AR(p) time series
Mean Value, autocovariance, autocorrelation function for an AR(p) time series
The autocovariance function for an AR(p) time series
Yule Walker Equations
and

64. The autocorrelation function for an AR(p) time series
Yule Walker Equations
for h > p
and

65. Example Consider the AR(2) time series:
xt = 0.7xt – xt – ut
where {ut} is a white noise time series with standard deviation s = 2.0
White noise ≡ independent, mean zero (normal)
Find m, s(h), r(h)

66. To find r(h) solve the equations:or
thus

67. for h > 2
This can be used in sequence to find:
results

68. To find s(0) use: or =

69. To find s(h) use:
To find m use:

70. An explicit formula for r(h)
Auto-regressive time series of order p.

71. Consider solving the difference equation:
This difference equation can be solved by:
Setting up the polynomial
where r1, r2, … , rp are the roots of the polynomial b(x).

72. The difference equation
has the general solution:
where c1, c2, … , cp are determined by using the starting values of the sequence r(h).

73. Example: An AR(1) time series
for h > 1
and

74. The difference equation
Can also be solved by:
Setting up the polynomial
Then a general formula for r(h) is:

75. Example: An AR(2) time series
for h > 1

76. Setting up the polynomial

77. Then a general formula for r(h) is:
For h = 0 and h = 1.
Solving for c1 and c2.

78. Solving for c1 and c2.
and
Then a general formula for r(h) is:

79. are real and If
is a mixture of two exponentials

80. are complex conjugates.If
are complex conjugates.
Some important complex identities

81. The above identities can be shown using the power series expansions:

82. Some other trig identities:

84. Hence

86. a damped cosine wave

87. Example Consider the AR(2) time series:
xt = 0.7xt – xt – ut
where {ut} is a white noise time series with standard deviation s = 2.0
The correlation function found before using the difference equation:
r(h) = 0.7 r(h – 1) r(h – 2)

88. Alternatively setting up the polynomial

89. Thus

90. Another Example Consider the AR(2) time series:
xt = 0.2xt – xt – ut
where {ut} is a white noise time series with standard deviation s = 2.0
The correlation function found before using the difference equation:
r(h) = 0.2 r(h – 1) r(h – 2)

91. Alternatively setting up the polynomial

92. Thus
where
and

94. Conditions for stationarity
Autoregressive Time series of order p, AR(p)

95. The Autoregressive Time series of order p, AR(p)
For certain values of b1, b2, … bp the time series is not stationary
Consider the AR(1) time series {xt|t  T} with d = 0
If b1 = 1
The series is a random walk which not stationary

96. If |b1 | > 1 and d = 0.
and the value of xt increases in magnitude and ut eventually becomes negligible.
The time series {xt|t  T} satisfies the equation:
The time series {xt|t  T} exhibits deterministic behaviour.
Finally if |b1 | < 1 then the time series {xt|t  T} is stationary |

97. Summarizing
If |b1 | < 1 then the time series {xt|t  T} is stationary |
If |b1 | > 1 then the time series {xt|t  T} is deterministic
If |b1 | = 1 then the time series {xt|t  T} is random but non-stationary

98. Generalizing
Let b1, b2, … bp denote p numbers.
Let {ut|t  T} denote a white noise time series with variance s2.
independent
mean 0, variance s2.
Let {xt|t  T} be defined by the equation.
i.e. {xt|t  T} is an Autoregressive time series of order p.

99. Consider the polynomial
with roots r1, r2 , … , rp
then {xt|t T} is stationary if |ri| > 1 for all i.
If |ri| < 1 for at least one i then {xt|t T} exhibits deterministic behaviour.
If |ri| ≥ 1 and |ri| = 1 for at least one i then {xt|t  T} exhibits non-stationary random behaviour.

100. Special Cases: The AR(1) time
Let {xt|t  T} be defined by the equation.

101. Consider the polynomial
with root r1= 1/b1
{xt|t T} is stationary if |r1| > 1 or |b1| < 1 .
If |ri| < 1 or |b1| > 1 then {xt|t T} exhibits deterministic behaviour.
If |ri| = 1 or |b1| = 1 then {xt|t T} exhibits non-stationary random behaviour.

102. Special Cases: The AR(2) time
Let {xt|t T} be defined by the equation.

103. Consider the polynomial
where r1 and r2 are the roots of b(x)
{xt|t T} is stationary if |r1| > 1 and |r2| > 1 .
This is true if b1+b2 < 1 , b2 –b1 < 1 and b2 > -1.
These inequalities define a triangular region for b1 and b2.
If |ri| < 1 then {xt|t T} exhibits deterministic behaviour.
If |ri| ≤ 1 for i = 1,2 and |ri| = 1 for at least on i then {xt|t T} exhibits non-stationary random behaviour.

104. Patterns of the ACF and PACF of AR(2) Time Series
In the shaded region the roots of the AR operator are complex b2