1. EE255/CPS226 Stochastic Processes
Dept. of Electrical & Computer engineering
Duke University
2/22/2019

2. What is a stochastic process?
Stochastic Process: is a family of rvs {X(t)|t ε T} (T is an index set; it may be discrete or continuous)
Values assumed by X(t) are called states.
State space (I): set of all possible states
Example: cosmic radio noise at antenna {a1, a2, .., ak}.
If x and y are mutually independent, then, p(y|x) = p(y). t1
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

3. Stochastic Process Characterization
Sample space S: set of antennas.
Sample the output of all antennas at time t1 (  rv), i.e. we can define rv {X(t1)}.
In general, we can define:
At a fixed time t=t1, we can define Xt1(s) = X(t1,s) (rv X(t1)). Similarly, we can define, X(t2), .., X(tk).
X(t1) can be characterized by its distribution function,
We can also a joint variable, characterized by its CDF as,
Discrete and continuous cases:
States X(t) (i.e. time t) may be discrete/continuous
State space I (i.e. sample space S) may be discrete/continuous
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

4. Classification of Stochastic Processes
Four classes of stochastic processes:
Discrete-state process  chain (e.g., DJIA index at any time)
discrete-time process  stochastic sequence {Xn | n є T} (e.g., probing a system every 10 ms.)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

5. Example: a Queuing System
Inter arrival times Y1, Y2, … (mutually independent) (FY)
Service times: S1, S2, … (mutually independent) (FS)
Notation for a queuing system: Fy /FY /m
Possible arrival/service time distributions types are:
M: Memory-less (i.e., EXP)
D: Deterministic
G: General distribution
Ek: k-stage Erlang etc.
M/M/1  Memory-less arrival/departure processes with 1-service station
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

6. Discrete/Continuous Stochastic Processes
Nk: Number of jobs waiting in the system at the time of kth job’s departure  Stochastic process {Nk|k=1,2,…}:
Discrete time, discrete space Nk
Discrete k
Discrete
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

7. Continuous Time, Discrete Space
X(t): Number of jobs in the system at time t. {X(t)|t є T} forms a continuous-time, discrete-state stochastic process, with,
X(t)
Discrete
Continuous
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

8. Discrete Time, Continuous Space
Wk: wait time for the kth job. Then {Wk| k є T} forms a Discrete-time, Continuous-state stochastic process, where, Wk
Continuous k
Discrete
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

9. Continuous Time, Continuous Space
Y(t): total service time for all jobs in the system at time t. Y(t) forms a continuous-time, continuous-state stochastic process, Where,
Y(t) t
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

10. Further Classification
Similarly, we can define nth order distribution:
Difficult to compute nth order distribution.
(1st order distribution)
(2nd order distribution)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

11. Further Classification (contd.)
Can the nth order distribution computations be simplified?
Yes. Under some simplifying assumptions:
Stationary (strict)
F(x;t) = F(x;t+τ)  all moments are time-invariant
Independence
As consequence of independence, we can define Renewal Process
Discrete time independent process {Xn|n=1,2,…} (X1, X2, .. are iid, non-negative rvs), e.g., repair/replacement after a failure. Markov process removes independence restriction.
Markov Process
Stochastic proc. {X(t) | t є T} is Markov if for any t0 < t1< … < tn< t, the conditional distribution
Stationary: E[x(t)] = E[x]  ensemble average. When the pdf or the CDF exhibits stationarity property, then, the process is said to strictly stationary. If only the first moment satisfies this property, then, the process is said to stationary in the mean etc.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

12. Markov Process
Mostly, we will deal with discrete state Markov process i.e., Markov chains
In some situations, a Markov process may also exhibit invariance wrt to the time origin, i.e. time-homogeneity
time-homogeneity does not imply stationarity. This also means that while conditional pdf may be stationary, the joint pdf may not be so.
Homogeneous Markov process  process is completely summarized by its current state (independent of how it reached this particular state).
Let, Y: time spent in a given state
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

13. Markov Process-Sojourn time
Y is also called the sojourn time
This result says that for a homogeneous discrete time Markov chain, sojourn time in a state follows EXP( ) distribution.
Semi-Markov process is one in which the sojourn time in state may not be EPX( ) distributed.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

14. Renewal Counting Process
Renewal counting process: # of renewals (repairs, replacements, arrivals) in time t: a continuous time process:
If time interval between two renewals follows EXP distribution, then  Poisson Process
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

15. Stationarity Properties
Strict sense Stationarity
Stationary in the mean  E[X(t)] = E[X]
In general, if
Then, a process is said to be wide-sense stationary
Strict-sense stationarity  wide-sense stationarity
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

16. Bernoulli Process
A set of Bernoulli sequences, {Yi|i=1,2,3,..}, Yi =1 or 0
{Yi} forms a Bernoulli Process. Often Yi’s are independent.
E[Yi] = p; E[Yi2 ] = p; Var[Yi] = p(1-p)
Define another stochastic process , {Sn|n=1,2,3,..}, where Sn = Y1 + Y2 +…+ Yn (i.e. Sn :sequence of partial sums)
Sn = Sn-1+ Yn (recursive form)
P[Sn = k| Sn-1= k] = P[Yn = 0] = (1-p) and,
P[Sn = k| Sn-1= k-1] = P[Yn = 1] = p
{Sn |n=1,2,3,..}, forms a Binomial process
P[Sn = k] =
{Yi} forms a discrete-time process.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

17. Binomial Process Properties
Viewing successes in a Bernoulli process as arrivals, then,
define discrete rv T1: # trials up to & including 1st success (arrival)
T1 : First order inter-arrival time and v has a Geometric distribution
P[T1 =i] = p(1-p)i-1, i=1,2,…; E[T1] = 1/p; Var[T1] = (1-p)/p2
Geometric Distribution  memory-less property.
Cond. pmf P[T1 =i| no success in the previous m trials ] = p
Since we treat arrival as success in {Sn}, occupancy time in state Sn is memory-less
Generalization to rth order inter-arrival time Tr: # trial trials up to including rth arrival.
Distribution for Tr : r-fold convolution of T1’s distribution.
Non-homogeneous Bernouli process.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

18. Poisson Process A continuous time, discrete state process.
N(t): no. of events occurring in time (0, t]. Events may be,
# of packets arriving at a router port
# of incoming telephone calls at a switch
# of jobs arriving at file/computer server
Number of failed components in time interval
Events occurs successively and that intervals between these successive events are iid rvs, each following EXP( )
λ: average arrival rate (1/ λ: average time between arrivals)
λ: average failure rate (1/ λ: average time between failures)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

19. Poisson Process (contd.)
N(t) forms a Poisson process provided:
N(0) = 0
Events within non-overlapping intervals are independent
In a very small interval h, only one event may occur (prob. p(h))
Letting, pn(t) = P[N(t)=n],
Hence, for a Poisson process, interval arrival times follow EXP( ) (memory-less) distribution. Such a Poisson process is non-stationary.
Mean = Var = λt ; What about E[N(t)/t], as t  infinity?
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

20. Merged Multiple Poisson Process Streams
Consider the system,
Proof: Using z-transform. Letting, α = λt, +
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

21. Decomposing a Poisson Process Stream
Decompose a Poisson process into multiple streams
N arrivals decomposed into {n1, n2, .., nk}; N= n1+n2, ..,+nk
Cond. pmf
Since,
The uncond. pmf +
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

22. Renewal Counting Process
Poisson process  EXP( ) distributed inter-arrival times.
What if the EXP( ) assumption is removed  renewal proc.
Renewal proc. : {Xi|i=1,2,…} (Xi’s are iid non-EXP rvs)
Xi : time gap between the occurrence of ith and (i+1)st event
Sk = X1 + X Xk  time to occurrence of the kth event.
N(t)- Renewal counting process is a discrete-state, continuous-time stochastic. N(t) denotes no. of renewals in the interval (0, t].
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

23. Renewal Counting Processes (contd.)
Sn t
For N(t), what is P(N(t) = n)?
nth renewal takes place at time t (account for the equality)
If the nth renewal occurs at time tn < t, then one or more renewals occur in the interval (tn < t]. tn
More arrivals possible
F(n+1) (t): prob(time taken for n-renewals + time for one more renewal)
= tn + t
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

24. Renewal Counting Process Expectation
Let, m(t) = E[N(t)]. Then, m(t) = mean no. of arrivals in time (0,t]. m(t) is called the renewal function.
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

25. Renewal Density Function
For example, if the renewal interval X is EXP(λ x), then
d(t) = λ , t >= 0 and m(t) = λ t , t >= 0.
P[N(t)=n] =
Fn(t) will turn out to be
e–λ t (λ t)n/n! i.e Poisson process pmf
n-stage Erlang
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

26. Availability Analysis
Availability: is defined is the ability of a system to provide the desired service.
If no repairs/replacements, Availability = Reliability.
If repairs are possible, then above def. is pessimistic.
MTBF = E[Di+Ti+1] = E[Ti+Di]=E[Xi]=MTTF+MTTR
MTBF
T D T D T D3 T D4 …….
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

27. Availability Analysis (contd.)
Two mutually exclusive situations:
System does not fail before time t  A(t) = R(t)
System fails, but the repair is completed before time t
Therefore, A(t) = sum of these two probabilities
renewal
Repair is completed with in this interval x t
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

28. Availability Expression
dA(x) : Incremental availability
dA(x) = Prob(that after renewal, life time is > (t-x) & that the renewal occurs in the interval (x,x+dx])
Repair is completed with in this interval x
x+dx t
Renewed life time >= (t-x)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

29. Availability Expression (contd.)
A(t) can also be expressed in the Laplace domain.
Since, R(t) = 1-W(t) or LR(s) = 1/s – LW(s) = 1/s –Lw(s)/s
What happens when t becomes very large?
However,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

30. Availability, MTTF and MTTR
Steady state availability A is:
for small values of s,
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

31. Availability Example Assuming EXP( ) density fn for g(t) and w(t)
Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University