1. Randomized Algorithms Markov Chains and Random Walks
Speaker: Chuang-Chieh Lin
Advisor: Professor Maw-Shang Chang
National Chung Cheng University
2018/11/29

2. References Professor S. C. Tsai’s slides.
Randomized Algorithms, Rajeev Motwani and Prabhakar Raghavan.
Probability and Computing - Randomized Algorithms and Probabilistic Analysis, Michael Mitzenmacher and Eli Upfal.
Wikipedia: Markov Chains
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

3. Outline Introduction to Markov chains Classification of states
Stationary distribution
Random walks on undirected graphs
Connectivity problem
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

4. Introduction to Markov Chains
Markov chains provide a simple but powerful framework for modeling random processes.
Markov chains can be used to analyze simple randomized algorithms applying random walks.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

5. Introduction to Markov Chains (cont’d)e n i t o : F A s c h a p r X = f ( ) ; 2 T g l d m v b . I u , y 1 w O - H
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

6. Introduction to Markov Chains (cont’d)F I f X n = i , t h e p r o c s a d b m . D P [ + 1 j ; : ] l y
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

7. Introduction to Markov Chains (cont’d), P ; j = r [ X n + 1 ] f o l e d S u c p k w M v .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

8. Formal definitions.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

9. Markov property
In probability theory, a stochastic process has the Markov property if the conditional probability distribution of future states of the process, given the present state and all past states, depends only upon the current state and not on any past states.
Mathematically, if X(t), t > 0, is a stochastic process, the Markov property states that P r [ X ( t + h ) = y j s x ; 8 ]
>
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

10. Markov chain
In mathematics, a Markov chain, named after Andrey Markov, is a discrete-time stochastic process with the Markov property.
June 14, 1856 – July 20, 1922
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

11. Homogeneous
Markov processes are typically termed (time-) homogeneous if P r [ X ( t + h ) = y j x ] ; 8
>
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

12. Transition matrix . Pi,1 1 F P ¸ . = 1 Pi,2 i 2 Pi,j j; j . = 1
Pi,2 i 2
Pi,j . j
transition matrix
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

13. Transition probabilityh e m - s t p r a n i o b l y P ; j f M k v c d , g u w . = [ X + ] : C q 1
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

14. Chapman-Kolmogorov equation
Generalize the previous result, we have Chapman-Kolmogorov equation as follows. P n + m i ; j = X k :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

15. Chapman-Kolmogorov equation (cont’d)f : B y t h e d n i - s p a b l , + m ; j = X k [ ] M v . W w
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

16. Recall: Transition matrix
Pi,1 1 F P i ; j . = 1
Pi,2 i 2
Pi,j . j
transition matrix
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

17. Recall: Homogeneous
Markov processes are typically termed (time-) homogeneous if P r [ X ( t + h ) = y j x ] ; 8
>
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

18. For example, F L e t P d o h m a r i x f - s p b l , C K g v q u = . (n ) d o h m a r i x f - s p b l ; j , C K g v q u = .
For example,
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

19. 1 2 3 2 3 1
0.3
0.4
0.5
0.6
0.8
0.2
0.7 1 2 3
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

20. Classification of states
A first step in analyzing the long-term behavior of a Markov chain is to classify its states.
In the case of a finite Markov chain, this is equivalent to analyzing the connected connectivity structure of the directed graph representing the Markov chain.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

21. Basic definitions F S t a e j i s d o b c l f r m P > ¸ . W y u h (. W y u h ( $ ) T M k v w
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

22. Basic definitions (cont’d)L e t r i ; j d n o h p b a l y s g , c u m . T = P [ X f 1 6 ] : S
<
transient：短暫的
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

23. Basic definitions (cont’d)j i n d c r m M k o v h p f x g
> 1 u P [ X + = ] l b y . ; , w
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

24. Basic definitions (cont’d)w i h p r o d 1 b c . W n y ; j x m f S v = P u - ,
< O l g
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

25. Null recurrent?
For example, consider a Markov chain whose states are the positive integers.
From state i, the probability of going to state i+1 is i / (i+1).
With probability 1 / (i+1), the chain returns to state 1.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

26. Null recurrent? (cont’d)
Starting at state 1, the probability of not having returned to state 1 within the first t steps is thus
Hence the probability of never returning to state 1 from state 1 is 0, then we have state 1 is recurrent. It follows that t Q j = 1 + . r t 1 ; = + ( ) .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

27. Null recurrent? (cont’d)
However, the expected number of steps until the first return to state 1 from state 1 is
which is unbounded.
Thus this Markov chain has null recurrent states. h 1 ; = P t r + ,
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

28. This is necessary for null recurrent states to exist.
In the foregoing example, the Markov chain had an infinite number of states.
This is necessary for null recurrent states to exist.
Yet for a finite Markov chain, we have the following lemma.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

29. Lemma 1 In a finite Markov chain:
At least one state is recurrent; and
All recurrent states are positive recurrent.
We omit the proof here, though it is not hard.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

30. Recall that… F S t a e i s d o b r c u n f P = , < . ¸ 1 ;
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

31. Proposition 1 F S t a e i s r c u n f P 1 ¢ = : < I T h , x p d m b1 ; = : I T h , x p d m b o v l .
<
Proof of this proposition is a little bit complicated, so we omit it here.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

32. Proof of the second statementL e t N i b h n u m r o f v s a l , E [ ] = P ; . G d V y g I O 1 c T p j
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

33. Proof of the second statement (cont’d)w i s ( . , V o c u ) x a m n y I g v l d T -
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

34. Proof of the second statement (cont’d)i n c e s t r a , h x y j u f o m P ;
> b l . I T w v V p [ ]
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

35. Proof of the second statement (cont’d)u n o i , s p b l y P [ V ]
> w v d . H g x m E N j = 1 ; +
< :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

36. Corollary 1
If state i is recurrent, and state i communicates with state j, then state j is recurrent.
Proof:
Exercise.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

37. Stationary Distributione n i t o : A s a r y d b u ( l c q m ) f M k v h p = P R - x .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

38. Computing the stationary distribution of a finite Markov chain
One way to compute the stationary distribution of a finite Markov chain is to solve the system of linear equations
This is particularly useful if one is given a specific chain. = P :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

39. For example, given the transition matrix
we have five equations for the four unknowns 0, 1, 2, and 3 given by and P 3 i = 1 . = P
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

40. Another technique
Another useful technique is to study the cut-sets of the Markov chain.
For any state i of the chain, or n P j = ; i n P j 6 = i ; .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

41. This observation can be generalized to sets of states as follows.
That is, in the stationary distribution the probability that a chain leaves a state equals the probability that it enters the state.
This observation can be generalized to sets of states as follows.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

42. Theorem
Let S be a set of states of a finite, irreducible, aperiodic Markov chain. In the stationary distribution, the probability that the chain leaves the set S equals the probability that it enters S.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

43. In other words, if C is a cut-set in the graph representation of the Markov chain, then in the stationary distribution the probability of crossing the cut-set in one direction is equal to the probability of crossing the cut-set in the other direction.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

44. That is, in the stationary distribution the probability that a chain leaves a state equals the probability that it enters the state.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

45. For example, 1 The transition matrix is p 1  p 1  q q 2018/11/291
1  q q
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

46. 1 This Markov chain is often used to represent bursty behavior.1
1  q q
This Markov chain is often used to represent bursty behavior.
For example, when bits are corrupted in transmissions they are often corrupted in large blocks, since errors are often caused by an external phenomenon of some duration.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

47. p
1  p 1
1  q q
In this setting, begin in state 0 after t steps represents that the tth bit was sent successfully, while being in state 1 represents that the bit was corrupted.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

48. p
1  p 1
1  q q
Blocks of successfully sent bits and corrupted bits both have lengths that follow a geometric distribution.
When p and q are small, state changes are rare, and the bursty behavior is modeled.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

49. p
1  p 1
1  q q
Solving corresponds to solving the following system of three equations: = P ( 1 p ) + q = ; :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

50. Using the cut-set formulation,
we have that in the stationary distribution the probability of leaving state 0 must equal the probability of entering state 0. p = 1 q :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

51. Again, now using 0 + 1 = 1 yields 0 = q/(p+q) and 1 = p/(p+q).
For example, with the natural parameters p = and q = 0.1, in the stationary distribution more than 95% of the bits are received successfully.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

52. Theorem 1
positive recurrent or aperiodic
Any finite, irreducible, and ergodic Markov chain has the following properties: F T h e c a i n s u q t o r y d b = ( ; 1 : ) l j , m ! P x p - f .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

53. Proof of Theorem 1 Please refer to the textbook for the details.
page 168170 in [MU05]
We omit the proof here.
Yet we can still eye on the following lemma.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

54. Lemma 2 For any irreducible, ergodic Markov chain and for any state i,
We explain instead of proving the lemma as follows. l i m t ! 1 P ; e x s a n d = h :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

55. Informal justifications…
The expected time between visits to i is hi,i and therefore state i is visited 1/hi,i of the time.
Thus the limit of , which represents the probability that, a state chosen far in the future is at state i when the chain starts at state i, must be 1/hi,i.
Since the limit exists, we can show that P t i ; l i m t ! 1 P j ; = h :
We omit the detail here for simplicity.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

56. Markov Chains and Random Walks on Undirected Graphs…
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

57. Random walks on undirected graphs
A random walk on G is a Markov chain defined by the sequence of moves of a particle between vertices of G.
In this process, the place of the particle at a given time step is the state of the system.
If the particle is at vertex i and if i has d(i) outgoing edges, then the probability that the particle follows edge (i, j) and moves to neighbor j is 1/d(i).
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

58. Lemma 3
A random walk on an undirected graph G is aperiodic iff G is not bipartite.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

59. Proof of Lemma 3
A graph is bipartite iff it does not have any odd cycle.
In an undirected graph, there is always a path of length 2 from a vertex to itself.
Thus, if the graph is bipartite then the random walk is periodic (d = 2).
If not bipartite, then it has an odd cycle and gcd(2,odd-number) = 1. Thus, the Markov chain is aperiodic. ) (
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

60. Some technical restrictions
Due to some technical reasons, for the remainder of our discussion, we assume that the graph G that we will discuss is not bipartite.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

61. Something needs to be clarified…
ergodic
aperiodic
A random walk on a finite, undirected, connected, and non-bipartite graph G satisfies the conditions of Theorem 1.
A Markov chain that is finite, irreducible, and ergodic.
Hence the random walk converges to a stationary distribution. (Due to Theorem 1.)
The following Theorem shows that this distribution depends only on the degree sequence of the graph.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

62. Theorem 2 A r a n d o m w l k G c v e g s t i y b u ¹ ¼ , h = ( ) 2 j:
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

63. Proof of Theorem 2 ¢ S i n c e P d ( ) = j E , ¼ 1 . T h a t s ¹ r b uv 2 V d ( ) = j E , 1 . T h a t s r b u o L p l y m x N g f
thus the theorem follows.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

64. Corollary 2 F o r a n y v e t x u i G , h = : R c l s p m b f . S w ;j E d ( ) : R c l s p m b f . S w
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

65. Lemma 4 I f ( u ; v ) 2 E , t h e n · j . 2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

66. Proof of Lemma 4 R e c a l t h N ( u ) d n o s i g b r f v p G . S = ¢2 j E = ; 1 P w + , T
<
w’s u .
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

67. The Covering Time Definition:
The cover time of a graph G = (V, E) is the maximum over all v  V of the expected time to visit all of the nodes in the graph by a random walk starting from v.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

68. Lemma 5 The cover time of G = (V, E) is bounded by 4|V|  |E|.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

69. Proof of Lemma 5
Choose a spanning tree of G. Then there exists a cyclic (Eulerian) tour on this tree, where each edge is traversed once in each direction, which can be found by doing a DFS.
Let v0,v1,…,v2|V|2= v0 be the sequence of vertices in the tour, starting from v0.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

70. Proof of Lemma 5 (cont’d)
Clearly the expected time to go through the vertices in the tour is an upper bound on the cover time.
Hence the cover time is bounded above by 2 j V 3 X i = h v ; + 1
< ( ) E 4 :
from Lemma 4
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

71. Application: s -t connectivity
Suppose we are given an undirected graph G (V, E) and two vertices s and t in G.
Let n = | V | and m = | E |.
We want to determine if there is a path connecting s and t.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

72. Application: s -t connectivity (cont’d)
This is easily done in linear time using a standard breadth-first search or depth-first search.
However, such algorithms require (n) space.
The following randomized algorithm works only O(log n) bits of memory.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

73. Application: s -t connectivity (cont’d)g r h m : I p u G a d w O Y N F S k f . 4 3 , E
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

74. Application: s -t connectivity (cont’d)
We have assumed that G has no bipartite connected component.
The results can be made to apply to bipartite graphs with some additional technical work.
Let us consider the following theorem.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

75. Theorem 3 T h e s - t C o n c i v y A l g r m u a w p b d f . 1 2
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

76. Proof of Theorem 3
The algorithm gives correct answer, when G has no s-t path.
If G has an s-t path, the algorithm errs if it does not find the path in 4n3 steps.
Now we consider the case that G has an s-t path.
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

77. Proof of Theorem 3 (cont’d)
The expected time to reach t from s is bounded by the cover time, which is at most 4| V |  | E | < 2n3.
| E |  n(n  1)/2
Let a random variable X denote the number of steps needed for the s-t Connectivity Algorithm.
By Markov’s inequality, P r [ X
> 4 n 3 ] E
< 2 = 1 :
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

78. Proof of Theorem 3 (cont’d)
The algorithm must keep track of its current position, which takes O(log n) bits, as well as the number of steps taken in the random walk, which also takes only O(log n) bits.
Since we count up only 4n3, which requires log(4n3) = O(log n) bits
2018/11/29
Computation Theory Lab, CSIE, CCU, Taiwan

79. Thank you.