Eager Markov Chains Parosh Aziz Abdulla Noomene Ben Henda Richard Mayr Sven Sandberg TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A A

Informationsteknologi Institutionen för informationsteknologi | Outline Introduction Expectation Problem Algorithm Scheme Termination Conditions Subclasses of Markov Chains  Examples Conclusion

Informationsteknologi Institutionen för informationsteknologi | Introduction Model: Infinite-state Markov chains  Used to model programs with unreliable channels, randomized algorithms… Interest: Conditional expectations  Expected execution time of a program  Expected resource usage of a program

Informationsteknologi Institutionen för informationsteknologi | Introduction Infinite-state Markov chain  Infinite set of states  Target set  Probability distributions Example

Informationsteknologi Institutionen för informationsteknologi | Introduction Reward function  Defined over paths reaching the target set Example

Informationsteknologi Institutionen för informationsteknologi | Expectation Problem Instance  A Markov chain  A reward function Task  Compute/approximate the conditional expectation of the reward function

Informationsteknologi Institutionen för informationsteknologi | Expectation Problem Example:  The weighted sum  The reachability probability  The conditional expectation *4+0.1*(-5)= = /0.9=3

Informationsteknologi Institutionen för informationsteknologi | Expectation Problem Remark  Problem in general studied for finite-state Markov chains Contribution  Algorithm scheme to compute it for infinite- state Markov chains  Sufficient conditions for termination

Informationsteknologi Institutionen för informationsteknologi | Algorithm Scheme At each iteration n  Compute paths up to depth n  Consider only those ending in the target set  Update the expectation accordingly Path Exploration

Informationsteknologi Institutionen för informationsteknologi | Algorithm Scheme Correctness  The algorithm computes/approximates the correct value Termination  Not guaranteed: lower-bounds but no upper- bounds

Informationsteknologi Institutionen för informationsteknologi | Termination Conditions Exponentially bounded reward function  The intuition: limit on the growth of the reward functions  Remark: The limit is reasonable: for example polynomial functions are exponentially bounded

Informationsteknologi Institutionen för informationsteknologi | Termination Conditions 0 The abs of the reward Bound on the reward

Informationsteknologi Institutionen för informationsteknologi | Termination Conditions Eager Markov chain  The intuition: Long paths contribute less in the expectation value  Remark: Reasonable: for example PLCS, PVASS, NTM induce all eager Markov chains

Informationsteknologi Institutionen för informationsteknologi | Termination Conditions 0 1 Prob. of reaching the target in more than n steps Bound on the probability

Informationsteknologi Institutionen för informationsteknologi | Termination Conditions Pf Ws Ce

Informationsteknologi Institutionen för informationsteknologi | Subclasses of Markov Chains Eager Markov chains Markov chains with finite eager attractor Markov chains with the bounded coarseness property NTM PVASS PLCS

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor Attractor:  Almost surely reached from every state Finite eager attractor:  Almost surely reached  Unlikely to stay ”too long” outside of it A EA

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor EA 0 1 b Prob. to return in More than n steps

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor Finite eager attractor implies eager Markov chain??  Reminder: Eager Markov chain: Prob. of reaching the target in more than n steps

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor FEA Paths of length n that visit the attractor t times

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor Proof idea: identify 2 sets of paths  Paths that visit the attractor often without going to the target set:  Paths that visit the attractor rarely without going the target set:

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor Paths visiting the attractor rarely: t less than n/c FEA Pr_n

Informationsteknologi Institutionen för informationsteknologi | Finite Eager Attractor Paths visiting the attractor often: t greater than n/c FEA PtPl Po_n

Informationsteknologi Institutionen för informationsteknologi | Probabilistic Lossy Channel Systems (PLCS) Motivation:  Finite-state processes communicating through unbounded and unreliable channels  Widely used to model systems with unreliable channels (link protocol)

Informationsteknologi Institutionen för informationsteknologi | PLCS ab b Send c!a ab Receive c?b a c?b q0 q3q2 q1 nop c!a c!b aba Channel c nop

Informationsteknologi Institutionen för informationsteknologi | PLCS c?b q0 q3q2 q1 nop c!a c!b aba Channel c nop ab b Loss b b a a

Informationsteknologi Institutionen för informationsteknologi | PLCS Configuration  Control location  Content of the channel Example  [q3,”aba”] c?b q0 q3q2 q1 nop c!a c!b aba Channel c nop

Informationsteknologi Institutionen för informationsteknologi | PLCS A PLCS induces a Markov chain:  States: Configurations  Transitions: Loss steps combined with discrete steps

Informationsteknologi Institutionen för informationsteknologi | PLCS Example:  [q1,”abb”] [q2,”a”]  By losing one of the messages ”b” and firing the marked step. Probability:  P=Ploss*2/3 c?b q0 q3q2 q1 nop c!a c!b aba Channel c nop

Informationsteknologi Institutionen för informationsteknologi | PLCS Result: Each PLCS induces a Markov chain with finite eager attractor.  Proof hint: When the size of the channels is big enough, it is more likely (with a probability greater than ½) to lose a message.

Informationsteknologi Institutionen för informationsteknologi | Bounded Coarseness The probability of reaching the target within K steps is bounded from below by a constant b.

Informationsteknologi Institutionen för informationsteknologi | Bounded Coarseness Boundedly coarse Markov chain implies eager Markov chain??  Reminder: Eager Markov chain: Prob. of reaching the target in more than n steps

Informationsteknologi Institutionen för informationsteknologi | Bounded Coarseness Prob. Reach. Within K steps KnK steps 2K PnP2 Pn:Prob. of avoiding the target in nK steps P1

Informationsteknologi Institutionen för informationsteknologi | Probabilistic Vector Addition Systems with states (PVASS) Motivation:  PVASS are generalizations of Petri-nets.  Widely used to model parallel processes, mutual exclusion program…

Informationsteknologi Institutionen för informationsteknologi | PVASS Configuration  Control location  Values of the variables x and y Example:  [q1,x=2,y=0] q0 q3q2 q1 nop --x --y ++x ++y x

Informationsteknologi Institutionen för informationsteknologi | PVASS A PVASS induces a Markov chain:  States: Configurations  Transitions: discrete steps

Informationsteknologi Institutionen för informationsteknologi | PVASS Example:  [q1,1,1] [q2,1,0]  By taking the marked step. Probability:  P=2/3 q0 q3q2 q1 nop --x --y ++x ++y x

Informationsteknologi Institutionen för informationsteknologi | PVASS Result: Each PVASS induces a Markov chain which has the bounded coarseness property.

Informationsteknologi Institutionen för informationsteknologi | Noisy Turing Machines (NTM) Motivation:  They are Turing Machines augmented with a noise parameter.  Used to model systems operating in ”hostile” environment

Informationsteknologi Institutionen för informationsteknologi | NTM Fully described by a Turing Machine and a noise parameter. q1 q3q2 q4 a/bb b # # RR R RR S S ab#b#aab

Informationsteknologi Institutionen för informationsteknologi | NTM q1 q3q2 q4 a/bb b # # RR R RR S S ab#b#aab Discret Step ab#b#aab bb#b#aab

Informationsteknologi Institutionen för informationsteknologi | NTM q1 q3q2 q4 a/bb b # # RR R RR S S ab#b#aab Noise Step ab#b#aab #b#b#aab

Informationsteknologi Institutionen för informationsteknologi | NTM Result: Each NTM induces a Markov chain which has the bounded coarseness property.

Informationsteknologi Institutionen för informationsteknologi | Conclusion Summary:  Algorithm scheme for approximating expectations of reward functions  Sufficient conditions to guarantee termination:  Exponentially bounded reward function  Eager Markov chains

Informationsteknologi Institutionen för informationsteknologi | Conclusion Direction for future work  Extending the result to Markov decision processes and stochastic games  Find more concrete applications

Thank you

Informationsteknologi Institutionen för informationsteknologi | PVASS Order on configurations: <=  Same control locations  Ordered values of the variables Example:  [q0,3,4] <= [q0,3,5] q0 q3q2 q1 nop --x --y ++x ++y x

Informationsteknologi Institutionen för informationsteknologi | PVASS Probability of each step > 1/10 Boundedly coarse: parameters K and 1/10^K q0 q3q2 q1 nop --x --y ++x ++y x Target set K iterations