Distributed Markov Chains P S Thiagarajan School of Computing, National University of Singapore Joint work with Madhavan Mukund, Sumit K Jha and Ratul Saha

Probabilistic dynamical systems Rich variety and theories of probabilistic dynamical systems – Markov chains, Markov Decision Processes (MDPs), Dynamic Bayesian networks Many applications Size of the model is a bottleneck – Can we exploit concurrency theory? We explore this in the setting of Markov chains.

Our proposal A set of interacting sequential systems. – Synchronize on common actions. a a

Our proposal A set of interacting sequential systems. – Synchronize on common actions. a

Our proposal A set of interacting sequential systems. – Synchronize on common actions. a

Our proposal A set of interacting sequential systems. – Synchronize on common actions. – This leads a joint probabilistic move by the participating agents. a, 0.8 a, 0.2

Our proposal A set of interacting sequential systems. – Synchronize on common actions. – This leads a joint probabilistic move by the participating agents. a, 0.8 a, 0.2

Our proposal A set of interacting sequential systems. – Synchronize on common actions. – This leads a joint probabilistic move by the participating agents. a, 0.8 a, 0.2

Our proposal A set of interacting sequential systems. – Synchronize on common actions. – This leads a joint probabilistic move by the participating agents. a, 0.8 a, 0.2

Our proposal A set of interacting sequential systems. – Synchronize on common actions. – This leads a joint probabilistic move by the participating agents. – More than two agents can take part in a synchronization. – More than two probabilistic outcomes possible. – There can also be just one agent taking part in a synchronization. Viewed as an internal probabilistic move (like in a Markov chain) by the agent.

Our proposal This type of a system has been explored by Pighizzini et.al (“Probabilistic asynchronous automata”; 1996) – Language-theoretic study. Our key idea: – impose a “determinacy of communications” restriction. – Study formal verification problems using partial order based methods. We study here just one simple verification method.

Some notations

{a} Determinacy of communications. s s’ s’’ i {a}

Determinacy of communications. s s’ s’’ i j

{a} Determinacy of communications. s s’ s’’ i j loc(a) = {i, j} (s, s’), (s, s’’)  en a a a a

{a} Not allowed! s s’ i j s’’ k act(s) will have more than one action.

Some notations

Example – Two players each toss a fair coin – If the outcome is the same, they toss again – If the outcomes are different, the one who tosses Heads wins

Example Two component DMC

Interleaved semantics. Coin tosses are local actions, deciding a winner is synchronized action

Goal We wish to analyze the behavior of a DMC in terms of its interleaved semantics. Follow the Markov chain route. – Construct the path space. The set of infinite paths from the initial state. Basic cylinder: a set of infinite paths with a common finite prefix. Close under countable unions and complements.

The transition system view /5 3/ /5 3/ B Pr(B) = 1  2/5  1  1 = 2/5 B – The set of all paths that have the prefix

Concurrency Events can occur independent of each other. Interleaved runs can be (concurrency) equivalent. We use Mazurkiewicz trace theory to group together equivalent runs: trace paths. Infinite trace paths do not suffice. We work with maximal infinite trace paths.

(in 1, in 2 ) (T 1, in 2 )(in 1, H 2 ) (in 1, T 2 ) (H 1, in 2 ) t1, 0.5 t2, 0.5 h1, 0.5 h2, 0.5 (H 1, H 2 )(T 1, H 2 )(H 1, T 2 )(T 1, T 2 ) W1, L2 w1 l2 w1 L1, W2

The trace space A basic trace cylinder is the one generated by a finite trace Construct the  -algebra by closing under countable unions and complements. We must construct a probability measure over this  -algebra. For a basic trace cylinder we want its probability to be the product of the probabilities of all the events in the trace.

(in 1, in 2 ) (T 1, in 2 )(in 1, H 2 ) (in 1, T 2 ) (H 1, in 2 ) t1, 0.5 t2, 0.5 h1, 0.5 h2, 0.5 (H 1, H 2 )(T 1, H 2 )(H 1, T 2 )(T 1, T 2 ) W1, L2 w1 l2 w1 L1, W2 B Pr(B) = 0.5  0.5 = 0.25

The probability measure over the trace space. But proving that this extends to a unique probability measure over the whole  -algebra is hard. To solve this problem : – Define a Markov chain semantics for a DMC. – Construct a bijection between the maximal traces of the interleaved semantics and the infinite paths of the Markov chain semantics. Using Foata normal form – Transport the probability measure over the path space to the trace space.

The Markov chain semantics.

Markov chain semantics

Probabilistic Product Bounded LTL

PBLTL over interleaved runs

Statistical model checking…

SPRT based model checking

Case study

Case study…

Distributed leader election protocol [Itai-Rodeh]

Case study

Case study… Dining Philosophers Problem

Other examples Other PRISM case studies of randomized distributed algorithms – consensus protocols, gossip protocols… – Need to “translate" shared variables using a protocol Probabilistic choices in typical randomized protocols are local DMC model allows communication to influence probabilistic choices – We have not exploited this yet! – Not represented in standard PRISM benchmarks

Summary and future work The interplay between concurrency and probabilistic dynamics is subtle and challenging. But concurrency theory may offer new tools for factorizing stochastic dynamics. – Earlier work on probabilistic event structures [Katoen et al, Abbes et al, Varacca et al] also attempt to impose probabilities on concurrent structures. – Our work shows that formal verification as the goal offers valuable guidelines Need to develop other model checking methods for DMCs. – Finite unfoldings – Stubborn sets for PCTL like specifications.