Efficient Decentralized Monitoring of Safety in Distributed Systems Koushik Sen Abhay Vardhan Gul Agha Grigore Rosu University of Illinois at Urbana-Champaign, USA

Software Reliability Software Validation Rigorous and Complete Methods Model Checking Theorem Proving Infeasible for large-scale open distributed systems Non-determinism and Asynchrony Testing Widely used Ad-Hoc Good Test Coverage Required Runtime Monitoring Adds rigor to Testing 11/12/2018

Centralized Monitoring Approach Monitoring – Use Formal Methods in Testing Synthesize light-weight Monitors from Specification Automata, Rewriting-based Monitors, State machines Instrument code to insert monitors Execute instrumented code Distributed System Monitoring Global state is distributed For every state update send state to a central monitor Central monitor assembles them to form consistent execution traces (Vector Clocks) Sequence of global states Monitor execution traces 11/12/2018

An Example Mobile node a requests certain value from node b b computes the value and sends it to a Property: no node receives a value from another node to which it had not sent a request 11/12/2018

Centralized Monitoring Example “If a receives a value from b then b calculated the value after receiving request from a” valRcv → (valComputed  valReq) valReq valReq valComputed  valReq valRcv → (valComputed  valReq) (valComputed  valReq) Monitor b valComputed a valReq valRcv 11/12/2018

Decentralized Monitoring Approach “If a receives a value from b then b calculated the value after receiving request from a” valRcv → @b((valComputed  @a(valReq))) valComputed  @a(valReq) @a(valReq) (valComputed  @a(valReq)) b valComputed a valReq valRcv valReq valRcv → @b((valComputed  @a(valReq))) 11/12/2018

Past time Distributed Temporal Logic (pt-DTL) Past Time Linear Temporal Logic [Pnueli] Extended with a Operator from epistemic logic (@) [Aumann76][Meenakshi et al. 00] Properties with respect to a process, say p Interpreted over sequence of knowledge that p has about global state 11/12/2018

Remote Formulas in pt-DTL @a F at process b @ makes remote formula F at process a local to process b “Alarm at process b implies that there was a fire at a” alarm → @afire a formula with respect to process b 11/12/2018

Remote Expressions in pt-DTL Remote expressions – arbitrary expressions related to the state of a remote process Propositions constructed from remote and local expressions “If my alarm is set then eventually in past difference between my temperature and temperature at process b exceeded the allowed value” alarm → ((myTemp - @btemp) > allowed) 11/12/2018

Safety in Airplane Landing “ If my airplane is landing then the runway that the airport has allocated matches the one that I am planning to use” landing → (runway = @airportallocRunway) 11/12/2018

Leader Election Example “If a leader is elected then if the current process is a leader then, at its knowledge, none of the other processes is a leader” elected → (state=leader → /\i≠j(@j(state ≠ leader))) 11/12/2018

pt-DTL syntax Fi ::= true | false | P(Ei) | : Fi | Fi Æ Fi propositional | ¯ Fi | ¡ Fi | Fi | Fi S Fi temporal | @jFj epistemic Ei ::= c | vi 2 Vi | f(Ei) functional | @jEj epistemic 11/12/2018

Interpretation of @jEj at process i m4 m1 m2 p2 @ 1(x=9) m3 p1 x=7 x=9 11/12/2018

Monitoring Algorithm Requirements Should be fast so that online monitoring is possible Little memory overhead Additional messages sent should be minimal; ideally zero 11/12/2018

KnowledgeVector Let KV be a vector one entry for each process appearing in formula KV[j] denotes entry for process j KV[j].seq is the sequence number of last event seen at process j KV[j].values stores values of j-expressions and j-formulae 11/12/2018

Monitoring using KnowledgeVector Maintain KnowledgeVector about global state at each process Attach KnowledgeVector with outgoing messages Update KnowledgeVector with incoming messages At each process monitor local KnowledgeVector 11/12/2018

KnowledgeVector Algorithm [internal event]: (at process i) store eval(Ei,si) and eval(Fi,si) for each @iEi and @iFi in KVi[i].values [send m]: KVi[i].seq à KVi[i].seq + 1. Send KVi with m as KVm [receive m]: for all j, if KVm[j].seq > KVi[j].seq then KVi[j].seq à KVm[j].seq KVi[j].values à KVm[j].value 11/12/2018

Example p3 p2 p1 ¡(Y ¸ @1X) at p2 Y=7 Y=3 violation X=5 X=9 X=6 5 2 6 5 2 6 2 6 p3 5 2 6 2 6 2 6 p2 Y=7 Y=3 violation 5 9 1 9 1 6 2 6 p1 X=5 X=9 X=6 KV[1].seq ¡(Y ¸ @1X) at p2 KV[1].values 11/12/2018

DIANA Architecture pt-DTL Monitor 11/12/2018

Conclusion pt-DTL can express interesting and useful safety properties of distributed systems Decentralized Technique to effectively verify Distributed Systems at runtime No extra message over-head for monitoring KnowledgeVector as monitors 11/12/2018