Tanakorn Leesatapornwongsa Haryadi S. Gunawi

ISSTA ’15 2 node1node2node3 TCP/UDP

ISSTA ’15 3 node1node2node3 C B A Message processing order 1. Node 2 processes A 2. Node 3 processes B 3. Node 2 processes C

ISSTA ’15 4 node1node2node3 C B A Message processing order 1. Network delays A 2. Node 3 processes B 3. Node 2 processes C 4. Node 2 processes A

ISSTA ’15 5 node1node2node3 C B A Message processing order 1. Node 2 processes A 2. Node 3 processes B 3. Node 2 processes C 1. Node 3 processes B 2. Node 2 processes A 3. Node 2 processes C 1. Node 3 processes B 2. Node 2 processes C 3. Node 2 processes A

ISSTA ’15 6 Model Checking Server node1node3node2 AB C D A, BC, D A, B, C, D Interposition layer

ISSTA ’15 7 Model Checking Server node1node3node2 AB C D A, BC, D D, A, C, B Interposition layer A, B, D, C D, C, B, A...

ISSTA ’15 8  SAMC demo  Integration of SAMC  Real integration  Conclusion

ISSTA ’15 9  SAMC demo  Integration of SAMC  Real integration  Conclusion

ISSTA ’15 10  Demo program  Leader election  Find which node has the BIGGEST ID at the election time  Have only one leader!

ISSTA ’15 11 node1node2node3 V=1 V=2 Support = 2 Support = 3 V=3  When start up, it supports itself  Broadcast support  If receiving ID is smaller, do nothing  If bigger, change support  After support change, broadcast again  Stop when majority agree Leader = 3

ISSTA ’15 12  Run SAMC with 2 exploration algorithms  Brute force  Slow and inefficient  Local-message independent (LMI)  Fast white-box testing  Requires semantic information  Message semantic and system state

ISSTA ’15 13  Replaying buggy execution path again  Use execution path output to replay  Debug the execution until the desired step Very easy for developers to debug code and fix bugs

ISSTA ’15 14  Re-order all messages as we want  Report execution path and execution result  SAMC is semantic-aware  Supporting semantic-aware exploration algorithms  Fast model checking  SAMC with LMI can catch 2-leader bug in 3 executions!!!  Execution replay function

ISSTA ’15 15  SAMC demo  Integration of SAMC  Real integration  Conclusion

ISSTA ’15 16  Aspect-oriented programming for interposition layer  Written separately, not clutter with system code  Intercept at message sending method  Inform message semantic to the server via key-value pairs LeaderElectionAspect.aj

ISSTA ’15 17  Basic algorithms  Brute force, random, etc.  Extendable dynamic-partial order reduction (DPOR)  Implement LMI by adding application-specific logic to DPOR

ISSTA ’15 18  Extends abstract class WorkloadDriver  How to start / stop / reset the system  How to start workload we want to check

ISSTA ’15 19 Start Java processes that run SampleSys with given config files

ISSTA ’15 20  Extend abstract class SpecificationVerifier  Does system behave as specification? How many leader? Does everyone agree on one leader?

ISSTA ’15 21  SAMC demo  Integration of SAMC  Real integration  Conclusion

ISSTA ’15 22  Non-determinism  Network communication  Disk I/O  Machine crash / machine restart  Model check 5 versions  Reproduce 7 old bugs  Leader election and atomic broadcast protocol  Some require multiple crashes and reboots  Find 1 new bug

ISSTA ’15 23 Issue#ProtocolBrute forceRandomSemantic-Aware ZK-335ZAB ZK-790ZLE ZK-975ZLE ZK-1075ZLE ZK-1419ZLE ZK-1492ZLE ZK-1653ZAB ZAB = ZooKeeper atomic broadcast protocol ZLE = ZooKeeper leader election protocol Number of execution to run to reproduce old bugs

ISSTA ’15 24  SAMC demo  Integration of SAMC  Real integration  Conclusion

ISSTA ’15 25  Semantic awareness for fast model checking  AOP for interposition layer  SAMC server is extendable and comes with replay function  Able to integrate to real systems

ISSTA ’15 26  Timeout interposition  Catching performance bugs  Step-by-step replay function

27 ISSTA ’15 Code can be found at

ISSTA ’15 28 Model Checking Server node1node3node2 A B C D A, BC, D A, B, C, D

29 A A B B Alloc Req X1X1 X1X1 Some code here Some texts here Test B ddafdafa abcc metadata New text L ISSTA ’15

30  Come with extendable dynamic-partial order reduction (DPOR)  Implement LMI by adding application-specific logic to DPOR  Testers write workload driver  What workload to feed to the system  How to check the correctness of the system

ISSTA ’15 31  AOP for interposition layer  Written separately, not clutter with system code  Intercept at sending method  Forward message semantic to model checking server pointcut write(Sender sender, ElectionMessage msg) : call(public void Sender.write(ElectionMessage)) && this(sender) &&...; void around(Sender sender, ElectionMessage msg) : write(sender, msg) { LeaderElectionPacket packet = new LeaderElectionPacket(...); packet.addKeyValue(LeaderElectionPacket.EVENT_ID_KEY, hash(msg, sender.otherId)); packet.addKeyValue(LeaderElectionPacket.SOURCE_KEY, id); packet.addKeyValue(LeaderElectionPacket.DESTINATION_KEY, sender.otherId); packet.addKeyValue(LeaderElectionPacket.LEADER_KEY, msg.getRole()); packet.addKeyValue(LeaderElectionPacket.ROLE_KEY, msg.getLeader()); nodeSenderMap.put(packet.getId(), packet); msgSenderMap.put(packet.getId(), sender); try { modelCheckingServer.offerPacket(packet); } catch (RemoteException e) { e.printStackTrace(); }