1. Verisim: Formal Analysis of Network Simulations
Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky, Mahesh Viswanathan
University of Pennsylvania

2. Internetwork Routing Protocols
Internetwork routing protocols enable interoperability between physical networks.
Routing protocols for the Internet: RIP, OSPF, and BGP.
Routing protocols for packet radio: AODV and DSR.
Routing protocols and software have growing importance and complexity.

3. Current Analysis Techniques
Rigorous mathematical models and proofs – limited by complexity.
Testing.
Testbed – expensive.
Operational – risky and inconvenient.
Simulation.
Performance attributes: throughput, latency, reliability, etc.

4. Limitations of Performance Analysis of Simulations
Flaws may not be detected if no expected performance can be used for comparison.
When flaws are suspected, finer means of analysis are useful.
Some flaws do not manifest themselves as performance problems (e.g. most security gaps).

5. Verisim
Verisim provides support for “logical” analysis of network simulations.
This talk describes its architecture and logic.
We provide a series of experiments aimed at assessing the approach.

6. Monitoring and Checking (MaC) Framework
Script
Program
Monitor
Checker
Properties
Metatrace
Trace
Data NS
MEDL

7. Meta Event Description Language (MEDL)
Expresses properties of traces.
Extension of Linear Temporal Logic (LTL) with auxiliary variables.
More expressive than LTL.
Properties classified into (instantaneous) events and (enduring) conditions.
S Kannan, M Kim, I Lee, O Sokolsky, M Viswanathan 98

8. NS Network Simulator NS Trace P P P VINT Project Instrumented
Protocol Code
P : C++ P P P
Protocol
Agents
Scenario
Configuration
Parameters:
OTcl N N N
Network
Model
Topology:
OTcl
Traffic
Agents
src/sink
src/sink
src/sink
Traffic
Model:
OTcl
NS Trace
VINT Project

9. Verisim Properties: MEDL Instrumented Protocol: C++ NS Checker Trace
Metatrace
Scenario:
OTcl

10. Ad Hoc Networks
Routing for a wireless internetwork without the aid of a central base station.
Connections are low-bandwidth, lossy, and highly transient.
Unique routing assumptions:
Most routes are seldom used.
Bandwidth must be protected.

11. Illustration Part 1 of 2
Movement
Routing

12. Illustration Part 2 of 2
New
Routing

13. AODV Protocol
If a node S needs a route to a destination D and does not have one, it floods a route-request (RREQ) packet through the network.
Each recipient R of this RREQ keeps a return pointer.
R broadcasts the request to its neighbors if it is not D and does not have a route to D.
If R is D, or has a route to D, it responds with a route-reply (RREP) packet using the return pointers for S.
Perkins and Royer 99

14. Loop Freedom in AODV Routing loops are undesirable.
AODV uses sequence numbers to indicate freshness of link information.
Key Invariant: If next(n) = n’, then
seqno(n)  seqno(n’), and
if seqno(n) = seqno(n’), then hops(n) > hops(n’).
The invariant ensures that there are no loops.

15. Outline of Experiment Run a scenario of modest complexity.
Analyze it in Verisim using a list of 9 properties of AODV expressed in MEDL.
First pass: Repair First Bug (RFB).
Second pass: “tune” the MEDL formulas to avoid rerunning the simulation discovering bugs in the metatrace.
We instrumented simulation code for AODVv0 supplied by the CMU Monarch Project.

16. Experiment Scenario 1 2 3 4 5

17. Experiment Scenario 1 2 3 4 5

18. Experiment Scenario 1 2 3 4 5

19. Experiment Scenario 1 2 3 5 4

20. Experiment Scenario 1 5 2 3 4

21. Sample MEDL Alarm
alarm LoopInv[at][nxt][dst] =
sendrout[at][dst] when
((atnxt)  (atdst)  (nxtdst) 
(obs_nexthop[at][dst] == nxt) 
((obs_seqno[at][dst] > obs_seqno[nxt][dst]) 
((obs_seqno[at][dst] == obs_seqno[nxt][dst]) 
(obs_hopcont[at][dst] <= obs_hopcnt[at][dst]))))
This is the negation of the fundamental invariant ensuring
no loops in AODV:
If the next hop from node at toward destination dst is node nxt then the sequence number (for dst) of at is less than or equal to that of nxt or they are equal and the hop count (for dest) of at is less than or equal to that of nxt.

22. Repair First Bug Experiment
Let  be the MEDL property set.
Run the simulation to get a trace T.
Run the checker to get a metatrace T.
Repair the first bug in the metatrace to get new protocol code.
Rerun the simulation with the new protocol code to get a new trace U.
Rerun the checker to get a new metatrace U.
Continue until an empty metatrace is obtained.

23. RFB Experiment Statistics
Meta-
trace
Dest
Rep
Detect
RErr
Node
Loop
Env
Total
Alarms T 4 54 38 43
220 U
216 V 48 39 44
206 W 1

24. Tuning Experiment Statistics
Meta-
trace
Dest
Rep
Detect
RErr
Node
Loop
Env
Total
Alarms T 4 54 38 43
220 T
216 T 50
166 T 21
No new
simulation
traces.

25. Analysis of “Off-The-Shelf” (OTS) Simulations
The prior experiment involved fewer than 10,000 events, and we designed it to exercise key scenarios.
Can we do useful analysis with OTS performance simulations?
Aim: Verisim can be added with small modifications, run alongside OTS simulations, and find logical failures in a practical period of time.

26. Experiment
Took largest available OTS simulation for AODV and ran it to create a trace.
Simulation from Monarch uses 50 mobile nodes on 1500x300m grid moving 20 m/s.
5220 seconds (1.5 hours) to complete simulation.
6,446,316 events.
Naïve effort: ran MEDL with MonSeqNo test on all nodes (2500 relations) using 550Mhz dual processor machine with 1GB of memory.
Aborted the experiment after 4 days; estimate 100 days to complete this analysis.

27. Optimizations For Network Routing Simulations
Population Abstraction: test the property for a subset of the routers.
Packet Type Abstraction: prune the trace to include only relevant events.

28. Sample Experiment Results for MonSeqNo
Population Abstraction for 5 nodes.
Trace size: 6,446,316 events
Property size: 14,543 bytes
Time: 51,054 seconds
Rate: 0.54 micro seconds per event per property.
Population and Packet Type Abstractions.
Trace size: 6812 events
Time: 51 seconds (found failures)
Rate: 0.51 micro seconds per event per property.

29. Sample Experiment Results for LoopInv
Population Abstraction for 5 nodes.
Trace size: 69,411 events
Property size: 75,508 bytes
Time: 8064 seconds
Rate: 1.54 micro seconds per event per property.
Population and Packet Type Abstractions.
Trace size: 48,735 events
Time: 5912 seconds (found failures)
Rate: 1.61 micro seconds per event per property.

30. Related Work
Communication-based test generation systems. STRESS and Verisoft.
Test oracles based on formulas or formal operational specifications. GIL and TETRA.
Formal instance verification of routing protocols. SPIN/PITHIA for PNNI.
Simulations based on logical specifications. MTSim and Maude.
Network monitoring systems. Emerald.

31. Summary Verisim integrates simulation and logical trace analysis.
The combination provides a more flexible approach to analyzing network simulations for safety properties.
It is able to find bugs in existing simulator studies.
Its flexibility can be exploited to improve debugging turn-around time.