Verisim: Formal Analysis of Network Simulations Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky, Mahesh Viswanathan University of Pennsylvania

Internetwork Routing Protocols Internetwork routing protocols enable interoperability between physical networks. Routing protocols for the Internet: RIP, OSPF, and BGP. Routing protocols for packet radio: AODV and DSR. Routing protocols and software have growing importance and complexity.

Current Analysis Techniques Rigorous mathematical models and proofs – limited by complexity. Testing. Testbed – expensive. Operational – risky and inconvenient. Simulation. Performance attributes: throughput, latency, reliability, etc.

Limitations of Performance Analysis of Simulations Flaws may not be detected if no expected performance can be used for comparison. When flaws are suspected, finer means of analysis are useful. Some flaws do not manifest themselves as performance problems (e.g. most security gaps).

Verisim Verisim provides support for “logical” analysis of network simulations. This talk describes its architecture and logic. We provide a series of experiments aimed at assessing the approach.

Monitoring and Checking (MaC) Framework Script Program Monitor Checker Properties Metatrace Trace Data NS MEDL

Meta Event Description Language (MEDL) Expresses properties of traces. Extension of Linear Temporal Logic (LTL) with auxiliary variables. More expressive than LTL. Properties classified into (instantaneous) events and (enduring) conditions. S Kannan, M Kim, I Lee, O Sokolsky, M Viswanathan 98

NS Network Simulator NS Trace P P P VINT Project Instrumented Protocol Code P : C++ P P P Protocol Agents Scenario Configuration Parameters: OTcl N N N Network Model Topology: OTcl Traffic Agents src/sink src/sink src/sink Traffic Model: OTcl NS Trace VINT Project

Verisim Properties: MEDL Instrumented Protocol: C++ NS Checker Trace Metatrace Scenario: OTcl

Ad Hoc Networks Routing for a wireless internetwork without the aid of a central base station. Connections are low-bandwidth, lossy, and highly transient. Unique routing assumptions: Most routes are seldom used. Bandwidth must be protected.

Illustration Part 1 of 2 Movement Routing

Illustration Part 2 of 2 New Routing

AODV Protocol If a node S needs a route to a destination D and does not have one, it floods a route-request (RREQ) packet through the network. Each recipient R of this RREQ keeps a return pointer. R broadcasts the request to its neighbors if it is not D and does not have a route to D. If R is D, or has a route to D, it responds with a route-reply (RREP) packet using the return pointers for S. Perkins and Royer 99

Loop Freedom in AODV Routing loops are undesirable. AODV uses sequence numbers to indicate freshness of link information. Key Invariant: If next(n) = n’, then seqno(n)  seqno(n’), and if seqno(n) = seqno(n’), then hops(n) > hops(n’). The invariant ensures that there are no loops.

Outline of Experiment Run a scenario of modest complexity. Analyze it in Verisim using a list of 9 properties of AODV expressed in MEDL. First pass: Repair First Bug (RFB). Second pass: “tune” the MEDL formulas to avoid rerunning the simulation discovering bugs in the metatrace. We instrumented simulation code for AODVv0 supplied by the CMU Monarch Project.

Experiment Scenario 1 2 3 4 5

Experiment Scenario 1 2 3 4 5

Experiment Scenario 1 2 3 4 5

Experiment Scenario 1 2 3 5 4

Experiment Scenario 1 5 2 3 4

Sample MEDL Alarm alarm LoopInv[at][nxt][dst] = sendrout[at][dst] when ((atnxt)  (atdst)  (nxtdst)  (obs_nexthop[at][dst] == nxt)  ((obs_seqno[at][dst] > obs_seqno[nxt][dst])  ((obs_seqno[at][dst] == obs_seqno[nxt][dst])  (obs_hopcont[at][dst] <= obs_hopcnt[at][dst])))) This is the negation of the fundamental invariant ensuring no loops in AODV: If the next hop from node at toward destination dst is node nxt then the sequence number (for dst) of at is less than or equal to that of nxt or they are equal and the hop count (for dest) of at is less than or equal to that of nxt.

Repair First Bug Experiment Let  be the MEDL property set. Run the simulation to get a trace T. Run the checker to get a metatrace T. Repair the first bug in the metatrace to get new protocol code. Rerun the simulation with the new protocol code to get a new trace U. Rerun the checker to get a new metatrace U. Continue until an empty metatrace is obtained.

RFB Experiment Statistics Meta- trace Dest Rep Detect RErr Node Loop Env Total Alarms T 4 54 38 43 220 U 216 V 48 39 44 206 W 1

Tuning Experiment Statistics Meta- trace Dest Rep Detect RErr Node Loop Env Total Alarms T 4 54 38 43 220 T 216 T 50 166 T 21 No new simulation traces.

Analysis of “Off-The-Shelf” (OTS) Simulations The prior experiment involved fewer than 10,000 events, and we designed it to exercise key scenarios. Can we do useful analysis with OTS performance simulations? Aim: Verisim can be added with small modifications, run alongside OTS simulations, and find logical failures in a practical period of time.

Experiment Took largest available OTS simulation for AODV and ran it to create a trace. Simulation from Monarch uses 50 mobile nodes on 1500x300m grid moving 20 m/s. 5220 seconds (1.5 hours) to complete simulation. 6,446,316 events. Naïve effort: ran MEDL with MonSeqNo test on all nodes (2500 relations) using 550Mhz dual processor machine with 1GB of memory. Aborted the experiment after 4 days; estimate 100 days to complete this analysis.

Optimizations For Network Routing Simulations Population Abstraction: test the property for a subset of the routers. Packet Type Abstraction: prune the trace to include only relevant events.

Sample Experiment Results for MonSeqNo Population Abstraction for 5 nodes. Trace size: 6,446,316 events Property size: 14,543 bytes Time: 51,054 seconds Rate: 0.54 micro seconds per event per property. Population and Packet Type Abstractions. Trace size: 6812 events Time: 51 seconds (found failures) Rate: 0.51 micro seconds per event per property.

Sample Experiment Results for LoopInv Population Abstraction for 5 nodes. Trace size: 69,411 events Property size: 75,508 bytes Time: 8064 seconds Rate: 1.54 micro seconds per event per property. Population and Packet Type Abstractions. Trace size: 48,735 events Time: 5912 seconds (found failures) Rate: 1.61 micro seconds per event per property.

Related Work Communication-based test generation systems. STRESS and Verisoft. Test oracles based on formulas or formal operational specifications. GIL and TETRA. Formal instance verification of routing protocols. SPIN/PITHIA for PNNI. Simulations based on logical specifications. MTSim and Maude. Network monitoring systems. Emerald.

Summary Verisim integrates simulation and logical trace analysis. The combination provides a more flexible approach to analyzing network simulations for safety properties. It is able to find bugs in existing simulator studies. Its flexibility can be exploited to improve debugging turn-around time.