On the Accuracy of MANET Simulators David Cavin Yoav Sasson & André Schiper Presented by Michael W. Totaro Mobile Computing and Wireless Systems (MoCWiS) group UL Lafayette - CACS

Topics Overview Introduction Related Work Flooding Algorithm The Simulators Simulations Conclusions Q & A

Overview The simulation phase of MANET applications or protocol deployment requires meaningful simulation results The model on which the simulator is based should match as closely as possible to reality Simulation results of a straightforward algorithm using several popular simulators are presented, whereby significant divergences exist between the simulators

Introduction Context – Interest in MANETs (Mobile Ad-hoc Networks) requires adaptation of solutions from the traditional wired networks to the wireless environment – Simulation is a tool that can often help to improve or validate protocols – Generally speaking, all simulators provide a complete toolkit to developers that facilitates metrics collection and various wireless network diagnostics

Introduction (2) Accuracy of simulation results – Popular simulators such as NS-2, OPNET Modeler, and GloMoSim provide advanced simulation environments to test and debug networking protocols, including wireless applications – It is essential that the simulated behaviors match as closely as possible the reality – This latter requirement assumes that several issues are sufficiently addressed

Introduction (3) Accuracy of simulation results (cont’d) – First Issue Application is likely to rely on components such as a collision detection module, as well as radio propagation or MAC protocols Correct definitions of these components in the simulator is critical Typically, the algorithm being evaluated is modeled in detail; however, cross-layer interactions are very rarely taken into account

Introduction (4) Accuracy of simulation results (cont’d) – Second Issue Simulation parameters and the environment (e.g., mobility schemes, power ranges, connectivity) must be realistic Incorrect initial conditions may lead to unexpected results that are not realizable in a real network

Introduction (5) Accuracy of simulation results (cont’d) – Focus of research The research presented in this paper shows the results of a set of measures collected during the simulation of a flooding algorithm on three different simulators: OPNET, NS-2, and GloMoSim Special attention was given to setting the same parameters and considering the same scenarios in each simulator; nevertheless, very different results—barely compatible—were collected

Related Work The research literature offers an abundance of papers on the efficiency of wireless algorithms comparing relative performances of each by means of simulation Few of these papers, however, focus on possible divergences that may occur between simulators, probably because the researchers work with only a single simulator—one with which they are most familiar—and thus do not expect nor anticipate significant differences among simulators

Related Work (2) The physical layer and the important parameters that influence its behavior have been modeled in NS-2 and OPNET – Results suggest that the configuration affects seriously the absolute performance of a protocol, and can even change the relative ranking among protocols for the same scenario

Related Work (3) The effect of detail in MANET simulations has been studied – Appropriate levels of detail in simulation models for radio propagation and energy consumption remain questionable Simulations that are too detailed may not be easily adapted to expeditiously explore alternatives Conversely, simulations that lack detail can lead to misleading or incorrect results

Flooding Algorithm Introduction – A frequently used operation to spread information to the whole network is the broadcast of messages – The performance of the broadcast is likely to affect the global efficiency of any protocol using it; hence, the broadcast should be implemented in the most efficient way

Flooding Algorithm (2) Introduction – Simulations Peer-to-peer wireless network, roughly 50 nodes randomly placed on a 1km x 1km area Ad-hoc mode, without any central access point (infrastructureless) Every node (peer) has the same possibilities and functionalities

Flooding Algorithm (3) Flooding – Flooding a message over the network is a simple way to implement broadcast Node initiates a broadcast Message is transmitted to its neighborhood (i.e., all nodes within the sender’s transmission range) When the message is received by a recipient for the first time, the recipient re-broadcasts it

Flooding Algorithm (4) Flooding example

Flooding Algorithm (5) Drawback of Flooding → overhead of flooded messages in the network Under ideal conditions (i.e., all nodes received the broadcast) in a network of N nodes, a single broadcast will generate exactly N copies of itself – Likely to increase probability of collisions – Most nodes will receive the same message several times, thus keeping the shared medium unnecessarily busy

Flooding Algorithm (6) Architecture Algorithm protocol stack Assume that every message has unique ID

The Simulators Introduction – The way a new algorithm is integrated can be considerably different from one simulator to another – A summary of the different implementation approaches for each simulator is presented, along with particular requirements and challenges

The Simulators (2) OPNET Modeler – Can simulate many kinds of wired networks, and a compliant MAC layer implementation is also provided – Phases of OPNET deployment process 1. Choose and configure node models to use in simulations—for example, a wireless node, a workstation, a router, and so on 2. Build and organize network by connecting the different entities 3. Select the statistics you wish to collect during simulations

The Simulators (3) OPNET Modeler (cont’d) – In this experiment, the authors created a new node model which encapsulates MAC layer of OPNET, as well as an application process that implements the flooding algorithm Flooding algorithm process model is described as a state machine, whereby each state has code that is executed upon state activation A transition that links two states is followed whenever a certain condition carried by the transition is true Difficulty with OPNET is actually building the state machine for each level of the protocol stack

The Simulators (4) NS-2 – Discrete event network simulator that supports both wired and wireless networks, including most MANET routing protocols as well as an MAC layer implementation – Source code is split between C++ for its core engine, and OTcl, an object-oriented version of PCL for configuration and simulation scripts – Implementation and simulation steps 1. Implement the protocol by adding a combination of C++ and OTcl code to NS-2’s source base 2. Describe the simulation in an OTcl script 3. Run the simulation 4. Analyze the generated trace file

The Simulators (5) NS-2 (cont’d) – In this experiment, the authors adapted the implementation of flooding provided in NS-2 An Agent (which, in NS-2, represents an endpoint where packets are constructed, processed, or consumed) was implemented at the Application layer for the broadcast source, and the simulation trace was collected at the MAC layer Major challenges with NS-2 include: a substantial learning curve; difficult debugging; a large memory footprint; and, a lack of scalability

The Simulators (6) GloMoSim – Scalable simulation environment for wireless and wired networks, developed initially at UCLA Computing Laboratory – Provides various applications (CBR, ftp, telnet), transport protocols (tcp, udp), routing protocols (AODV, flooding), and mobility schemes (random waypoint, random drunken) – User must define specific scenarios in text configuration files app file—contains description of traffic to generate (e.g., app type, bit rate, and so on) Config file—contains description of other (remaining) parameters

The Simulators (3) GloMoSim (cont’d) – Statistics collected can be either textual or graphical – According to the authors, compared to OPNET, GloMoSim’s architecture is much less flexible

Simulations Static parametersVarying parameters Common constant parameters of the simulations Varying parameters that describe the behavior of an ad-hoc network and that can be set in a controlled way

Simulations (2) Metrics – First metric gives information about the time needed to flood a message Time delay: For a node n, this is the average time needed for one packet to reach n – Second metric measures the general efficiency of the algorithm Success rate: For a node n, this is the difference between the expected and the actual number of messages received at n – Third metric stores the overhead of messages that are unnecessarily flooded in the network Overhead: For a node n, this is the sum of duplicated packets received by n

Simulations (3) Results – Only the most striking graphs are provided in the paper – Several scenarios are defined by varying one or more parameters from the previous table labeled “Varying parameters” – For each scenario, the set of varied parameters is given in the table just above the graph

Simulations (4) This scenario depicts a critical factor that influences the success rate in MANETs: the effective transmission range Notice the apparent differences in trend between the simulators Success rate vs. Power range GloMoSim OPNET NS-2 Success rate vs. Power range

Simulations (5) This scenario evaluates the effects of node mobility on the flooding’s ability to deliver packets reliably Again, we see a significant difference in success rate Success rate vs. Mobility GloMoSim OPNET NS-2 Success rate vs. Mobility

Simulations (6) This scenario presents the average overhead of messages flooded in the network for a single simulation run This metric is related to the mean number of reachable neighbors (that is, within transmission range OPNET NS-2 GloMoSim Overhead vs. Mobility

Simulations (7) The final scenario compares average time delay needed to flood a message throughout the whole network This metric increases with the number of hops from source to destination and also whenever collisions occur Time delay vs. Mobility OPNET NS-2 GloMoSim Time delay vs. Mobility

Simulations (8) Analysis and interpretation – Simulation results of the flooding algorithm demonstrate that modeling of the MAC protocol and of the physical layer can lead to different results, depending upon the simulator – Possible reasons Differing physical layer implementations Implementation of a new protocol is itself difficult to transpose from one simulator to another Given that successive releases provide bug fixes, it is reasonable to assume that MANET simulators still contain errors or incompatibilities to IEEE standard

Conclusions (Authors) Instead of simulations, a more realistic scheme might entail a hybrid approach in which only the lowest layers—MAC and physical—and the mobility model are simulated and all the upper layers (from transport to application) are executed on a cluster of machines There is an important lack of real experiments the prove the feasibility of wireless protocols

Questions? ?