1 Per Gunningberg© A Real-World Test-bed for Mobile Ad hoc Networks: Methodology, Experimentations, Simulation and Results. Per Gunningberg, Erik Nordström, Christian Rohner, Oskar Wibling Uppsala University

2 Per Gunningberg© Background and problem IETF is standardizing MANET (Mobile Adhoc NETwork) routing protocols: –One proactive protocol - knowledge about all nodes –One reactive protocol - path on the need basis Based on experiences from three protocols: –AODV - Adhoc On Demand Distance Vector (reactive) –DSR - Dynamic Source Routing (reactive) –OLSR - OptiMized Link State Routing(proactive) Problem: But majority of research done through simulations...

3 Per Gunningberg© Part One A test-bed for evaluating ad hoc routing protocols. Close to reality What to measure and how to analyze Repeatable experiments Grey Zone Phenomena Conclusion

4 Per Gunningberg© The Uppsala Ad hoc Protocol Evaluation Testbed (APE) People carrying laptops with b Suitable for indoor experiments that are hard to model in simulation

5 Per Gunningberg©

6 The Ad hoc Protocol Evaluation Testbed (APE) Execution environment on top of existing OS. –Runs on Win and Linux Scenarios with movement choreography. Emphasizes easy management for scaling downloads.

7 Per Gunningberg© Laptop instructions (choreography) node.11.action.0.msg=Test is starting... node.11.action.0.command=start_spyd node.11.action.0.duration=1 node.11.action.1.command=my_iperf c 2 t 330 node.11.action.1.msg=Stay at this location. node.11.action.1.duration=30 node.11.action.2.msg=Start moving! Go to Point A, the end of building. node.11.action.2.duration=75 node.11.action.3.msg=You should have arrived at Point A. Please stay. node.11.action.3.duration=30

8 Per Gunningberg© Measurement procedures Every node collects SNR from every other node it can hear during the test session Every event is time stamped Received Packets/Application results are collected at all nodes Routing state snapshots are collected Analysis is done after the test session.

9 Per Gunningberg© Replaying a scenario SNR mapped to virtual distance Each time interval corresponds to a topological map T Point A Point D

10 Per Gunningberg© APE is a Testbed for… 1.Relative protocol performance comparisons 2.Radio channel effects on ad hoc routing protocols 3.Interactions between hardware, software, protocol, mobility and radio environment Example: Grey Zone Phenomena 4.Validation of simulation models 5.Generation of traces

11 Per Gunningberg© Gray Zone Phenomena AA Broadcast Unicast 3

12 Per Gunningberg© Challenge Results should be reproducible and comparable between tests It follows that experiments must be repeatable......and therefore stochastic factors need to be dealt with So – what can we achieve?

13 Per Gunningberg© Stochastic Factors in Real World Experiments Node mobility adds frequent changes in the network topology. –We use choreography and “measure topology differences” Variations in hardware and software configuration. –We use identical hardware and software. Time varying radio environment affects link quality and error rates.

14 Per Gunningberg© Topology differences - visual check RED = Average mobility GREEN = 25% with lowest mobility BLUE = 25% with highest mobility Experiment 1Experiment 2

15 Per Gunningberg© Part Two Evaluating MANET protocols with the APE testbed, simulation and emulation. Scenarios UDP, Ping and TCP Side-by-side comparison Faulty protocol constructs Conclusion

16 Per Gunningberg© Coupling Simulation, Emulation and Real World

17 Per Gunningberg© Routing protocols ability to adapt OLSR - Proactive Link state routing. Monitors neighbors and exchange link state info. AODV - broadcasts to set up path. HELLO or Link feedback to detect link failure. DSR - broadcasts with source route. Listens to other traffic to find shorter route. RTT measurements and network ACKs. React to connectivity changes

18 Per Gunningberg© Emulation Same configuration as Real world Table-top emulation MAC filters force connectivity changes Reduces radio and mobility factors Interference reduces bandwidth

19 Per Gunningberg© Simulation Scenarios recreated in a ns2-simulation using “default” models: –Transmission range tuned to better match indoors –Mobility with jitter modeled after real world measurements –Results averaged over 10 runs Results provide a baseline Can simulations using default (simple) models be used to predict routing protocol performance in complex real world environments?

20 Per Gunningberg© Multidimensional Comparison Three MANET routing protocol implementations: –OOLSR, AODV-UU, DSR-UU Three traffic types: –UDP (20 pkts/s CBR) –Ping (20 pkts/s CBR) –TCP (File transfer) Three mobility scenarios: –End node swap, Relay node swap, Roaming node Three environments (dimensions): –Simulation, Emulation, Real world 3x3x3x(10 runs) = 270 runs

21 Per Gunningberg© Experimental Test Environment Indoors with offices and corridors Four nodes (0, 1, 2, 3) Four waypoints (A, B, C, D) One data stream from node 3 to node 0

22 Per Gunningberg© Relay Node swap AA ABCD 0123

23 Per Gunningberg© Scenarios – Relay Node Swap End nodes stationary Intermediate nodes changes position Hop count never smaller than 2

24 Per Gunningberg© End node swap AA ABCD 0123

25 Per Gunningberg© Scenarios – End Node Swap End nodes change positions Intermediary nodes stationary Hop count changes from 3 to (2) and 1 and back

26 Per Gunningberg© Roaming node AA ABCD 0123

27 Per Gunningberg© Scenarios – Roaming Node Roaming node is source node All other nodes stationary

28 Per Gunningberg© Results – Relay Node Swap

29 Per Gunningberg© Results – End Node Swap

30 Per Gunningberg© Results – Roaming Node

31 Per Gunningberg© AODV - UDP - End Node Swap

32 Per Gunningberg© OLSR - UDP - End Node Swap

33 Per Gunningberg© TCP - Simulation/Real World

34 Per Gunningberg© Observations Simulation and Emulation similar in absolute CBR performance but not in relative protocol ranking Real world CBR performance is significantly lower Discrepancy grows with traffic complexity and scenario TCP performance is orders of magnitude lower for real world compared to simulation periods of no-progress time in real world

35 Per Gunningberg© Observations (continued) OLSR tries less hard to re-route and therefore achieves more even performance Radio factors account for most of the discrepancy between simulation and real world......but secondary effects, such as cross-layer interactions that are protocol specific, dominate, e.g.: –Lost HELLOs (AODV) –Excessive buffering (DSR)

36 Per Gunningberg© Protocol comparison conclusion If one protocol performs better than another in simulation, is it possible to assume the same for the real world? NO

37 Per Gunningberg© Latency - Ping - Relay Node

38 Per Gunningberg© Flip-Flop Routing DSR Real Word Simulation

39 Per Gunningberg© Adapting to topology change

40 Per Gunningberg© Routing Control Overhead

41 Per Gunningberg© Conclusions APE aims to address the lack of real world ad hoc experimental research test-beds Repeatability addressed at a level that allows relative protocol comparisons The value of cross-environment evaluation Revealing of sensing problems leading to instabilities and poor performance Not visible in simulations

42 Per Gunningberg© The End Paper: publications/GC_technical_report.pdf APE testbed: The Research group:

43 Per Gunningberg© Extra Slides More details…

44 Per Gunningberg© Self Interference Simulation

45 Per Gunningberg© UDP

46 Per Gunningberg© Ping

47 Per Gunningberg© TCP