On the Accuracy of MANET Simulators

1
On the Accuracy of MANET Simulators

• David Cavin
• Yoav Sasson
• André Schiper

Presented by Michael W. Totaro Mobile Computing
and Wireless Systems (MoCWiS) group UL Lafayette
- CACS
2
Topics

• Overview
• Introduction
• Related Work
• Flooding Algorithm
• The Simulators
• Simulations
• Conclusions
• Q A

3
Overview

• The simulation phase of MANET applications or
  protocol deployment requires meaningful
  simulation results
• The model on which the simulator is based should
  match as closely as possible to reality
• Simulation results of a straightforward algorithm
  using several popular simulators are presented,
  whereby significant divergences exist between the
  simulators

4
Introduction

• Context
• Interest in MANETs (Mobile Ad-hoc Networks)
  requires adaptation of solutions from the
  traditional wired networks to the wireless
  environment
• Simulation is a tool that can often help to
  improve or validate protocols
• Generally speaking, all simulators provide a
  complete toolkit to developers that facilitates
  metrics collection and various wireless network
  diagnostics

5
Introduction (2)

• Accuracy of simulation results
• Popular simulators such as NS-2, OPNET Modeler,
  and GloMoSim provide advanced simulation
  environments to test and debug networking
  protocols, including wireless applications
• It is essential that the simulated behaviors
  match as closely as possible the reality
• This latter requirement assumes that several
  issues are sufficiently addressed

6
Introduction (3)

• Accuracy of simulation results (contd)
• First Issue
• Application is likely to rely on components such
  as a collision detection module, as well as radio
  propagation or MAC protocols
• Correct definitions of these components in the
  simulator is critical
• Typically, the algorithm being evaluated is
  modeled in detail however, cross-layer
  interactions are very rarely taken into account

7
Introduction (4)

• Accuracy of simulation results (contd)
• Second Issue
• Simulation parameters and the environment (e.g.,
  mobility schemes, power ranges, connectivity)
  must be realistic
• Incorrect initial conditions may lead to
  unexpected results that are not realizable in a
  real network

8
Introduction (5)

• Accuracy of simulation results (contd)
• Focus of research
• The research presented in this paper shows the
  results of a set of measures collected during the
  simulation of a flooding algorithm on three
  different simulators OPNET, NS-2, and GloMoSim
• Special attention was given to setting the same
  parameters and considering the same scenarios in
  each simulator nevertheless, very different
  resultsbarely compatiblewere collected

9
Related Work

• The research literature offers an abundance of
  papers on the efficiency of wireless algorithms
  comparing relative performances of each by means
  of simulation
• Few of these papers, however, focus on possible
  divergences that may occur between simulators,
  probably because the researchers work with only a
  single simulatorone with which they are most
  familiarand thus do not expect nor anticipate
  significant differences among simulators

10
Related Work (2)

• The physical layer and the important parameters
  that influence its behavior have been modeled in
  NS-2 and OPNET
• Results suggest that the configuration affects
  seriously the absolute performance of a protocol,
  and can even change the relative ranking among
  protocols for the same scenario

11
Related Work (3)

• The effect of detail in MANET simulations has
  been studied
• Appropriate levels of detail in simulation models
  for radio propagation and energy consumption
  remain questionable
• Simulations that are too detailed may not be
  easily adapted to expeditiously explore
  alternatives
• Conversely, simulations that lack detail can lead
  to misleading or incorrect results

12
Flooding Algorithm

• Introduction
• A frequently used operation to spread information
  to the whole network is the broadcast of messages
• The performance of the broadcast is likely to
  affect the global efficiency of any protocol
  using it hence, the broadcast should be
  implemented in the most efficient way

13
Flooding Algorithm (2)

• Introduction
• Simulations
• Peer-to-peer wireless network, roughly 50 nodes
  randomly placed on a 1km x 1km area
• Ad-hoc mode, without any central access point
  (infrastructureless)
• Every node (peer) has the same possibilities and
  functionalities

14
Flooding Algorithm (3)

• Flooding
• Flooding a message over the network is a simple
  way to implement broadcast
• Node initiates a broadcast
• Message is transmitted to its neighborhood (i.e.,
  all nodes within the senders transmission range)
• When the message is received by a recipient for
  the first time, the recipient re-broadcasts it

15
Flooding Algorithm (4)
Flooding example
16
Flooding Algorithm (5)

• Drawback of Flooding ? overhead of flooded
  messages in the network
• Under ideal conditions (i.e., all nodes received
  the broadcast) in a network of N nodes, a single
  broadcast will generate exactly N copies of
  itself
• Likely to increase probability of collisions
• Most nodes will receive the same message several
  times, thus keeping the shared medium
  unnecessarily busy

17
Flooding Algorithm (6)

• Architecture

Assume that every message has unique ID
Algorithm protocol stack
18
The Simulators

• Introduction
• The way a new algorithm is integrated can be
  considerably different from one simulator to
  another
• A summary of the different implementation
  approaches for each simulator is presented, along
  with particular requirements and challenges

19
The Simulators (2)

• OPNET Modeler
• Can simulate many kinds of wired networks, and a
  802.11 compliant MAC layer implementation is also
  provided
• Phases of OPNET deployment process
• Choose and configure node models to use in
  simulationsfor example, a wireless node, a
  workstation, a router, and so on
• Build and organize network by connecting the
  different entities
• Select the statistics you wish to collect during
  simulations

20
The Simulators (3)

• OPNET Modeler (contd)
• In this experiment, the authors created a new
  node model which encapsulates 802.11 MAC layer of
  OPNET, as well as an application process that
  implements the flooding algorithm
• Flooding algorithm process model is described as
  a state machine, whereby each state has code that
  is executed upon state activation
• A transition that links two states is followed
  whenever a certain condition carried by the
  transition is true
• Difficulty with OPNET is actually building the
  state machine for each level of the protocol stack

21
The Simulators (4)

• NS-2
• Discrete event network simulator that supports
  both wired and wireless networks, including most
  MANET routing protocols as well as an 802.11 MAC
  layer implementation
• Source code is split between C for its core
  engine, and OTcl, an object-oriented version of
  PCL for configuration and simulation scripts
• Implementation and simulation steps
• Implement the protocol by adding a combination of
  C and OTcl code to NS-2s source base
• Describe the simulation in an OTcl script
• Run the simulation
• Analyze the generated trace file

22
The Simulators (5)

• NS-2 (contd)
• In this experiment, the authors adapted the
  implementation of flooding provided in NS-2
• An Agent (which, in NS-2, represents an endpoint
  where packets are constructed, processed, or
  consumed) was implemented at the Application
  layer for the broadcast source, and the
  simulation trace was collected at the MAC layer
• Major challenges with NS-2 include a substantial
  learning curve difficult debugging a large
  memory footprint and, a lack of scalability

23
The Simulators (6)

• GloMoSim
• Scalable simulation environment for wireless and
  wired networks, developed initially at UCLA
  Computing Laboratory
• Provides various applications (CBR, ftp, telnet),
  transport protocols (tcp, udp), routing protocols
  (AODV, flooding), and mobility schemes (random
  waypoint, random drunken)
• User must define specific scenarios in text
  configuration files
• app filecontains description of traffic to
  generate (e.g., app type, bit rate, and so on)
• Config filecontains description of other
  (remaining) parameters

24
The Simulators (3)

• GloMoSim (contd)
• Statistics collected can be either textual or
  graphical
• According to the authors, compared to OPNET,
  GloMoSims architecture is much less flexible

25
Simulations
Static parameters
Varying parameters
Varying parameters that describe the behavior
of an ad-hoc network and that can be set in
a controlled way
Common constant parameters of the simulations
26
Simulations (2)

• Metrics
• First metric gives information about the time
  needed to flood a message
• Time delay For a node n, this is the average
  time needed for one packet to reach n
• Second metric measures the general efficiency of
  the algorithm
• Success rate For a node n, this is the
  difference between the expected and the actual
  number of messages received at n
• Third metric stores the overhead of messages that
  are unnecessarily flooded in the network
• Overhead For a node n, this is the sum of
  duplicated packets received by n

27
Simulations (3)

• Results
• Only the most striking graphs are provided in the
  paper
• Several scenarios are defined by varying one or
  more parameters from the previous table labeled
  Varying parameters
• For each scenario, the set of varied parameters
  is given in the table just above the graph

28
Simulations (4)
Success rate vs. Power range

• This scenario depicts a critical factor that
  influences the success rate in MANETs the
  effective transmission range
• Notice the apparent differences in trend between
  the simulators

OPNET
NS-2
GloMoSim
Success rate vs. Power range
29
Simulations (5)
Success rate vs. Mobility

• This scenario evaluates the effects of node
  mobility on the floodings ability to deliver
  packets reliably
• Again, we see a significant difference in success
  rate

GloMoSim
OPNET
NS-2
Success rate vs. Mobility
30
Simulations (6)
Overhead vs. Mobility

• This scenario presents the average overhead of
  messages flooded in the network for a single
  simulation run
• This metric is related to the mean number of
  reachable neighbors (that is, within transmission
  range

OPNET
NS-2
GloMoSim
Overhead vs. Mobility
31
Simulations (7)
Time delay vs. Mobility

• The final scenario compares average time delay
  needed to flood a message throughout the whole
  network
• This metric increases with the number of hops
  from source to destination and also whenever
  collisions occur

OPNET
GloMoSim
NS-2
Time delay vs. Mobility
32
Simulations (8)

• Analysis and interpretation
• Simulation results of the flooding algorithm
  demonstrate that modeling of the MAC protocol and
  of the physical layer can lead to different
  results, depending upon the simulator
• Possible reasons
• Differing physical layer implementations
• Implementation of a new protocol is itself
  difficult to transpose from one simulator to
  another
• Given that successive releases provide bug fixes,
  it is reasonable to assume that MANET simulators
  still contain errors or incompatibilities to IEEE
  802.11 standard

33
Conclusions (Authors)

• Instead of simulations, a more realistic scheme
  might entail a hybrid approach in which only the
  lowest layersMAC and physicaland the mobility
  model are simulated and all the upper layers
  (from transport to application) are executed on a
  cluster of machines
• There is an important lack of real experiments
  the prove the feasibility of wireless protocols

34
Questions?
?
Thank You!