DIMES

1
DIMES

• Distributed Internet MEasurement and Simulation
• Yuval Shavitt Eran Shir
• Tel-Aviv University

2
What is DIMES All About?

• Let the network measure itself
• Persuade people to download our DIMES measurement
  agent
• Carry on experiments to measure and map the
  Internet and track its growth.
• Target
• 1,000s of users in the first year
• More than 10,000s of users in three years

3
How does the Internet look like?

• FFF99 The Internet topology degree distribution
  obeys a power law
• Dramatically changed the way researchers view the
  Internet
• Become the standard simulation topology
• Chen et al. 02 The Origin of Power Laws in
  Internet Topologies Revisited.
• FFF99 data is only partial
• There is no power law.
• Problem their data is problematic as well

4
Our claim

• All current studies are based on partial data.
• Chen et al 02 has a problem with neighbor
  definition
• Our initial aim collect the network topology
  with high accuracy

5
Past Attempts

• Measurements were done from a few (up to 100s)
  points
• - too many links are missed especially in the
  periphery
• - measurements traffic was too dense
• Some maps were created based on central databases
• data is incossistent and partial

6
For Example Segments of ILAN as viewed from
Outside ILAN (Inside Israel)
7
Segments of ILAN as viewed from Inside Ilan
8
Are there any differences?
9
Well, Yes!
10
The Goal

• Creating a distributed platform that will enable
• Global scale measurement of Internet graph
  structure, packet traffic statistics, demography
• Simulation of Internet behavior under different
  conditions (let the net simulate itself)
• Simulation of the Internet future
• Active networks
• Novel routing algorithms
• Distributed resource allocation grid computing
• P2P

11
Challenges

• Optimize the architecture
• Minimize the number of measurements
• Expedite the discovery rate
• Flying under the NOC radar screens
• Study self-emerging agent collaboration
• Data analysis
• and more .

12
Architecture - Overview

• Building blocks
• A lightweight, programmable and sandboxed agent
  installed in thousands of nodes all over the net
• A planning and scheduling server(s) assigns
  measurement scripts and manages experiments
• Two communication channels (due to security)
• Control for script receiving and status report
• Data for measurement results transfer
• The agents have a separate (sandboxed)
  communication
• channel for performing measurements

13
Architecture - continue

• Each agent implements a set of measurement
  modules (e.g. Ping)
• The agent uses these modules according to
  received scripts from server e.g.
• OnTime local startTime 1800 endTime 0300
  period 30m do ping 132.66.(20-127).2
• while (sysDatelt20/01/04 ilt127)
• OnTime local startTime 0330 endTime 0730 period
  20m do traceroute i.0.0.1
• i
• end while
• Scripts are created by the planner in the server
  according to the experiment guidelines and agent
  profile
• The agent schedules tasks carefully in order not
  to exceed normal user traffic profile. Also,
  whenever possible, TCP and UDP are used
• Received results are used at the server to update
  the maintained maps and other calculations

14
Architecture - Server
15
Architecture - Agent
16
Methodology

• Use uncorrelated Measurements (e.g. Ping,
  Traceroute clones) to
• Identify AS level topology and delay map
• Identify Routers level topology and delay map
• Identify the evolution nature of the Internet
• Use correlated measurements to
• Expedite measurement discovery
• Study routing optimality
• Study overlay networks
• Study network self-organization through agent
  collaboration

17
New Algorithms

• New algorithms are developed for the two
  endpoints of a DIMES experiment planning and
  results analysis
• The planning algorithms should enable optimal
  usage of DIMES resources (i.e. agents) by
  carefully assigning the right measurement tasks
  in accordance with experiment objectives
• Network modeling and embedding algorithms greatly
  reduce agent designation overhead
• The analysis and map creation algorithm should be
  able to cope with data received from multitude of
  sources with varying profile

18
DIMES PlanetLab

• We plan to use PlanetLab in order to leverage the
  DIMES infrastructure
• Initial Deployment
• Top Tier in two tiers experimental architecture
• Enablement of correlated measurements
• (How to get routing diversity?)

19
Deliverables (partial EU funding starting
date January 2004)

• Connectivity statistics including hidden links
  12 months
• Delay map 12 months
• Topology (K-Core, small worldness) including
  hidden links 18 months
• Corresponding I/O traffic statistics 24 months
• Usage mode statistics (e.g. HTTP vs. P2P)
• Traffic flow mapping

20
Growthpath directions over DIMES lifetime
Time
21
Who

• PI Yuval Shavitt
• Ph.D. students Eran Shir, Tomer Tankel, Amir
  Shay
• Post-doc Osnat Mokryn
• Programmer Anat Halpern

22
Thank you

• Those who wish, are encouraged to go to our
  website www.netDimes.org download DIMES agents
  beta version and participate in our experiments.