New Directions in Traffic Measurement and Accounting

1
New Directions in TrafficMeasurement and
Accounting

• Cristian Estan
• (joint work with George Varghese)

2
The Problem

• Measurement and monitoring of network traffic
  required for Internet backbones.
• Useful for short-term monitoring (e.g., DOS
  attacks), traffic engineering (e.g., rerouting),
  and accounting (e.g., usage based pricing)
• How can we do so without tracking millions of
  ants to track a few elephants?

3
State of the art Cisco NetFlow

• Sample packets at high speeds
• Per flow information based on samples
• Aggregate (based on ASes, prefixes, ports) at the
  router
• Problems inaccurate (due to sampling and loss),
  memory-intensive, slow (needs DRAM).

4
Towards a NetFlow Alternative

• Small Percentage of flows (elephants) account for
  large percentage of traffic.
• Top 9 of flows account for 90 of AS pair
  traffic in backbones (Fang-Peterson).
• Can we directly track flows that send say over 1
  of link bandwidth without keeping track of all
  flows?

5
How to identify large flows?
We introduce two new methods for this purpose

• Sample-Counting uses sampling only to decide
  which flows to watch exhaustively.
• Multistage filter uses multiple hash tables
  allowing large flows to be identified while only
  allowing a small number of small flows (false
  positives) to pass through filter.

6
Identify large flows by sampling
7
Multistage filters
8
Operation of Sampled NetFlow
How accurate is Sampled NetFlow? 1 gigabyte/100
megabytes of data Sampling one in 100 packets
Error 1GB 100MB
1 39.24 79.03
3 1.07 41.48
9
Operation of our algorithms
Error 1GB 100MB
1 5.6E-32 0.09
3 1.8E-94 6.96E-10
Sampling 1/1000
Error 1GB 100MB
1 0.08 49.69
3 4.66E-10 12.25
Filter error 0
10
Comparison
Sampled NetFlow Identify by sampling Multistage filters
Accuracy Medium Good Good
False neg. Few (high var.) Few (low var.) Never
False pos. Few Few Few
Memory Big Small Very small
Complexity Low Low Medium
11
High Speed Implementation?

• John Huber of MMC Networks did a design of a chip
  doing filter counting.
• 450,000 transistors, under 1 watt of power, runs
  at OC-192 rates, 32 nsec per packet
• Seems easily feasible to implement sample
  counting with similar complexity.

12
Potential Application scalable threshold
accounting

• Measure flows sending over x of link bandwidth
  using sample/filter counting.
• Bill using flat fee per byte charge for flows
  over x
• Track aggregates directly to avoid evasion using
  several flows, each lt x
• Generalizes usage based (x 0) and duration
  based (x 100) pricing.

13
Conclusions

• Paradigm shift for measurement by concentrating
  only on heavy flows
• Two new techniques (sample and filter counting)
  with small memory footprints and provable
  performance.
• Techniques make threshold accounting feasible,
  generalizing usage and duration based pricing.