Sizing Router Buffers

1
Sizing Router Buffers

• Isaac Keslassy (Technion)
• Guido Appenzeller Nick McKeown (Stanford)

2
Routers Need Packet Buffers

• Its well known that routers need packet buffers
• Its less clear why and how much
• Goal of this work is to answer the question
• How much buffering do routers need?

• Given that queueing delay is the only variable
  part of packet delay in the Internet, youd
  think wed know the answer already!

3
How Much Buffer Does a Router Need?
Source
Destination
Router
C
2T

• Universally applied rule-of-thumb
• A router needs a buffer size
• 2T is the two-way propagation delay (or just
  250ms)
• C is capacity of bottleneck link
• Context
• Mandated in backbone and edge routers.
• Appears in RFPs and IETF architectural
  guidelines.
• Usually referenced to Villamizar and Song High
  Performance TCP in ANSNET, CCR, 1994.
• Already known by inventors of TCP Van Jacobson,
  1988.
• Has major consequences for router design.

4
Example

• 10Gb/s linecard
• Requires 300Mbytes of buffering.
• Read and write 40 byte packet every 32ns.
• Memory technologies
• DRAM require 4 devices, but too slow.
• SRAM require 80 devices, 1kW, 2000.
• Problem gets harder at 40Gb/s
• Hence RLDRAM, FCRAM, etc.

5
Main Result in This Talk

• The rule of thumb is wrong for a core router
  today
• Required buffer is instead of

6
Outline of this Talk

• The Rule-of-Thumb on Buffer Sizing is incorrect
• Where the rule of thumb comes from
• Why it is incorrect for a core router in the
  Internet today
• Real Buffer Requirements in case of Congestion
• Real Buffer Requirements without Congestion
• Experimental results from real Networks

7
TCP
Only W2 packets may be outstanding
Source
Dest
C
C gt C

• TCP Congestion Window controls the sending rate
• Sender sends packets, receiver sends ACKs
• Sending rate is controlled by Window W,
• At any time, only W unacknowledged packets may be
  outstanding
• The sending rate of TCP is

8
Single TCP FlowRouter with large enough buffers
for full link utilization
B
Dest
Source
C
C gt C
9
Required buffer is height of sawtooth
B
0
t
10
Origin of rule-of-thumb

• Before and after reducing window size, the
  sending rate of theTCP sender is the same
• Inserting the rate equation we get
• The RTT is part transmission delay T and part
  queueing delay B/C . We know that after reducing
  the window, the queueing delay is zero.

?
11
Rule-of-thumb

• Rule-of-thumb makes sense for one flow
• Typical backbone link has gt 20,000 flows
• Does the rule-of-thumb still hold?
• Answer
• If flows are perfectly synchronized, then Yes.
• If flows are desynchronized then No.

12
Outline of this Talk

• The Rule-of-Thumb on Buffer Sizing is incorrect
• Real Buffer Requirements in case of Congestion
• Correct buffer requirements for a congested
  router
• Result
• Real Buffer Requirements without Congestion
• Experimental results from real Networks

13
If flows are synchronized
t

• Aggregate window has same dynamics
• Therefore buffer occupancy has same dynamics
• Rule-of-thumb still holds.

14
When are Flows Synchronized?

• Small numbers of flows tend to synchronize
• Large aggregates of flows are not synchronized
• For gt 200 flows, synchronization disappears
• Measurements in the core give no indication of
  synchronization

15
If flows are not synchronized
B
0
16
Central Limit Theorem

• CLT tells us that the more variables (congestion
  windows of flows) we have, the narrower the
  Gaussian (fluctuation of sum of windows)
• Width of Gaussian decreases with
• Buffer size should also decrease with

17
Required buffer size
Simulation
18
Summary

• Flows in the core are desynchronized
• For desynchronized flows, routers need only
  buffers of

19
Outline of this Talk

• The Rule-of-Thumb on Buffer Sizing is incorrect
• Real Buffer Requirements in case of Congestion
• Real Buffer Requirements without Congestion
• Correct buffer requirements for an
  over-provisioned network
• Result Even smaller buffers
• Experimental results from real Networks

20
Short Flows

• So far we were assuming a congested router with
  long flows in congestion avoidance mode.
• What about flows in slow start?
• Do buffer requirements differ?
• Answer Yes, however
• Required buffer in such cases is independent of
  line speed and RTT (same for 1Mbit/s or 40
  Gbit/s)
• In mixes of flows, long flows drive buffer
  requirements
• Short flow result relevant for uncongested routers

21
A single, short-lived TCP flowFlow length 62
packets, RTT 140 ms
32
Flow Completion Time (FCT)
16
8
4
fin ackreceived
2
syn
RTT
22
Average Queue length
(S is burst distribution of flows)
23
Queue Distribution

• We derived closed-form estimates of the queue
  distribution using Effective Bandwidth
• Gives very good closed form approximation
• Buffer requirements for short flows
• Small independent of line speed and RTT
• In mixes of flows, long flows dominate buffer
  requirements

24
Outline of this Talk

• The Rule-of-Thumb on Buffer Sizing is incorrect
• Real Buffer Requirements in case of Congestion
• Real Buffer Requirements without Congestion
• Results from Real Networks
• Lab results with a physical router
• Experiments on production networks with real
  traffic

25
Experimental Evaluation Overview

• Simulation with ns2
• Over 10,000 simulations that cover range of
  settings
• Simulation time 30s to 5 minutes
• Bandwidth 10 Mb/s - 1 Gb/s
• Latency 20ms -250 ms,
• Physical router
• Cisco GSR with OC3 line card
• In collaboration with University of Wisconsin
• Experimental results presented here
• Long Flows - Utilization
• Mixes of flows - Flow Completion Time (FCT)
• Mixes of flows - Heavy Tailed Flow Distribution
• Short Flows Queue Distribution

26
Long Flows - Utilization (I)Small Buffers are
sufficient - OC3 Line, 100ms RTT
99.9
2
99.5
98.0
27
Long Flows Utilization (II) Model vs. ns2 vs.
Physical RouterGSR 12000, OC3 Line Card
28
Short Flows Queue DistributionModel vs.
Physical Router, OC3 Line Card
29
Experiments with live traffic (I)

• Stanford University Gateway
• Link from internet to student dormitories
• Estimated 400 concurrent flows, 25 Mb/s
• 7200 VXR (shared memory router)

Thanks to Sunia Yang, Wayne Sung and the Stanford
Backbone Team
30
Experiment with live traffic (II)Internet2 link
Indianapolis to Kansas City

• Link Setup
• 10Gb/s link, T640
• Default Buffer 1000 ms
• Flows of 1 Gb/s
• Loss requirement lt 10-8

• Experiment
• Reduced buffer to 10 ms (1) - nothing happened
• Reduced buffer to 5 ms (0.5) - nothing happened
• Next buffer of 2ms (0.2)
• Experiment ongoing

Thanks to Stanislav Shalunov of Internet2 and Guy
Almes (now at NSF)
31
Outline

• The Rule of Thumb
• The buffer requirements for a congested router
• Buffer requirements for short flows (slow-start)
• Experimental Verification
• Conclusion

32
Impact on Router Design

• 10Gb/s linecard with 200,000 x 56kb/s flows
• Rule-of-thumb Buffer 2.5Gbits
• Requires external, slow DRAM
• Becomes Buffer 6Mbits
• Can use on-chip, fast SRAM
• Completion time halved for short-flows
• 40Gb/s linecard with 40,000 x 1Mb/s flows
• Rule-of-thumb Buffer 10Gbits
• Becomes Buffer 50Mbits
• For more details
• Sizing Router Buffers Guido Appenzeller, Isaac
  Keslassy and Nick McKeown, to appear at SIGCOMM
  2004

33
Open Questions

• Since buffers can be made much smaller than the
  rule-of-thumb, can we make all-optical buffers?
• How small can buffers be?
• What is the congestion control algorithm that
  minimizes the buffer size?

34
(No Transcript)
35
Buffer rule of thumb
36
Over-buffered Link
37
Under-buffered Link
38
Quantitative Model

• Model congestion window of a flow as random
  variable

model as
where

• For many de-synchronized flows
• We assume congestion windows are independent
• All congestion windows have the same probability
  distribution

• Now central limit theorem gives us the
  distribution of the sum of the window sizes

39
Buffer vs. Number of Flowsfor a given Bandwidth

• If for a single flow we have

• For a given C, the window W scales with 1/n and
  thus

• Standard deviation of sum of windows decreases
  with n

• Thus as n increases, buffer size should decrease

40
Long Flows - Utilization (I)Small Buffers are
sufficient - OC3 Line, 100ms RTT
99.9
2
99.5
98.0