Do Optics Belong

1
Do Optics Belong in Internet Core
Routers? Keynote, Opticomm 2001 Denver, Colorado
Nick McKeown Professor of Electrical Engineering
and Computer Science, Stanford
University nickm_at_stanford.edu http//www.stanford.
edu/nickm
2
There seem to be 3 opinions

• Optics and routers dont belong together
• Optics are ill-suited to packet switching.
• CMOS technology and architectural techniques will
  scale just fine.
• Optical circuit switches will kill off (core)
  routers
• Optical circuit switches are simpler and faster
  than routers.
• We dont need packet switching anymore.
• Optics and routers belong together
• Electronic switched backplanes will be replaced
  by optics.
• Leads to lower power, higher density routers.

3
Optics and routers dont belong togetherOptics
are ill-suited to packet switching

• Buffering
• Packet switches inherently require buffers for
  times of congestion,
• Buffers provide statistical multiplexing for
  outgoing links,
• Optical buffers are not economically feasible.
• Processing
• Packet processing is too complex to be done
  optically.

4
Buffering
rate
A
x
x

• The Internet is built
• on the assumption of
• expensive, congested
• links.
• Statistical multiplexing enables sharing of
  expensive links.
• All routers have big buffers.
• Rule of thumb buffersize RTT
  line-rate.At 10Gb/s 2.5Gbits.

A
time
B
rate
x
x
B
time
5
Processing
Typical IP Router Linecard

• Buffer
• State
• Memory

Lookup Tables
Buffered or Bufferless Fabric
Optics
Packet Processing
Buffer Mgmt Scheduling
Physical Layer
Framing Maintenance
Buffer Mgmt Scheduling

• Buffer
• State
• Memory

Arbitration

• OC192c linecard
• 30M gates
• 2.5Gbits of memory
• 2 square feet of board
• 200W
• 20k cost

Backplane
6
1. Optics and routers dont belong
togetherCMOS and router architectures will
scale just fine

• Growth in capacity of electronic routers
• Capacity 1992 2Gb/s
• Capacity 1995 10Gb/s
• Capacity 1998 40Gb/s
• Capacity 2001 160Gb/s
• Capacity 2003 1-40Tb/s
• Main techniques for increasing capacity in
  electronic routers
• Separating linecards from switch cores.
• Parallelism and load-balancing.

7
Current 3rd generation Routers
Switched Backplane
Line Interface
CPU
Memory
Typically lt160Gb/s aggregate capacity
8
3rd Generation RoutersQueueing Structure
Switch
Arbiter
Per-flow/class or per-output queues (VOQs)
Per-flow/class or per-input queues
9
3rd Generation Routers
Size-constrained 19 or 23 wide. Power-constrain
ed 5kW for 640Gb/s is typical.
7
19 or 23
10
Separating linecards from switch cores 4th
Generation Routers/Switches
Optical links
Switch Core
Linecards
0.3 - 10Tb/s routers in development
11
1. Optics and routers dont belong
togetherCMOS and router architectures will
scale just fine

• Growth in capacity of electronic routers
• Capacity 1992 2Gb/s
• Capacity 1995 10Gb/s
• Capacity 1998 40Gb/s
• Capacity 2001 160Gb/s
• Capacity 2003 1-40Tb/s
• Main techniques for increasing capacity in
  electronic routers
• Separating linecards from switch cores.
• Parallelism and load-balancing.

12
Parallelism and Load-Balancing

• Techniques in development for linecards at 10s
  of Gb/s, but not discussed here
• Parallel packet buffers.
• Parallel lookup tables.
• Discussed here
• Load-balancing across multiple parallel routers.

13
Multiple parallel routers
IP Router capacity 100s of Tb/s
Big Router
R
R
NxN
R
R
14
Multiple parallel routers Load Balancing
architectures
R
R
R
15
Method 1 Random packet load-balancing

• Method As packets arrive they are randomly
  distributed, packet by packet over each router.
• Advantages
• Almost unlimited capacity
• Load-balancer is simple
• Load-balancer needs no packet buffering
• Disadvantages
• Random fluctuations in traffic a each router is
  loaded differently
• Packets within a flow may become mis-sequenced
• It is not possible to predict the system
  performance

16
Method 2 Random flow load-balancing

• Method Each new flow (e.g. TCP connection) is
  randomly assigned to a router. All packets in a
  flow follow the same path.
• Advantages
• Almost unlimited capacity
• Load-balancer is simple (e.g. hashing of flow
  ID).
• Load-balancer needs no packet buffering.
• No mis-sequencing of packets within a flow.
• Disadvantages
• Random fluctuations in traffic a each router is
  loaded differently
• It is not possible to predict the system
  performance

17
Observations

• Random load-balancing Its hard to predict
  system performance.
• Flow-by-flow load-balancing Worst-case
  performance is very poor.
• If designers, system builders, network operators
  etc. need to know the worst case performance,
  random load-balancing will not suffice.
• (Conversely If they dont, then it will).

18
Method 3 Intelligent packet load-balancing

• Goal Each new packet is carefully assigned to a
  router so that
• Packets are not mis-sequenced.
• The throughput is maximized and understood.
• Delay of each packet can be controlled.
• We call this Parallel Packet Switching

19
Method 3 Intelligent packet load-balancingParal
lel Packet Switching
Router
R/k
R/k
1
rate, R
rate, R
1
1
2
rate, R
rate, R
N
N
k
Bufferless
20
Parallel Packet Switching

• Advantages
• Single-stage of buffering
• No excess link capacity
• kh a power per subsystem i
• kh a memory bandwidth i
• kh a lookup rate i

21
Example of an IP Router with Parallel Packet
Switching
10Tb/s router
R/k
R/k
1
160Gb/s
rate, R
1
1
2
160Gb/s
rate, R
1024
1024
16
Overall capacity 160Tb/s
22
1.Optics and routers dont belong
togetherSummary

• If optics cannot buffer or process packets, and
• If electronic CMOS-based routers can be built
  that are fast enough, then
• Why would anyone try and build an optical router?

23
There seem to be 3 opinions

• Optics and routers dont belong together
• Optics are ill-suited to packet switching.
• CMOS technology and architectural techniques will
  scale just fine.
• Optical circuit switches will kill off (core)
  routers
• Optical circuit switches are simpler and faster
  than routers.
• We dont need packet switching anymore.
• Optics and routers belong together
• Electronic switched backplanes will be replaced
  by optics.
• Leads to lower power, higher density routers.

24
2. Optical circuit switches will kill off (core)
routersOptical circuit switches are simpler and
faster than routers.

• A survey of available equipment suggests that,
  with electronics, you can build a circuit switch
  that has about 10x the capacity of a packet
  switch.
• This is because a packet switch requires lots of
  complex per-packet processing,
• While a circuit switch requires no per-packet
  processing.

25
Processing steps

• IP Router
• Per packet
• IP lookup.
• Update header CRC.
• Forward to correct output.
• Schedule departure.
• Per route
• Maintain routing entry.

• Circuit Switch
• Continuously
• Transfer bits, bytes, photons from input to
  output.
• Per circuit
• Establish circuit
• Remove circuit

26
Why its hard for capacity to keep up with link
rates
Packet processing Power
Link Speed
10000
1000
2x / 2 years
2x / 7 months
100
Fiber Capacity (Gbit/s)
10
1
1985
1990
1995
2000
0,1
TDM
DWDM
Source SPEC95Int David Miller, Stanford.
27
Instructions per packet
Instructions per packet
What wed like (more per-packet processing
features) More efficient use of links,
differentiated services, Multicast, Security,
time
28
Normalized number of instructions per packet
29
2. Optical circuit switches will kill off (core)
routersWe dont need packet switching anymore.

• Original reasons for packet switching no longer
  hold.
• There are new techniques, such MPLambaS, burst
  switching, and TCP Switching that all make it
  possible to use circuit switching in the core.
• Actually, most of the core is circuit switched
  already!

30
Original reasons for packet switching

• Efficient use of expensive links
• Circuit switching is rarely used for data
  networks, ... because of very inefficient use of
  the links Gallager.
• Resilience to failure of links routers
• For high reliability, ... the Internet was to
  be a datagram subnet, so if some lines and
  routers were destroyed, messages could be ...
  rerouted Tanenbaum.

Source Networking 101
31
Neither reason is true today

• Link capacity is abundant and under used
• Most links are unused due to lack of switching
  capacity.
• Most links are utilized lt 10.
• Utilization continues to decrease.
• Routers rarely fail
• They are designed for lt5s down-time per year.
• They take gt1min to recover when they do (circuit
  switches must recover in lt50ms).

32
How networking people think the Internet is
Router
33
How the Internet really is
35Bn
6Bn
Packet Switched (IP routers)
Circuit Switched (SONET)
34
How the Internet really is
Your Local CO
IP routers
Your Local CO
IP routers
SONET/SDH
35
2. Optical circuit switches will kill off (core)
routersSummary

• If the original rationale for packet switching no
  longer holds, and
• If circuit switching is inherently faster, and
  cheaper than packet switching, and
• If circuit switching is already working fine for
  most of the Internet already, then
• Packet switching doesnt appear to have a
  long-term future.

36
There seem to be 3 opinions

• Optics and routers dont belong together
• Optics are ill-suited to packet switching.
• CMOS technology and architectural techniques will
  scale just fine.
• Optical circuit switches will kill off (core)
  routers
• Optical circuit switches are simpler and faster
  than routers.
• We dont need packet switching anymore.
• Optics and routers belong together
• Electronic switched backplanes will be replaced
  by optics.
• Leads to lower power, higher density routers.

37
3. Optics and routers belong togetherElectronic
switched backplanes will be replaced by optics.

• The first step is already happening physical
  separation of linecards and switch cores.
• Optical switching is feasible.
• Scheduling/arbitration is hard.

38
Separating linecards from switch cores 4th
Generation Routers/Switches
Optical links
Switch Core
Linecards
39
Replacing the switch fabric with optics
Typical IP Router Linecard
Typical IP Router Linecard
Electrical

• Buffer
• State
• Memory

Lookup Tables

• Buffer
• State
• Memory

Lookup Tables
Packet Processing
Buffer Mgmt Scheduling
Packet Processing
Buffer Mgmt Scheduling
Optics
Optics
Physical Layer
Framing Maintenance
Physical Layer
Framing Maintenance
Buffer Mgmt Scheduling
Buffer Mgmt Scheduling
Req/Grant
Req/Grant

• Buffer
• State
• Memory

• Buffer
• State
• Memory

Candidate technologies MEMs, gratings, passive
optical couplers tunable lasers, holography,
But this is the difficult part
40
Architecture of most routers today
Scheduler
Per-output queues (VOQs)

• Scheduler picks new configuration each cell
  time (lt50ns for OC192).
• Scheduling decisions are complex
• Ideal algorithm
• O(N3) maximum weight bipartite matching
• Good algorithm
• O(N2) maximal size bipartite matching
• Requires speedup which reduces cell time.
• Scheduler chip is typically several million
  gates,
• It is hard to use a distributed algorithm.

The scheduler is often the bottleneck in the
system.
41
Overcoming the scheduler bottleneck

• Increase the internal cell size to reduce rate
  of arbitration and reconfiguration.
• Today 64B is common.
• Expect 100s or 1000s of bytes per cell Kar.
• Throughput is not affected.
• When does it become circuit switching?
• Eliminate the need for a scheduler
• Two-stage switch Chang.

42
Two-Stage SwitchBackground
Outputs
Inputs
Of course, real traffic is non-uniform and bursty.
Simple Round-Robin
It is known that if traffic is uniform and
non-bursty, Then a single stage, with virtual
output queues, and trivial round-robin (TDM)
scheduling, gives 100 throughput.
43
Two-Stage Switch
Internal Inputs
External Inputs
External Outputs
First Round-Robin
Second Round-Robin
Switch gives 100 throughput for non-uniform,
bursty traffic, without a scheduler or speedup!
44
An optical two-stage switch
1
Phase 2
2
3
45
3. Optics and routers belong togetherSummary

• Optical switches can replace electronic crossbar
  switches now,
• Arbitration requires
• Faster (compromised?) schedulers, or
• A 2-stage switch fabric.

46
So which will it be?

• Optics and routers dont belong together
• Optics are ill-suited to packet switching.
• CMOS technology and architectural techniques will
  scale just fine.
• Optical circuit switches will kill off (core)
  routers
• Optical circuit switches are simpler and faster
  than routers.
• We dont need packet switching anymore.
• Optics and routers belong together
• Electronic switched backplanes will be replaced
  by optics.
• Leads to lower power, higher density routers.