NIRA: A New Internet Routing Architecture

1
NIRA A New Internet Routing Architecture

• Xiaowei Yang
• MIT CSAIL
• yxw_at_mit.edu

2
Why NIRA?

• Problems with todays routing system
• No user choice
• Does not scale well
• Continuing growth of global routing state
• No fault isolation
• NIRA solves these problems.

3
No User Choice!
Backbones

• Unlike in the telephone system, users cannot
  choose wide area providers separately from local
  providers.
• Pricing, quality of service, security
• Wide area providers cannot offer differentiated
  services directly to users.
• Quality of service

Comcast
Verizon
21 Million Broadband Subscribers in June 2003.
Duopoly / Monopoly
4
We Want to Let Users Choose Domain-Level Routes
ATT
UUNET

• Our hypothesis
• User choice stimulates competition.
• Competition fosters innovation.
• Validation requires market deployment.
• NIRA the technical foundation.

Local ISP
5
Problems with Current Inter-domain Routing

• No user choice
• Does not scale well
• Continuing growth of global routing state
• No fault isolation

6
Continuing Growth of Global Routing State
Courtesy of http//bgp.potaroo.net/

• Real world requirements such as multi-homing are
  not well supported.

7
No Fault Isolation

• Local failure causes global routing update.
• Routing loop and packet drops happen in
  transient state.
• Routing convergence takes on the order of
  minutes.

8
My Approach NIRA
9
Overview of NIRA

• A scalable architecture that gives users the
  ability to select routes.
• User is an abstract entity, e.g., software
  agent
• Domain-level choices
• Encourage ISP competition
• Individual domains decision to offer
  router-level choices

10
Design Overview (1) Route Discovery
Cindy
N11
N10
N12
N13
N14
N9
N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
X
N4
R1
R3
R2
N18
Alice
Bob
N1
N3
N2
11
Design Overview (2) Route Representation
Cindy
N11
N10
N12
N13
N14
N9
N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
N4
R1
R3
R2
N18
Alice
Bob
N1
N3
N2
12
Design Overview (3) Failure handling
Cindy
N11
N10
N12
N13
N14
N9
N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
N4
R1
R3
R2
N18
Alice
Bob
N1
N3
N2

• Will not discuss provider compensation in
  details.

13
Cindy
N11
N10
N12
N13
N14
N9
N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
N4
R1
R3
R2
N18
Bob
N1
N3
Alice
N2
14

System Components of NIRA

• Addressing
• Route discovery
• Name-to-Route mapping
• Route failure handling

15
NIRAs Addressing
Core
B2
B1
1/16
2/16
13/32 21/32

• Strict provider-rooted hierarchical addressing
• An address represents a valid route to the core.

11/32
12/32
R3
R2
R1
131/48 211/48
111/48 121/48
N3
122/48
N1
N2
1111000 1211000
1312000 2112000
16
Why is NIRAs Addressing Scalable?
Core
B2
B1
1/16
2/16
13/32 21/32

• Financial factors limit the size of core.
• Provider hierarchy is shallow.
• A domain has a limited number of providers.

11/32
12/32
R3
R2
R1
131/48 211/48
111/48 121/48
N3
122/48
N1
N2
1111000 1211000
1312000 2112000
17
Efficient Route Representation
Core
B2
B1
1/16
2/16
13/32 21/32
11/32
12/32
R1
R3
R2
131/48 211/48
111/48 121/48
122/48
N3
N1
N2
1111000 1211000
1312000 2112000

• A source and a destination address unambiguously
  represent a common type of route.
• General routes may use source routing headers.

18
Routers Forwarding Tables
Uphill table
Core
B2
B1
1/16
2/16

• Uphill table providers
• Downhill table customers, self
• Bridge table all others

13/32 21/32
11/32
12/32
Downhill table
R1
R3
R2
111/48 121/48
131/48 211/48
N3
122/48
N1
N2
1111000 1211000
1312000 2112000
19
Basic Forwarding Algorithm
Core
B2
B1
1/16
2/16
13/32 21/32
11/32

• Look up destination address in the downhill
  table. If no match,
• Look up the source address in the uphill table.

12/32
R1
R3
R2
131/48 211/48
111/48 121/48
N3
122/48
N1
N2
1312000 2112000
1111000 1211000
20

System Components of NIRA

• Addressing
• Route discovery
• Topology Information Propagation Protocol (TIPP)
• Name-to-Route mapping
• Failure handling

21
What does TIPP Do?
Core
B1
B2
X

• Propagates addresses
• Propagates up-graph providers and their
  inter-connections on a users routes to the
  core.

R1
R3
R2
N3
N1
N2
1111000 1211000
temporarily unusable
2211000
22
What is TIPP Like?

• Link-state style protocol
• Supports scoped propagation
• Provides a consistent view of network
• A simple, proven to be correct algorithm SG89
• No sequence numbers, no periodic refreshments, no
  timestamps
• Fast convergence

23
Why is TIPP Scalable? (1)
Cindy
N11
N10
N12
N13
N14
N9
N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
N4
R1
R3
R2
N18
Bob
N1
N3
Alice
N2

• Up-graph is small.

24
Why is TIPP Scalable? (2)
Cindy
N11
N10
N12
N13
N14
N9

• Scoped propagation ? fault isolation

N8
R7
R8
R9
R6
N7
N15
core
N6
B4
B3
R5
N16
R10
B2
B1
N5
R4
N17
X
N4
R1
R3
R2
N18
Bob
N1
N3
Alice
N2
25

System Components of NIRA

• Addressing
• Route discovery
• Name-to-Route mapping
• Failure handling

26
Name-to-Route Lookup Service (NRLS)
Foo.com server
Core
Alice.foo.com 1312000 2112000
B2
B1

• An enhanced DNS service

R1
R3
R2
N3
N1
N2
1111000 1211000
1312000 2112000
1312000 2112000
Alice.foo.com 1312000 2112000
27

System Components of NIRA

• Addressing
• Route discovery protocol
• Name-to-Route mapping
• Failure handling

28
How Route Failures are Handled
Foo.com server
Core
B2
B1

• A combination of TIPP notifications and router
  feedbacks or timeouts.
• Switching addresses to switch to a different
  route
• HIP Moskowitz04, SCTP RFC 2960, TCP migrate
  Snoeren00

X
X
R1
R3
R2
Alice.foo.com 1312000 2112000
N3
N1
N2
1111000 1211000
29
NIRA Solves these Problems

• User choice
• Choosing addresses ? choosing routes ? choosing
  providers
• Scalability
• Modularized route discovery.
• Constrained failure propagation.

30
Evaluation
31
Data Sets

• Domain-level topologies from BGP routing tables
• Inferred domain relationships Gao00
  Subramanian02
• Not completely accurate, but best practice.
• Data from 2001 to 2004 Agarwal

32
Evaluation of Scalability

• Methodology
• Measure the amount of state each domain keeps
  assuming NIRA
• Number of providers in the core
• Number of address prefixes
• Size of up-graphs
• Size of forwarding tables
• Conclusions
• Scalable in practice

33
The Internet Continues to Grow
34
Provider-rooted Hierarchical Addressing is
Practical.

• In practice
• Level of provider hierarchy (h) is shallow.
• A domain has a limited number (p) of providers.
• In theory ph

35
Up-graphs are Small.

• Analysis
• Level of provider hierarchy h
• Number of a domains providers p
• Number of a domains peers q
• ?i1h pi-1(pq)

36
Evaluation of TIPP in Dynamic Networks

• Methodology
• Packet-level simulations using sampled topologies
• Conclusion
• Low communication cost
• Fast convergence

37
Communication Cost of TIPP is Low.
Convergence No message churning

• Average total messages (bytes) / link / failure
• Maximum max seen messages (bytes) over one link
  / failure

Scalable scoped propagation
38
TIPP Converges Fast

• Link delay uniformly distributes in 10ms,
  110ms.
• Single failure convergence time is proportional
  to the shortest path propagation delay.

39
Conclusion

• User choice
• Choosing addresses ? choosing routes ? choosing
  providers
• Scalability
• Modularized route discovery.
• Constrained failure propagation.
• Evaluation shows NIRA is practical.
• Looking forward
• New provider compensation model
• Stable routing with user choice
• Deployment of NIRA

40
Core Routing Region is Scalable

• Financial factors limit the size of core.

41
Forwarding Tables

• No need to dynamically compute paths to reach a
  prefix.
• Common case
• Small
• Analysis
• Number of prefixes r
• Number of customers c
• Number of peers q
• r r rc q

42
TIPP in Dynamic Networks

• Simulation topologies
• Pick random leaf domains
• Recursively include their providers and peers

43
Workload Analysis of NRLS Update

• A fundamental tradeoff
• Topology change will cause address change
• Root servers reside in top-level providers.
• Route record updates mimic a renumbering event
• How often can route update happen?
• Route server processing time 1ms per update
• 1000 updates / second / server
• 100,000 updates ! 100 seconds 2 minutes
• Bandwidth 5 of 100Mb/s for 100,000 users, 100
  bytes per update, ! 625 updates / second ! 160
  seconds 3 minutes to update all users
• Route update causes contractual or physical
  topology change
• Could be scheduled
• Allow for a grace period, say 30 minutes
• Conclusion manageable

44
Architectural Problem in the Internet
Adds one entry into BGP tables
45

b000/16
a000/16
Core
B2
B1
a0001/32
a0002/32
a0003/32
b0001/32
R1
R3
R2
b00011/48
a00021/48
a00022/48
a00011/48
a00031/48
N3
N1
N2
b00011/96 a00031/96
a00011/96 a00021/96
a000111000 a000211000 1111N11000
b000111000 a000311000 1111N31000
Hierarchy address
Non-hierarchy address
46
Address Allocation

• A hierarchical address prefix is a leased
  resource.
• survives connection breakdown and node failures.
• periodic lease renewal.
• de-allocated when lease expires.

B
R
N
Established
a000/16
Address allocation decision making
a0001/32

R
a00011/48
N
Time
47
Edge Record Origination and Distribution
B
R
N
a000/16
a0001/32

R
Internal reachable B ? R a0001/32 External
reachable B ? R e
Internal reachable B? R a0001/32 External
reachable B ? R e Internal reachable R ? B
a000/32 External reachable R ? B
Topology update procedure
a00011/48
edge (B, R)
N
Time

• One directional attributes exchanged during
  connection setup.
• Topology update procedure distributes edge
  records to neighbors.

48
A Sampled Topology
49
Ensuring Edge Record Consistency

• Common techniques
• Sequence numbers (OSPF, IS-IS)
• Flooding
• Timestamps
• Loosely synchronized clocks
• Modified Shortest Path Topology Algorithm (SPTA)
  SG89
• Pros
• No sequence numbers or timestamps
• Simple, proven correct
• Cons
• Computation cost per update.
• Communication cost in a theoretical worst case
  is unbounded.

50
Scalable Route Discovery with Transit Policies
Net3
Net4

• Addressing must take policies into consideration.
• A natural solution provider-rooted hierarchical
  addressing Tsuchiya91 Francis94

Backbone1
Backbone2
X
ISP1
ISP2
peer
peer
Net1
Net2
provider
customer
51
Design Components and Requirements


X

• Design components
• Route discovery
• Route representation
• Route failure handling
• Provider compensation

• Design requirements
• Scalable
• Efficient
• Robust
• Heterogeneous user choice
• Practical provider compensation

52
Design Overview of TIPP

• Design focus simplicity and correctness
• Address allocation straightforward
• Topology information propagation tricky
• OSPF (RFC 2328) 244 pages, BGP (RFC 1771) 57
  pages
• Policy-controlled topology propagation
• Information hiding
• Scope
• Key design decisions
• Shortest Path Topology Algorithm SG89
• simple, proven to be correct
• No sequence numbers or timestamps
• Not guaranteed to discover all possible routes

53
Policy-controlled Topology Propagation
Core
B2
B1
1/16
2/16
13/32 21/32

• Edge record
• Originated by a domain
• Contains bidirectional attributes and policy
  specification
• An edge record is propagated based on policies.

11/32
12/32
R1
R3
R2
131/48 211/48
111/48 121/48
N3
122/48
N1
N2
1111000 1211000
1312000 2112000
Originator ID R3 Neighbor ID B2 Status
up Internal reachable (R3 ? B2) (0,
2/16) External reachable (R3 ? B2) Internal
reachable (B2 ? R3) (1, 21/32) External
reachable (B2 ? R3) e
54
Shortest Path Topology Algorithm
Ns summary of edge records
Network
R1
R2
(P1, P2, down) (R2, P1, down) (N, R2, up)
(P1, P2, up) (R1, P1, up) (N, R1, up)
N
(N, R1, up) (N, R2, up) (R1,P1, up) (R2, P1,
down) (P1, P2, up)
Trust news from the neighbor on the shortest
path.
Ns summary
55
Scalable Route Discovery

• Three system components
• A strict provider-rooted hierarchical addressing
  scheme
• TIPP a user learns his addresses and topology
  information.
• NRLS a user learns destinations addresses and
  optional topology information.
• Combining information from TIPP and NRLS, a user
  is able to select an initial route.
• TIPP a new protocol in NIRA

56
Analysis of Strict Provider-rooted Hierarchical
Addressing
Core
B2
B1
1/16
2/16
13/32 21/32
11/32
12/32

• Provider hierarchy is shallow.
• Pro a scalable core routing region.
• Pro scalable route discovery
• A valid route to a provider implies a
  policy-valid route to its customers.
• Con topology-dependent addresses
• Con multiple address selection

R1
R3
R2
131/48 211/48
22/32
111/48 121/48 221/48
N3
122/48
N1
N2
1111000 1211000 2211000
1312000 2112000
57
Provider Compensation

• Contractual agreements.
• Users cannot use arbitrary routes.
• Providers use policy checking to prevent
  illegitimate route usage.
• Direct business relationships
• Common case verify source address
• General case packet filtering
• Indirect business relationship end users pay
  remote providers
• Policy is made upon the originator or the
  consumer of a packet.
• An open and general problem. Working in progress
  with Jennifer Mulligan and David Clark.
• Various billing schemes are possible.
• Flat fee, usage-based billing.

58
Protocol Organization (TIPP)

• Connection Management

• Control plane

Idle
Connect
Active
OpenSent
open / address request
OpenConfirm
address request / address
AddreqConfirm
address / topology
AddrSynced
topology / keepalive
TIPP over a reliable transport connection
Established
keepalive
59
TIPP in Action
B
R
N
Established
1/16
B
Address allocation
11/32

R
Topology update
111/48
(B, R) up
N
Time
1111000, ready to use
60
Evaluation Checklist

• Scalability
• Size of core
• Number of address prefixes
• Number of link records
• Size of forwarding tables
• Forwarding efficiency
• Dynamic networks
• Communication cost of TIPP
• Convergence time of TIPP
• Route setup time with reactive failure detection
• Workload of NRLS

61
Scalable Route Discovery without Policy Routing
2.1.1
1
2.1
2
1.2
1.1.1

• A well-studied problem
• At the hearts of scalable routing
• An ingenious addressing scheme
• Nice theoretical bound O(logN) N the total
  number of nodes
• Cluster-based KK77, landmark routing
  Tsuchiya88

1.1
2.2
3
3.1.1
3.2
3.1
3.2.1