The Stable Paths Problem As A Model Of BGP Routing

1
The Stable Paths Problem As A Model Of BGP Routing
Timothy G. Griffin ATT
Research griffin_at_research.att.com http//www.resea
rch.att.com/griffin

• NJIT
• April 24, 2002

2
Outline
Part I The glue that holds the Internet
together interdomain routing with
The Border Gateway Protocol (BGP)
Part II A formal model of BGP routing
policies Joint work with Bruce
Shepherd and Gordon Wilfong (Bell Labs)
3
Architecture of Dynamic Routing
OSPF
BGP
AS 1
EIGRP
IGP Interior Gateway Protocol
Metric based OSPF, IS-IS, RIP,
EIGRP (cisco)
AS 2
EGP Exterior Gateway Protocol
Policy based BGP
The Routing Domain of BGP is the entire Internet
4
Technology of Distributed Routing
Link State
Vectoring

• Topology information is flooded within the
  routing domain
• Best end-to-end paths are computed locally at
  each router.
• Best end-to-end paths determine next-hops.
• Based on minimizing some notion of distance
• Works only if policy is shared and uniform
• Examples OSPF, IS-IS

• Each router knows little about network topology
• Only best next-hops are chosen by each router for
  each destination network.
• Best end-to-end paths result from composition of
  all next-hop choices
• Does not require any notion of distance
• Does not require uniform policies at all routers
• Examples RIP, BGP

5
The Gang of Four
6
Many Routing Processes Can Run on a Single Router
BGP
OS kernel
RIP Domain
OSPF Domain
Forwarding Table Manager
Forwarding Table
7
AS Numbers (ASNs)
ASNs are 16 bit values.
64512 through 65535 are private
Currently over 12,000 in use.

• Yale 29
• MIT 3
• Harvard 11
• Genuity 1
• ATT 7018, 6341, 5074,
• UUNET 701, 702, 284, 12199,
• Sprint 1239, 1240, 6211, 6242,

ASNs represent units of routing policy
8
Autonomous Routing Domains Dont Always Need BGP
or an ASN
Qwest
Nail up routes 130.132.0.0/16 pointing to Yale
Nail up default routes 0.0.0.0/0 pointing to Qwest
Yale University
130.132.0.0/16
Static routing is the most common way of
connecting an autonomous routing domain to the
Internet. This helps explain why BGP is a
mystery to many
9
ASNs Can Be Shared (RFC 2270)
AS 701 UUNet
AS 7046 Crestar Bank
AS 7046 NJIT
AS 7046 Hood College
128.235.0.0/16
ASN 7046 is assigned to UUNet. It is used
by Customers single homed to UUNet, but needing
BGP for some reason (load balancing, etc..) RFC
2270
10
How Many ASNs are there?
Thanks to Geoff Huston. http//www.telstra.net/ops
on June 23, 2001
11
AS Graphs Can Be Fun
The subgraph showing all ASes that have more than
100 neighbors in full graph of 11,158 nodes. July
6, 2001. Point of view ATT route-server
12
BGP Table Growth
Thanks to Geoff Huston. http//www.telstra.net/ops
/bgptable.html on August 8, 2001
13
Nontransit vs. Transit ASes
Internet Service providers (often) have transit
networks
ISP 2
ISP 1
NET A
Nontransit AS might be a corporate or campus
network. Could be a content provider
Traffic NEVER flows from ISP 1 through NET A to
ISP 2 (At least not intentionally!)
14
Selective Transit
NET B
NET C
NET A provides transit between NET B and NET
C and between NET D and NET C
NET A DOES NOT provide transit Between NET D and
NET B
NET A
NET D
Most transit networks transit in a selective
manner
15
Customers and Providers
provider
customer
Customer pays provider for access to the Internet
16
The Peering Relationship
Peers provide transit between their respective
customers Peers do not provide transit between
peers Peers (often) do not exchange
traffic allowed
traffic NOT allowed
17
Peering Provides Shortcuts
Peering also allows connectivity between the
customers of Tier 1 providers.
18
BGP-4

• BGP Border Gateway Protocol
• Is a Policy-Based routing protocol
• Is the de facto EGP of todays global Internet
• Relatively simple protocol, but configuration is
  complex and the entire world can see, and be
  impacted by, your mistakes.

• 1989 BGP-1 RFC 1105
• Replacement for EGP (1984, RFC 904)
• 1990 BGP-2 RFC 1163
• 1991 BGP-3 RFC 1267
• 1995 BGP-4 RFC 1771
• Support for Classless Interdomain Routing (CIDR)

19
BGP Operations (Simplified)
Establish session on TCP port 179
AS1
BGP session
Exchange all active routes
AS2
While connection is ALIVE exchange route UPDATE
messages
Exchange incremental updates
20
Four Types of BGP Messages

• Open Establish a peering session.
• Keep Alive Handshake at regular intervals.
• Notification Shuts down a peering session.
• Update Announcing new routes or withdrawing
  previously announced routes.

announcement
prefix attributes values
21
BGP Attributes
Value Code
Reference ----- -----------------------------
---- --------- 1 ORIGIN
RFC1771 2 AS_PATH
RFC1771 3 NEXT_HOP
RFC1771 4
MULTI_EXIT_DISC RFC1771 5
LOCAL_PREF RFC1771
6 ATOMIC_AGGREGATE
RFC1771 7 AGGREGATOR
RFC1771 8 COMMUNITY
RFC1997 9 ORIGINATOR_ID
RFC2796 10 CLUSTER_LIST
RFC2796 11 DPA
Chen 12
ADVERTISER RFC1863 13
RCID_PATH / CLUSTER_ID RFC1863
14 MP_REACH_NLRI
RFC2283 15 MP_UNREACH_NLRI
RFC2283 16 EXTENDED
COMMUNITIES Rosen ... 255
reserved for development
Most important attributes
Not all attributes need to be present in every
announcement
From IANA http//www.iana.org/assignments/bgp-par
ameters
22
Attributes are Used to Select Best Routes
192.0.2.0/24 pick me!
192.0.2.0/24 pick me!
192.0.2.0/24 pick me!
Given multiple routes to the same prefix, a BGP
speaker must pick at most one best route (Note
it could reject them all!)
192.0.2.0/24 pick me!
23
BGP Route Processing
Open ended programming. Constrain
ed only by vendor configuration language
Apply Policy filter routes tweak attributes
Apply Policy filter routes tweak attributes
Receive BGP Updates
Best Routes
Transmit BGP Updates
Based on Attribute Values
Best Route Selection
Apply Import Policies
Best Route Table
Apply Export Policies
Install forwarding Entries for best Routes.
IP Forwarding Table
24
Route Selection Summary
Highest Local Preference
Enforce relationships
Shortest ASPATH
Lowest MED
traffic engineering
i-BGP lt e-BGP
Lowest IGP cost to BGP egress
Throw up hands and break ties
Lowest router ID
25
Tweak Tweak Tweak

• For inbound traffic
• Filter outbound routes
• Tweak attributes on outbound routes in the hope
  of influencing your neighbors best route
  selection
• For outbound traffic
• Filter inbound routes
• Tweak attributes on inbound routes to influence
  best route selection

outbound routes
inbound traffic
inbound routes
outbound traffic
In general, an AS has more control over outbound
traffic
26
ASPATH Attribute
AS 1129
135.207.0.0/16 AS Path 1755 1239 7018 6341
Global Access
AS 1755
135.207.0.0/16 AS Path 1239 7018 6341
135.207.0.0/16 AS Path 1129 1755 1239 7018 6341
Ebone
AS 12654
RIPE NCC RIS project
135.207.0.0/16 AS Path 7018 6341
AS7018
135.207.0.0/16 AS Path 3549 7018 6341
135.207.0.0/16 AS Path 6341
ATT
AS 3549
AS 6341
135.207.0.0/16 AS Path 7018 6341
Global Crossing
ATT Research
135.207.0.0/16
Prefix Originated
27
AS Graphs Do Not Show Topology!
BGP was designed to throw away information!
28
AS Graphs Depend on Point of View
peer
peer
provider
customer
1
3
1
3
2
2
5
4
6
5
4
6
This explains why there is no UUNET (701) Sprint
(1239) link on previous slide!
29
Shorter Doesnt Always Mean Shorter
Mr. BGP says that path 4 1 is better
than path 3 2 1
In fairness could you do this right and
still scale? Exporting internal state would
dramatically increase global instability and
amount of routing state
Duh!
AS 4
AS 3
AS 2
AS 1
30
Shedding Inbound Traffic with ASPATH Padding Hack
AS 1
provider
192.0.2.0/24 ASPATH 2 2 2
192.0.2.0/24 ASPATH 2
Padding will (usually) force inbound traffic
from AS 1 to take primary link
backup
primary
customer
192.0.2.0/24
AS 2
31
Padding May Not Shut Off All Traffic
AS 1
AS 3
provider
provider
192.0.2.0/24 ASPATH 2 2 2 2 2 2 2 2 2 2 2 2 2 2
192.0.2.0/24 ASPATH 2
AS 3 will send traffic on backup link because
it prefers customer routes and local preference
is considered before ASPATH length! Padding in
this way is often used as a form of load balancing
backup
primary
customer
192.0.2.0/24
AS 2
32
COMMUNITY Attribute to the Rescue!
AS 3 normal customer local pref is 100, peer
local pref is 90
AS 1
AS 3
provider
provider
192.0.2.0/24 ASPATH 2 COMMUNITY 370
192.0.2.0/24 ASPATH 2
backup
primary
Customer import policy at AS 3 If 390 in
COMMUNITY then set local preference to 90 If
380 in COMMUNITY then set local preference
to 80 If 370 in COMMUNITY then set local
preference to 70
customer
192.0.2.0/24
AS 2
33
Hot Potato Routing Go for the Closest Egress
Point
192.44.78.0/24
egress 2
egress 1
IGP distances
56
15
This Router has two BGP routes to 192.44.78.0/24.
Hot potato get traffic off of your network as
Soon as possible. Go for egress 1!
34
Getting Burned by the Hot Potato
2865
High bandwidth Provider backbone
17
SFF
NYC
Low bandwidth customer backbone
56
15
San Diego
Many customers want their provider to carry the
bits!
tiny http request
huge http reply
35
Cold Potato Routing with MEDs(Multi-Exit
Discriminator Attribute)
Prefer lower MED values
2865
17
192.44.78.0/24 MED 56
192.44.78.0/24 MED 15
56
15
192.44.78.0/24
This means that MEDs must be considered
BEFORE IGP distance!
Note1 some providers will not listen to MEDs
Note2 MEDs need not be tied to IGP distance
36
Policies Can Interact Strangely(Route Pinning
Example)
backup
customer
1
2
Install backup link using community
3
Disaster strikes primary link and the backup
takes over
Primary link is restored but some traffic remains
pinned to backup
4
37
News at 1100h

• BGP is not guaranteed to converge on a stable
  routing. Policy interactions could lead to
  livelock protocol oscillations.
  See Persistent Route Oscillations in
  Inter-domain Routing by K. Varadhan, R.
  Govindan, and D. Estrin. ISI report, 1996
• Corollary BGP is not guaranteed to recover from
  network failures.

38
PART II
39
What Problem is BGP solving?
Can we model BGP?
X could

• aid in the design of policy analysis algorithms
  and heuristics
• aid in the analysis and design of BGP and
  extensions
• help explain some BGP routing anomalies
• provide a fun way of thinking about the protocol

40
Separate dynamic and static semantics
dynamic semantics
static semantics
41
An instance of the Stable Paths Problem (SPP)
2

• A graph of nodes and edges,
• Node 0, called the origin,
• For each non-zero node, a set or permitted paths
  to the origin. This set always contains the
  null path.
• A ranking of permitted paths at each node. Null
  path is always least preferred. (Not shown in
  diagram)

1
most preferred least preferred
When modeling BGP nodes represent BGP speaking
routers, and 0 represents a node originating
some address block
Yes, the translation gets messy!
42
A Solution to a Stable Paths Problem
2
2 1 0 2 0
A solution is an assignment of permitted paths
to each node such that
4 2 0 4 3 0

• node us assigned path is either the null path or
  is a path uwP, where wP is assigned to node w and
  u,w is an edge in the graph,
• each node is assigned the highest ranked path
  among those consistent with the paths assigned to
  its neighbors.

3 0
1 3 0 1 0
1
A Solution need not represent a shortest path
tree, or a spanning tree.
43
An SPP may have multiple solutions
1 2 0 1 0
1 2 0 1 0
1 2 0 1 0
2 1 0 2 0
2 1 0 2 0
2 1 0 2 0
First solution
Second solution
DISAGREE
44
Multiple solutions can result in Route
Triggering
1 0 1 2 3 0
1 0 1 2 3 0
primary link
2 3 0 2 1 0
2 3 0 2 1 0
backup link
3 2 1 0 3 0
3 2 1 0 3 0
Remove primary link
Restore primary link
45
BAD GADGET No Solution
Persistent Route Oscillations in Inter-Domain
Routing. Kannan Varadhan, Ramesh Govindan, and
Deborah Estrin. Computer Networks, Jan. 2000
46
SURPRISE Beware of Backup Policies
2 1 0 2 0
Becomes a BAD GADGET if link (4, 0) goes down.
2
4 0 4 2 0 4 3 0
4
BGP is not robust it is not guaranteed to
recover from network failures.
0
3
1
3 4 2 0 3 0
1 3 0 1 0
47
PRECARIOUS
Has a solution, but can get trapped
48
Solving an SPP
Just enumerate all path assignments And check
stability of each.

Exponential complexity
But, in worst case you (probably) cant do any
better
49
Use 3-SAT
Variables V X1, X2, , Xn
Clauses C1 X17 or X23 or X3,
C2 X2 or X3 or X12
. Cm X6 or X7 or X18
Question Is there an variable assignment
A V true, false such that
each clause C1, ,Cm is true?
3-SAT is NP-complete
50
Modeling assignment to variable X
X X 0 X 0
X true
X false
51
SPP Solvability is NP-complete
52
SPVP protocol
Pick the best path available at any given time
process spvpu receive P from w ?
rib-in(u?w) u P if rib(u) ! best(u)
rib(u) best(u) foreach v in
peers(u) send rib(u) to v

53
SPVP wanders around assignment space
assignment
solution
54
Distributed algorithms to solve SPP?

• OSPF-like
• Distribute topology, path ranks
• Solve SPP locally
• Exponential worst case
• How can loops be avoided when multiple solutions
  exist?
• RIP-like
• Pick the best path from the set of your
  neighbors paths, tell your neighbors when you
  change your mind
• Can diverge
• Not guaranteed to find a solution, even when one
  exists
• Even when converges, no bound on convergence time

This is BGP
55
A sufficient condition for sanity
If an instance of SPP has an acyclic dispute
digraph, then
Static (SPP)
Dynamic (SPVP)
solvable
safe (cant diverge)
unique solution
predictable restoration
all sub-problems uniquely solvable
robust with respect to link/node failures
56
Dispute Digraph
P
(u v)P (u v)Q ...
Q P ...
Q
Gives the dispute arc
57
Dispute Digraph (cont.)
P
(u,v)P ...
P ...
Gives the transmission arc
P
(u,v)P
58
Dispute Digraph Example
2 0
1 0
2 1 0
4 2 0
CYCLE
3 4 2 0
4 3 0
1 3 0
BAD GADGET II
3 0
59
What is to be done?
Static Approach
Dynamic Approach
Extend BGP with a dynamic means of detecting
and suppressing policy-based oscillations?
Inter-AS coordination
Automated Analysis of Routing Policies
These approaches are complementary
60
Some Applications SPP Theory

• A Safe Path Vector Protocol. Timothy G. Griffin,
  Gordon Wilfong. INFOCOM 2001
• Dynamic solution for SPVP based on histories
  (dynamically constructed dispute cycles).
• Inherently safe backup routing with BGP. Lixin
  Gao, Timothy G. Griffin, Jennifer Rexford.
  INFOCOM 2001
• Show that if customer/provider peer/peer model is
  followed, then all is well,
• Show that this can be exteded with complex backup
  policies and remain safe.
• Analysis of cold potato routing problems (MED
  oscillation). Griffin and Wilfong. Work in
  progress
• MED requires a modification to SPP model
• Analysis of Internal BGP (IBGP) configuration.
  Griffin and Wilfong. Work in progress.

61
A Few Research Topics

• Dynamic Behavior of BGP
• Convergence time, message overhead
• BGP Security
• S-BGP defined, but not deployed. Is it a good
  solution.
• Need an interdomain trust model
• Beyond BGP?
• When will it break? What will replace it?

62
Selected Papers on BGP Sanity

• Persistent Route Oscillations in Inter-Domain
  Routing. Kannan Varadhan, Ramesh Govindan, and
  Deborah Estrin. Computer Networks, Jan. 2000.
  (Also USC Tech Report, Feb. 1996)
• Shows that BGP is not guaranteed to converge
• An Architecture for Stable, Analyzable Internet
  Routing. Ramesh Govindan, Cengiz Alaettinoglu,
  George Eddy, David Kessens, Satish Kumar, and
  WeeSan Lee. IEEE Network Magazine, Jan-Feb 1999.
• Use RPSL to specify policies. Store them in
  registries. Use registry for conguration
  generation and analysis.
• An Analysis of BGP Convergence Properties.
  Timothy G. Griffin, Gordon Wilfong. SIGCOMM 1999
• Model BGP, shows static analysis of divergence in
  policies is NP complete
• Policy Disputes in Path Vector Protocols. Timothy
  G. Griffin, F. Bruce Shepherd, Gordon Wilfong.
  ICNP 1999
• Define Stable Paths Problem and develop
  sufficient condition for sanity
• A Safe Path Vector Protocol. Timothy G. Griffin,
  Gordon Wilfong. INFOCOM 2001
• Dynamic solution for SPVP based on histories
• Stable Internet Routing without Global
  Coordination. Lixin Gao, Jennifer Rexford.
  SIGMETRICS 2000
• Show that if certain guidelines are followed,
  then all is well.
• Inherently safe backup routing with BGP. Lixin
  Gao, Timothy G. Griffin, Jennifer Rexford.
  INFOCOM 2001
• Use SPP to study complex backup policies

63
Pointers

• SIGCOMM 2001 Tutorial on BGP
• http//www.research.att.com/griffin/sigcomm2001_b
  gp_tutorial/abstract.html

• Links on Interdomain routing and BGP
• http//www.research.att.com/griffin/interdomain.h
  tml

• Papers on BGP theory
• http//www.research.att.com/griffin/bgpresearch.h
  tml

griffin_at_research.att.com
64
Thank You!