How to Multi-Home

1
How to Multi-Home

• Avi Freedman
• VP Engineering
• AboveNet Communications

2
What is Multi-Homing?

• Multi-homing is the process of selecting,
  provisioning, and installing a redundant
  connection to the Internet.
• Could be the same provider, or a different
  provider.

3
Why Multi-Home?

• Slow is 1,000,000 better than dead.
• You may be out of bandwidth.
• And
• Telco circuits die.
• Routers die.
• Providers networks fail.
• Different networks have better performance to
  different sites.

4
A Multi-Homed Architecture

• Ideally, take advantage of the opportunity to
  multi-home to remove all single points of failure
  in your network.
• Use -
• Multiple providers, unless your current provider
  will let you have cheap backup
• Multiple routers
• Multiple telco vendors

5
Multi-Homed Architecture

• Two routers, each with a different WAN connection
  from a different telco vendor.
• Use HSRP or VRRP internally to make both routers
  look like one virtual router.
• Eventually, multiple providers.
• Upcoming Boardwatch article with configs.

6
How the Internet Works

• Well, it breaks more than it works but when it
  does work -
• The Internet is a network of networks.
• Each network (called Autonomous System) on the
  Internet announces routes, which are lists of
  the IP addresses of the boxes on their network.
• You need to be able to send packets to, and get
  packets from, everywhere.

7
Inbound Traffic - Routes

• Routes are announced via BGP4 (the Border Gateway
  Protocol)
• Routers are announced to BGP peers.
• Each BGP peer can be a network peer or a
  transit peer.
• Network peers exchange just lists of customer
  routes.
• Each route is tagged by the ASNs it passes
  through.

8
Inbound Traffic - Routes

• So when AboveNet and UUNET peer, only AboveNet
  and UUNET routes are exchanged. No Sprint, PSI,
  etc...
• Transit peers -
• Announce to their customers all of the routes on
  the net (AboveNet, UUNET, Sprint, PSI, and the
  60,000 routes on the net).
• Announce to their peers all routes heard via
  transit.

9
Inbound Traffic - Routes

• So if you advertise 207.106.96.0/19 to AboveNet,
  -
• If youre a network peer, they only re-announce
  207.106.96.0/19 to customers (and use it
  internally)
• If youre a transit peer/customer, they announce
  207.106.96.0/19 to all of their network peers.
• Thats how you get global inbound reachability.

10
Address Space Issues

• Noone wants to hear a route for you unless -
• You are multi-homed (even then, some people dont
  want to hear routers), or
• You have your own direct IP space allocation from
  ARIN, RIPE, or APNIC.
• So, when youre single-homed without your own
  space, your IPs are reachable because theyre
  part of your providers aggregate block.

11
Address Space Issues

• For example, your provider has 207.8.128.0/17.
• You have 27.8.197.0/24 from them.
• Youre single-homed.
• The only route on the net for you is the
  207.8.128.0/17 route, originated by your
  providers ASN (and you dont have to do anything
  special).

12
Address Space Issues

• If you have your own CIDR block and are
  single-homed, your provider will originate it.
• So, if you have 219.190.64.0/19, itll be visible
  as an announcement by your provider, originated
  into the BGP mesh with your providers ASN as the
  origin.

13
Address Space Issues

• If you have your own IP space and want to
  multi-home, addressing issues are simple.
• Your other provider will start also originating
  your IP blocks.
• Or youll start speaking BGP, originate your IP
  blocks, and your providers will re-advertise them
  to the world.

14
Address Space Issues

• If you dont have your own IP space, its a bit
  more complicated.
• So, normally your ISP will only be advertising
  207.8.128.0/17 if you have 207.8.200.0/23.
• If youre multi-homed, your other provider will
  have to advertise 207.8.200.0/23.
• But so will your first provider.
• Why?

15
Address Space Issues

• Routes are chosen first by specificity.
• That is, to how many IP addresses they refer.
• The route covering the fewest IP is the most
  specific, and wins.
• (Otherwise default would always win and nothing
  would work.)

16
Address Space Issues

• So, if ISP 1 advertises only 207.8.128.0/17 and
  ISP 2 advertises only 207.8.200.0/23, all inbound
  traffic from the net will come in on ISP2.
• So, ISP 1 needs to blow a hole in their filters
  to leak the more specific 207.8.200.0/23 route.

17
Address Space Filtering

• Some ISPs do or did filter on routes smaller than
  (more specific than) /19s in gt 205.0.0.0 space.
• But it doesnt matter as long as your two
  upstreams have good connectivity.
• Why?

18
Address Space Filtering

• If Sprint doesnt see 207.8.200.0/23 from ISP1 or
  ISP2, theyll still see your providers
  207.8.128.0/17 route.
• So if your connectivity to ISP1 (the owner of
  207.8.128.0/17) goes down, all will be well as
  long as ISP1 still sees 207.8.200.0/23 from ISP2.
• Sprint -gt ISP1 -gt ISP2
• This is why people dont let you take IPs...

19
Load-Balancing Outbound

• You can use static default routes to control
  outbound packets.
• ip route 0.0.0.0 0.0.0.0 serial0/0
• ip route 0.0.0.0 0.0.0.0 serial1/0
• If theyre equal-cost (no metric at the end),
  itll load-balance based on destination, by
  default.

20
Load-Balancing Outbound

• Why load-balance based on destination?
• For internal networking, sometimes
  per-packet-load balancing makes sense.
• But if youre trying to talk to England and one
  provider has a 60ms path and the other has a
  150ms path, packets will arrive out of order and
  TCP and UDP apps get unhappy and slow.

21
How it works, Single-Homed

• Outbound (easy)
• Use a default route to your provider.
• Inbound
• Your provider originates a large (aggregate) BGP
  route, and gives you some space from inside it
  and/or
• Your provider originates BGP routes for your
  ARIN/RIPE/APNIC CIDR blocks as well.

22
How it Works, Multi-Homed, Static

• Outbound (easy)
• Load-balance default routes to deal with outbound
  packets.
• Inbound
• Your providers both originate BGP routes for just
  the address space youre using, even if its out
  of one providers space and/or
• Your providers both originate BGP routes for your
  ARIN/RIPE/APNIC CIDR blocks as well.

23
How it Works, Multi-Homed, Static

• Special note
• When providers configure BGP for single-homed
  customers, they will generally nail up your
  routes (even your directly-issued) CIDR blocks,
  so that if your connection goes down and up and
  down and ..., they dont have to flap that route
  out to the whole Internet. This is a good thing.

24
How it Works, Multi-Homed, Static

• Special note (ctd)
• But you NEED to make sure, when youre
  multi-homed, that the providers are NOT nailing
  your routes up.
• Why?
• Because if they do, when one T1 goes down, that
  provider will still advertise you to the world,
  thus blackholing you.

25
How it Works, Multi-Homed, BGP

• Topic of next talk.
• You either load-balance outbound with statics, or
  take full routes from your providers (if you
  can).
• You originate advertisements under your ASN for
  your directly-issued CIDR blocks, AND for the
  parts of your providers space that youre using
  (with their permission).

26
The Transition Static Routing

• To transition
• Turn up the other T1/T3/Ethernet.
• Put IPs on the interface.
• Run tests end-end.
• Start load-balancing default to the new T1.
• Then, in the middle of the night, have the new
  provider start advertising your IP space. Make
  sure you have reachability to every other ISP you
  can think of afterwards.

27
The Transition Static Routing

• To transition (ctd)
• After testing it live, turn off your other
  transit pipes and make sure that, after a few
  minutes, you still have connectivity.

28
The Transition BGP Routing

• To transition
• Turn up the other T1/T3/Ethernet.
• Put IPs on the interface.
• Run tests end-end.
• Start load-balancing default to the new T1.
• Then, undo that and bring up a BGP session that
  permits no routes either way.
• Then start taking routes, and watch outbound
  traffic.

29
The Transition BGP Routing

• To transition (ctd)
• Then, start announcing your routes.
• Then, in the middle of the night, have your ISP
  take out the static route and BGP announcement
  they were making.
• Make sure your route is propagating.
• Test reachability.
• Turn off your other pipes.
• Test reachability.

30
BGP or no?

• Advantages of doing static -
• Cheaper/smaller routers (less true nowadays)
• Simpler to configure
• Advantages of doing BGP -
• More control of your destiny (have providers stop
  announcing you)
• Faster/more intelligent selection of where to
  send outbound packets.
• Better debugging of net problems (you can see the
  Internet topology now)

31
Same Provider or Multiple?

• If your provider is reliable and fast, and
  affordably, and offers good tech-support, you may
  want to multi-home initially to them via Frame,
  SMDS, or some backup path (slow is 1,000,000
  better than dead).
• Eventually youll want t multi-home to different
  providers, to avoid failure modes due to one
  providers architecture decisions.

32
Questions?

• avi_at_freedman.net
• inet-access mailing list

33

• Nailing routes