Steve Gibbard

1
Exchange Point Data Collection

• Steve Gibbard
• Packet Clearing House

2
Introduction

• What is PCH?
• Non-profit research and development organization.
• We research and help develop exchange points
  around the world.
• We also run the INOC-DBA phone system.
• Who am I?
• Working for PCH for the last year and a half.
• Also consulting for some ISPs.
• Previously senior network engineer at Cable and
  Wireless and at a local ISP in Detroit.

3
Outline of the talk

• What is an exchange quick overview?
• What Packet Clearing House does for exchange
  setup
• Research

4
What is an exchange?

• An exchange is a facility that allows ISPs to
  interconnect and exchange traffic with each
  other.
• Also known as peering.
• Two purposes
• Keeps local traffic local.
• Allows traffic to be exchanged for free.

5
In the US

• In the US, its often not cheaper for a local ISP
  to peer at an exchange than to buy transit.
  Theres also often no noticeable performance
  increase.
• Most big exchange points are for-profit
  commercial entities, with large (and expensive)
  staffs.
• Local ISPs still peer for a variety of reasons.
• They think it improves performance
• They think its still cheaper
• Sometimes it is.
• Even ISPs who only buy transit benefit
• Having a local exchange can keep prices down for
  their transit providers.
• Transit is much cheaper in areas that have a
  local exchange.
• Areas without local exchanges often see their
  traffic go hundreds of miles to get across town.
• Having the transit providers exchange traffic
  locally helps performance.

6
In developing countries

• PCH is working on a lot of exchange construction
  efforts.
• ISPs generally buy connectivity via satellite.
• Adds 500 ms to round trip time
• When going between two satellite-connected ISPs,
  adds up to a full second of latency.
• Adds even more latency if the other ends of the
  satellite links are in different parts of the
  world.
• Expensive, for limited bandwidth.

7
Traceroute between two sites in Nepal, via
Honolulu, Los Angeles, and Singapore

• 1 lt10 ms 10 ms lt10 ms
  gw.conference.sanog.org 169.223.0.1 --
  Kathmandu
• 2 270 ms 160 ms 100 ms
  gw-pck-sp.wlink.com.np 202.79.55.9
• 3 40 ms 50 ms 40 ms gw.wlink.com.np
  202.79.40.1
• 4 231 ms 80 ms 110 ms
  mahesh.wlink.com.np 202.79.32.60
• 5 892 ms 1131 ms 216.236.105.33
• 6 1643 ms 1221 ms 1533 ms 69.88.1.189 --
  Honolulu
• 7 3105 ms 1442 ms 1962 ms 216.236.111.25
• 8 1041 ms 912 ms 1432 ms
  hnl-edge-01.inet.qwest.net 67.129.94.1
• 9 1222 ms 1322 ms 1131 ms
  bur-core-01.inet.qwest.net 205.171.13.169 --
  Los Angeles
• 10 1062 ms 1031 ms 1022 ms
  lap-brdr-01.inet.qwest.net 205.171.213.46
• 11 1432 ms 2563 ms 203.208.168.185
• 12 1743 ms 1552 ms 921 ms 203.208.168.221
• 13 2784 ms 851 ms 1102 ms 203.208.182.133
• 14 1542 ms 1672 ms 1643 ms 203.208.172.138
  Singapore?
• 15 1572 ms 1222 ms 1342 ms 203.208.145.38
• 16 1251 ms 1122 ms 1432 ms 203.208.140.133
• 17 1432 ms 1542 ms 203.208.140.133
• 18 Request timed
  out.

8
Exchanges can help both in performance and price

• Allow some portion of traffic to be kept local
• Round trip times between local ISPs can be
  measured in milliseconds, rather than seconds.
• Cost of local fiber or local wireless link is
  less than that of satellite links.
• Still not free
• Somebody needs to maintain the exchange switch,
  and that somebody needs to be paid somehow.
• Need to connect to it somehow.
• Still much better to have some connectivity
  through the exchange than to send everything via
  satellite.

9
Traceroute between ISPs that peer locally

• 1 gw.sfo.gibbard.org (216.93.185.185) 0.214 ms
  0.161 ms 0.173 ms
• 2 border-core2-ge6-0.sfo2.servepath.net
  (69.59.136.17) 0.659 ms 0.218 ms 0.639 ms
• 3 paix.pch.net (198.32.176.249) 2.941 ms
  3.808 ms 2.408 ms
• 4 host.paix.pch.net (206.220.231.245) 2.521 ms
  2.468 ms

10
What we do at exchanges

• In general
• Consulting
• Exchange-construction mailing list.
• Equipment donation

11
What we do at exchanges (cont.)

• Route servers
• Routers that peer with everyone in the exchange
  and reannounce routes.
• One peering session gets you all the routes of
  other route server participants
• And gives all the other participants your routes.
• Out of favor in the US, due to a desire for local
  control.
• Popular in other parts of the world, where a lot
  ISPs just want to get all the local routes
  without having to do a lot of thinking about it.

12
Route server participation

• Route server participation is optional.
• Route server session we distribute your routes
  to the other route server participants and
  distribute their routes to you.
• Non-route server session we collect your routes
  and use them internally, and display in
  looking-glass but dont use BGP to redistribute
  them.

13
What we do at exchanges (cont.)

• Anycast
• Services with the same IP address, used in
  different places.
• DNS for a few ccTLDs
• .ke -- Kenya
• .mu -- Mauritius
• .bs -- Bahamas
• Can add Anycast DNS for other ccTLDs as
  requested.
• INOC-DBA SIP proxy
• INOC-DBA is a NOC hotline phone system

14
What we do at exchanges (cont.)

• Looking Glass (more on this under research)

15
Research

• Route Collection
• Traffic patterns

16
Route Collection

• Route collection
• What networks are reachable from which exchange
  points now?
• What networks were reachable from those exchange
  points at some point in the past?
• Useful data
• Mapping. Useful for research
• Network planning. Which exchanges are worth
  going into. How can you reach a certain hard to
  reach network? How can you get to a network that
  wont peer with you directly?

17
Route collection -- Methodology

• Cisco 1760 router as collector
• Can only pass around 20 Mb/s of traffic, which is
  ok.
• Holds 192 MB of memory, so can take full routing
  tables.
• Peer with every network thats willing to peer
  with us.
• Get just peering routes
• Easy for the peers same as peering with anybody
  else.
• Do this at exchange points around the world
• Biggest are PAIX in Palo Alto, Equinix in
  Ashburn, and LINX in London?
• Equipment in about 25 exchange points.
• Several of them have only a few peers.

18
Routing data (cont.)

• How we display the data
• Topology archive
• http//archive.pch.net
• Real time looking-glass
• http//lg.pch.net
• Not necessarily complete we only have data from
  those who peer with us.
• Added benefits to participating
• ISPs that participate get access to our Anycast
  services.

19
Looking glass main screen
20
Looking Glass list of peers
21
Looking glass routes from AS 6316 -- StarNet
22
Traffic patterns

• Where is traffic going from various countries
  now?
• Developing country traffic mostly going through
  Europe or North America on satellite links.
• What we want to find out
• How much of that traffic would stay local?
• Same city, same country, same region, etc.
• Other parts of the world

23
Traffic pattern methodology

• How were going to do this
• Gather Netflow feeds from a bunch of ISPs
• ISPs will need to agree to participate
• Collector software running at exchange points
• Various forms of existing collector software
• Old Agilent stuff
• MCI modified flow-tools.
• Commercial solutions including Arbor and Adlex
• Were going to start out using the MCI software.

24
Traffic pattern methodology (cont.)

• Manual process for now.
• Collect stats for end ASes.
• Sort end ASes by country -- use whois for this.
• Put everything into a spreadsheet and total
  country volumes.

25
Future directions

• Obviously, this methodology is inefficient.
• We now have programmers working on our own
  homegrown solution.
• Do the country stuff automatically
• Give individual ISPs ability to see their own
  stats.

26
What were looking for

• Exchange points we want to install collectors
• 4U of rack space
• Really tiny amount of transit

27
What were looking for (cont.)

• Service providers
• Peer with us, so we can show you exist
• Netflow feeds let us study your traffic
• Aggregated with other Netflow feeds from region.
• Destinations broken down by region, but not by
  contacted sites.
• Working on ways of getting the information back
  to the individual ISPs in near real time and
  greater detail, to help with peering decisions.

28
Thanks

• Bill Woodcock
• Gaurab Raj Upadhaya

29
Contact information

• Steve Gibbard
• scg_at_pch.net
• Peering questions peering_at_pch.net
• http//www.pch.net
• 1 510 528-1263