KIOW 2002: Trends in Internet Exchange Point Development

1
KIOW 2002Trends in Internet Exchange Point
Development

• Version 1.0
• December, 2002
• Bill Woodcock
• Packet Clearing House

2

• Brief history of Internet exchange development
• Overview of Internet exchanges in the
  Asia-Pacific region
• What makes Korea unique
• Economic lessons to promote future development

3
Part 1 Brief history of Internet exchange
development
4
First Exchanges

• Metropolitan Area Ethernet
• Washington, D.C.
• 10mb shared FOIRL into assorted switches
• No fixed topology
• MFS fiber plant
• Shared administration

5
First Exchanges

• Commercial Internet Exchange
• Moved from Washington, D.C. to Palo Alto
• Layer-3 MMLPA
• DS1 (1.5mbps) lines into a Cisco 7010
• Not-for-profit industry association

6
First Exchanges

• MAE-West / Federal Internet Exchange
• San Jose / Mountain View
• FDDI dumbbell ring
• Bridged to 10mb Ethernet in many locations
• Two locations, two administrations

7
First Exchanges

• Hong Kong Internet Exchange
• Chinese University of Hong Kong
• Single location Ethernet switch
• Administered by the University
• First major free exchange

8
Technological Progression

• Shared 10Base-T / FOIRL Ethernet
• Switched 10mb Ethernet
• Shared FDDI
• Switched FDDI
• 100Base-T / 100Base-FX
• Gigabit Ethernet
• 10Gigabit Ethernet

9
Less-Successful Technologies

• Layer-3 route-servers
• Frame Relay
• ATM
• Wireless Ethernet
• Crossconnect mesh
• DPT

10
Common Services

• Route-server
• Looking-glass
• Measurement and instrumentation
• Network Time Protocol
• Web cache parent
• News server
• Root server mirror

11
Common Business Models

• Hosted by a university or government
• Informal
• Industry association
• Neutral for-profit
• Anything else may not be recognized

12
Size Differentiation

• Municipal
• Large metro-area
• National
• Regional (meaning changing)

13
Peering / Transit Differentiation

• New concept
• Very different pricing
• Very different competitiveness

14
Part 2 Overview of Internet exchanges in the
Asia-Pacific region
15
AP-Region Internet Exchanges
16
AP-Region Internet Exchanges
Seoul
17
AP-Region Internet Exchanges
Beijing Seoul
18
AP-Region Internet Exchanges
Beijing Seoul Tokyo
19
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka
20
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei
21
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong Kong
22
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong Kong Manila
23
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne
24
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland
25
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
26
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk
27
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar
28
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi
29
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu
30
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu Bangkok
31
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu Bangkok
Kuala Lumpur
32
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu Bangkok
Kuala Lumpur Singapore
33
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu Bangkok
Kuala Lumpur Singapore Jakarta
34
AP-Region Internet Exchanges
Beijing Seoul Tokyo Osaka Taipei Hong
Kong Manila Melbourne Auckland Wellington
Novosibirsk Ulaanbaatar Karachi Kathmandu Bangkok
Kuala Lumpur Singapore Jakarta Perth
35
Part 3 What makes Korea unique
36
Combination of Favorable Attributes

• High concentration of customers in a manageable
  geographical area
• High density of fiber in the ground
• Language not globally prevalent
• Fewer overseas viewers of hangul content
• Fewer Korean customers wish to access foreign
  content
• Economically rational behavior among service
  providers

37
Beneficial Results

• Peering in Seoul is inexpensive.
• Seoul has the highest volume of Internet traffic
  of any exchange in the world.
• Seoul has the highest ratio of local to
  international traffic of any major exchange.
• Seoul enjoys higher quality and lower prices than
  anywhere else in the world.

38
Part 4 Economic lessons topromote future
development
39

• Exchanges must be inexpensive, not reliable.
• There should be multiple exchange point
  operators, but only one switch fabric per
  geographic region.
• Open peering promotes growth closed peering
  causes stagnation.

40
Inexpensive, not Reliable

• All but about a dozen ISPs purchase transit.
• Thus all peering is simply an economic
  optimization versus transit. It exists simply to
  reduce the average per-bit cost. Shorter paths
  are a collateral benefit.

41
Inexpensive, not Reliable (2)

• The effectiveness of peering can be directly
  measured as a function of its reduction of
  per-bit cost versus transit.
• Thus making peering inexpensive is more
  beneficial to its effectiveness than making it
  reliable.

42
Inexpensive, not Reliable (3)

• Example
• Transit may cost 0.50/gigabit.
• Reliable (99.999, 26 sec/mon downtime) peering
  may cost 0.40/gigabit.
• Unreliable (99.9, 45 min/mon downtime) peering
  may cost 0.005/gigabit.

43
Inexpensive, not Reliable (4)

• Transit cost 500/megabit/second/month at 40
  utilization 0.50/gigabit.
• Reliable exchange cost 10,000/month for 100
  megabits/second at 15 utilization
  0.40/gigabit.
• Unreliable exchange cost 500/month for 100
  megabits/second at 50 utilization
  0.005/gigabit.

44
Inexpensive, not Reliable (5)

• Example ISP ships 10 terabits/month
  (approximately 40 megabits/second average)
• If exclusively by transit, 5,000/month.
• If 50.001 by transit and 49.999 by reliable
  exchange, 12,500.05/month.
• If 50.1 by transit and 49.9 by unreliable
  exchange, 3,005/month.
• Reliable exchange saves 4.95 of transit, but
  costs 9,500 extra each month.

45
Switch Fabric Fragmentation

• Every region needs exactly one switch fabric,
  where a region is defined as any area bounded by
  a step-function in the cost of backhaul.
• Additional switch fabrics damage connectivity,
  increase costs, and decrease value.
• Multiple exchange point operators increase
  price/performance options and increase value, as
  long as one fabric spans them all.

46
Switch Fabric Fragmentation (2)

• Example
• Fifty peers, with 100 routes each, peer at a
  single exchange (A) in a region. They pay
  500/month each to participate in the exchange.
• Each pays 0.102/month/route for peering.
• (500/month divided by 4,900 routes.)

47
Switch Fabric Fragmentation (3)

• A second, unconnected exchange (B) is started
  in the same region, offering service at
  400/month.
• Fifteen providers leave exchange A to join
  exchange B instead.
• Ten providers join B as well as A.
• Twenty-five providers remain just at A.

48
Switch Fabric Fragmentation (4)

• Cost Routes Cost/Route
• A only 25 500 3400 0.147
• B only 15 400 2400 0.167
• Both 10 900 4900 0.184
• Average 550 3400 0.161
• When a second unconnected exchange is added,
  costs double, or reachability is halved.

49
Switch Fabric Fragmentation (5)

• Cost Routes Cost/Route
• A only 11 500 3200 0.156
• B only 9 400 3000 0.133
• C only 8 600 2900 0.207
• A B 7 900 4100 0.220
• B C 6 1000 3800 0.263
• A C 5 1100 4000 0.275
• A, B C 4 1500 4900 0.306
• Average 706 3530 0.206
• When a third unconnected exchange is added,
• the effects become correspondingly worse.

50
Open Peering is the Only Effective Way to Create
Value

• Any one ISPs customers make up an
  insignificantly small portion of the Internet.
• The amount your customers are paying to reach
  your other customers is insignificant, relative
  to the amount theyre paying to reach everyone
  elses customers.

51
Peering Creates Value (2)

• The value which you as an ISP have to sell to
  your customers is the sum of the bandwidth at
  each of the exits of your network, weighted by
  the number of routes available through each.

52
Peering Creates Value (3)

• There are three ways to increase the value which
  you have to sell to customers
• Buy it (purchase transit)
• Sell it (sell transit) or
• Peer

53
Peering Creates Value (4)

• Purchasing transit is expensive. Although it is
  generally necessary and desirable to purchase
  some transit, economic optimization requires that
  it be used as little as possible.
• A network cannot survive by reselling transit
  alone, as that would be an unnecessary middleman
  position.

54
Peering Creates Value (5)

• Selling transit is necessary and desirable, as
  that increases the number and size of your
  customer base, the group of people who pay you
  money.
• However, you cannot sell as fast as the Internet
  grows overall, so the portion of the Internet
  which consists of your customers will decline
  over time.
• Thus selling transit is too slow a means of
  increasing value. It also constitutes a
  chicken-and-egg problem if you depend upon sales
  for growth of value, and depend upon growth of
  value to fulfill new sales, you cannot gain
  momentum.

55
Peering Creates Value (6)

• Adding peering bandwidth both costs less and can
  be achieved more quickly than adding either
  purchased or sold transit bandwidth, since its
  both geographically aggregated and temporally
  flexible.

56
Peering Creates Value (7)

• Switching from an open peering policy to a closed
  peering policy will necessarily retard the
  growth-rate of your network, both in absolute
  terms, and relative to your competitors who are
  growing through the addition of new peering
  bandwidth.
• No network has ever been profitable while
  pursuing a closed-peering strategy.

57
Collateral LessonPeering and Sale of Transit
are Complementary, not Mutually Exclusive

• There exist a set of related fallacious beliefs
  which cause innumerate people in this industry to
  lose money
• that it is not advantageous to peer with ones
  customers,
• that refusing to peer with another ISP can do
  them disproportionately more harm than it does
  ones self,
• and most ridiculously, that if you refuse
  someone peering, they might become your customer
  instead.

58
Collateral Lesson 1Peering with Customers is
Good

• Any peering increases the amount of bandwidth you
  have available to sell to your customers.
• If you peer with a customer, it increases the
  amount of bandwidth which you can sell to other
  customers.

59
Collateral Lesson 1Peering with Customers is
Good (2)

• Peering with a customer means offering them free
  routes to your other customers within the same
  region.
• If 0.1 of your traffic is between your own
  customers, and you peer with 10 of your
  customers, 0.001 of your traffic is between a
  pair of customers which are both also peers.

60
Collateral Lesson 1Peering with Customers is
Good (2)

• Thus by peering with 10 of your customers, you
  increase the bandwidth you have to sell to
  customers 0.1 of the time. In exchange, you
  either sacrifice payment for 0.001 of your
  traffic or need to create a new billing method
  for it. These numbers are both insignificant.
• Whats important is that it allows you to have a
  uniform peering policy without having to
  special-case an exception class.

61
Collateral Lesson 2Refusing Peering Hurts You
Both

• When an ISP refuses to peer with another ISP,
  both are hurt.
• ISPs which refuse to peer are generally failing
  to peer with a set of other ISPs which
  collectively advertise more routes than the ISP
  which is failing to peer.

62
Collateral Lesson 3If You Refuse Someone
Peering, You Create a Customer for your Competitor

• If ISP A refuses to peer with ISP B, two
  possibilities exist
• B will buy transit from one of As competitors
  to reach A, or
• B will peer with one of As transit providers
  to reach A.
• At best, A loses the possibility of selling B
  transit, and creates a customer for one of their
  own competitors.
• At worst, A loses the possibility of selling B
  transit, and has to pay to receive traffic which
  B can send for free in any volume.

63
Summary

• Only an inexpensive exchange can succeed.
• Only one switch fabric should exist in a metro
  area.
• Connections should be offered by multiple
  exchange point operators with different
  facilities at different price points.
• ISPs which wish to grow and be profitable must
  peer with everyone they can.

64

• Bill Woodcock
• woody_at_pch.net
• Critical contributions made by John Milburn,
• Pindar Wong and Dorian Kim are
• appreciatively acknowledged.
• www.pch.net/documents/papers/brief-history-of-ixes
  
• www.pch.net/documents/papers/intro-economics