How is the Internet Performing?

1
How is the Internet Performing?

• Les Cottrell SLAC
• Lecture 2 presented at the 26th International
  Nathiagali Summer College on Physics and
  Contemporary Needs, 25th June 14th July,
  Nathiagali, Pakistan

Partially funded by DOE/MICS Field Work Proposal
on Internet End-to-end Performance Monitoring
(IEPM), also supported by IUPAP
2
Overview

• Internet characteristics
• packet sizes, protocols, hops, hosts
• complexity, flows, applications
• Application requirements
• How the Internet worldwide is performing as seen
  by various measurements and metrics
• How well are requirements met?
• Many sources of measurements

CAIDA/Skitter PingER/IEPM
Matrix Surveyor
3
Packet size

• primarily 3 sizes
• close to minimumtelnet and ACKs, 1500 (max
  Ethernet payload, e.g. FTP, HTTP) 560Bytes for
  TCP implementations not using max transmission
  unit discovery

Mean 420Bytes, median 80Bytes
Measured Feb 2000 at Ames Internet eXchange
Packets
84M packets, lt 0.05 fragmented
Cu,mulative probability
Bytes
Packet size (bytes)
4
Internet protocol use

• There are 3 main protocols in use on the
  Internet
• UDP (connectionless datagrams, best effort
  delivery),
• TCP (Connection oriented, guaranteed delivery)
• ICMP (Control Message protocol)

TCP dominates today
SLAC protocol flows
ICMP
In
TCP
Flows/10min
UDP
Out
Time Feb-May 2001
5
Web use characteristics

• Size of web objects varies from site to site,
  server to server and by time of day.
• Typical medians vary from 1500 to 4000 bytes
• Also varies by object type, e.g. medians for
• movies few 100KB to MBs, postscript audio few
  100KB
• text, html, applets and images few thousand KB

Big peaks for error messages
Bytes
6
Hops

• Hop counts seen from 4 Skitter sites (Japan, S.
  Cal, N. Cal, E. Canada, i.e. 10-15 hops on average

Weak RTT dependence on hop count
95
RTT
50
5
Hops
Hop Count
7
Autonomous Systems (AS) Disperson

• Color indicates the AS responsible for the router
  at the hop, height is number of probes for that
  route
• Seen by Skitter at Palo Alto US (F root name
  server)

Hop number
8
Country dispersion

• Seen from Japan
• After 3 to 4 hops most goes to US.
• In some cases goes US back to jp
• Some goes to UK onto other European countries

Probes
Hops
9
Route maps

• Simple routes from TRIUMF, Canada to several
  sites already gets quite complex

TRIUMF
DESY
SLAC
UW
CERN
FNAL
KEK
10
Getting more complex

• PingER Beacon sites in US seen from TRIUMF,
  Vancouver (from Andrew Daviel, TRIUMF)

11
Connections by country
NL
Unknown
IT
RU
US
UK
JP
DE
12
Richness of connectivity

• Angle longitude of AS HQ in whois records
• Radius1-log(outdegree(AS)1)/(maxoutdegree 1)
• Outdegree number of next Hops As accepting
  traffic
• Deeper blue red more connections

• All except 1 of top 15 AS are in US, exception
  in Canada
• Few links between ISPs in Europe and Asia

13
Hosts by regions

• Jan 2001, 109 Million hosts
• Source Internet Software Consortium
  (www.isc.org)
• see web site also for hosts/population

• Notes
• Many .com are in N. America
• S. Asia in (36K), pk (6K), lk, bd
• E. Asiajp, cn, my, sg, tw, hk, th, id, bn, mm
• Mid Eastil, kw, lb, ae, tr, sa
• TLDs with hosts238
• Total TLDs258

14
Backbone utilization
Shows utilization of I2/Abilene backbone links,
NB Backbone lt 30 loaded Most losses at exchange
points edges
15
Flow sizes
SNMP
Real A/V
AFS file server
Heavy tailed, in out, UDP flows shorter than
TCP, packetbytes 75 TCP-in lt 5kBytes, 75
TCP-out lt 1.5kBytes (lt10pkts) UDP 80 lt 600Bytes
(75 lt 3 pkts), 10 more TCP than UDP Top UDP
AFS (gt55), Real(25), SNMP(1.4)
16
Flow lengths
Measured by Netflow flows tied off at 30 mins
TCP outbound flows
Active time in secs

• 60 of TCP flows less than 1 second
• Would expect TCP streams longer lived
• But 60 of UDP flows over 10 seconds, maybe due
  to heavy use of AFS at SLAC
• Another (CAIDA) study indicates UDP flows are
  shorter than TCP flows

17
Typical Internet traffic by Application

• CERFnet link
• Dominated by WWW (http)

Mail
WWW
FTP
RealAudio
18
SLAC Traffic profile
SLAC offsite links OC3 to ESnet, 1Gbps to
Stanford U thence OC12 to I2 OC48 to
NTON Profile bulk-data xfer dominates
HTTP
Mbps in
iperf
2 Days
Last 6 months
Mbps out
SSH
FTP
bbftp
19
SLAC Internet Application usage
Ames IXP approximately 60-65 was HTTP, about
13 was NNTP Uwisc 34 HTTP, 24 FTP, 13 Napster
20
What does performance depend on?

• End-to end internet performance seen by
  applications depends on
• round trip times
• packet loss
• jitter
• reachability
• bottleneck bandwidth
• implementation/configurations
• application requirements
• Data transmitted in packets

21
Application requirements

• Based on ITU Y1541
• The VoIP loss of 10-3 used to be 0.25 but that
  assumed random flat loss
• actual loss is often bursty
• Tail drop in routers
• Sync loss in circuits, bridge spanning tree
  reconfiguration, route changes

22
RTT from ESnet to Groups of Sites
RTT distance/(0.6c) hops router
delay Router delay queuing clocking in out
processing
ITU G.114 300 ms RTT limit for voice
20/year
23
RTT Region to Region
OK White 0-64ms Green 64-128ms Yellow
128-256ms NOT OK Pink 256-512ms Red gt 512ms
OK within regions, N. America OK with Europe,
Japan
24
RTT from California to world
Europe
E. Coast
Brazil
E. Coast US
W. Coast US
300ms
RTT (ms)
Europe S. America
0.30.6c
Longitude (degrees)
300ms
Frequency
Source Palo Alto CA, W. Coast
RTT (ms.)
Data from CAIDA Skitter project
25
RTT from Japan to world
RTT(ms)
Longitude
Seen from Japan
26
Cumulative RTT distributions

• Gives quality measure
• Seen from San Diego, US Skitter
• Steeper less jitter, i.e. better
• Small values better

Cumulative
RTT ms
27
Routes are not symmetric
Advanced to U. Chicago

• Min, 50 90 RTT measured by Surveyor
• Notice big differences in RTTs
• May be due to different paths in the 2 directions
  or to different loading

RTT ms
U. Chicago to Advanced
RTT ms
28
Loss seen from US to groups of Sites
50 improvement / year
ETSI DTR/TIPHON-05001 V1.2.5 threshold for good
speech
29
Detailed example of improvements
Increase of bandwidth by factor of 460 in 6
years, more than kept pace - factor of 50 times
improvement in loss
Note valleys when students on vacation
30
Loss to world from US
Using year 2000, fraction of worlds
population/country from www.nua.ie/surveys/how_ma
ny_online/
31
How are the U.S. Nets doing?
In general performance is good (i.e. lt 1) ESnet
holding steady, still better than others Edu
(vBNS/Abilene) .com improving
32
Losses for 28 days in May 2001
DNS
Loss
WWW
Internet
ISP

• Measured by MIDS to 583 DNS services, 383 Web
  services, 1367 Internet (ping) hosts, 1225 ISPs
  (routers)

33
Losses between Regions
34
Bulk throughput

• Important for long TCP flows where we want to
  copy large amounts of data from one site to
  another in a relatively short time, e.g. file
  transfer
• Depends on RTT, loss, timeouts, window sizes

35
Throughput quality
TCPBW lt 1/(RTTsqrt(loss))
Note E. Europe catching up
Macroscopic Behavior of the TCP Congestion
Avoidance Algorithm, Matthis, Semke, Mahdavi,
Ott, Computer Communication Review 27(3), July
1997
36
Throughput also depends on window

• Optimal window size depends on
• Bandwidth end to end, i.e. min(BWlinks) AKA
  bottleneck bandwidth
• Round Trip Time (RTT)
• For TCP keep pipe full
• Window (sometime called pipe) RTTBW
• Can increase bandwidth by
• orders of magnitude
• If no loss Throughput Window/RTT

Src
Rcv
t bits in packet/link speed
RTT
37
Jitter from N. America to W. Europe
Jitter IQR(ipdv), where ipdv(i) RTT(i)
RTT(i-1) 214 pairs
ETSI DTR/TIPHON-05001 V1.2.5 (1998-09) good
speech lt 75ms jitter
38
Jitter between regions
ETSI DTR/TIPHON-05001 V1.2.5 (1998-09)
125msMed
225msPoor
75msGood
Jitter varies with loading
39
SLAC-CERNJitter
40
Reachability
Within N. America, W. Europe loss, RTT and
jitter is acceptable for VoIP
41
Reachability Outage Probability
Surveyor probes randomly 2/second Measure time
(Outage length) consecutive probes dont
get through Heavy tailed outage lengths (packet
loss not Poisson)
http//www-iepm.slac.stanford.edu/monitoring/surve
yor/outage.html
42
Europe seen from U.S.
43
Asia seen from U.S.
44
Latin America, Africa Australasia
45
Animated monthly 2000
20 loss
Big is Bad
200ms RTT
20 unreachable
46
RTT worldwide from the Matrix
47
More Information

• IEEE Communications, May 2000, Vol 38, No 5, pp
  120-159
• IEPM/PingER home site
• www-iepm.slac.stanford.edu/
• CAIDA/Skitter home site
• www.caida.org/home/
• Matrix Net home site
• www.matrix.net/index.html
• Surveyor home site
• www.advanced.org/csg-ippm/