Title: Tracing Routes
1Tracing Routes
2Introduction to Trace Route
- Traceroute is the generic name of a program that
shows you the route over the network between two
systems, listing all the intermediate routers a
connection must pass through to get to its
destination. - Can help you determine why your connections to a
given server might be poor, and can often help
you figure out where exactly the problem is. - Shows you how systems are connected to each
other, letting you see how your gateway connects
to the Internet as well as how the target system
is connected.
3Traceroute Utilities
- Typically a command-line-invoked program, but is
often provided with a GUI front-end and is always
one of the tools in a network probing tool
box. - On a Unix system (including Mac OS X) traceroute
is run from the command line - traceroute
- In a Microsoft system, the command has a slightly
different spelling, due to the 8-character
limitations of earlier operating systems - C tracert
- There are also a number of visual routing
programs available. These provide geographical
information as a supplement to the IP/Domain name
information provided by the typical traceroute
program.
4Sample Traceroute
- root_at_Inside1 hunt-1.5 traceroute
216.239.51.101 - traceroute to 216.239.51.101 (216.239.51.101), 30
hops max, 38 byte packets - 1 192.168.1.254 (192.168.1.254) 1.175 ms
1.087 ms 1.068 ms - 2 216.249.153.254 (216.249.153.254) 3.052 ms
2.111 ms 2.049 ms - 3 192.168.169.254 (192.168.169.254) 2.528 ms
2.278 ms 2.273 ms - 4 192.168.70.254 (192.168.70.254) 3.630 ms
3.249 ms 3.095 ms - 5 216.249.175.253 (216.249.175.253) 4.780 ms
4.059 ms 4.075 ms - 6 10.0.1.125 (10.0.1.125) 13.872 ms 13.653 ms
12.055 ms - 7 lou-belknap-1-0-0-1-a.kec.net
(199.120.154.161) 13.497 ms 12.713 ms 14.064
ms - 8 sl-gw31-chi-1-1.sprintlink.net (160.81.92.13)
25.693 ms 26.044 ms 27.206 ms - 9 sl-bb21-chi-4-0.sprintlink.net
(144.232.26.29) 244.835 ms 91.989 ms 28.618 ms - 10 p4-0.sprint.chcgil06.us.bb.verio.net
(144.232.19.234) 25.931 ms 25.924 ms 25.773 ms - 11 p16-3-0-0.r02.chcgil01.us.bb.verio.net
(129.250.5.115) 27.203 ms 26.883 ms 25.841 ms - 12 p16-5-0-0.r01.chcgil01.us.bb.verio.net
(129.250.2.220) 28.527 ms 26.427 ms 26.869 ms - 13 p16-1-0-1.r20.asbnva01.us.bb.verio.net
(129.250.5.103) 54.239 ms 52.364 ms 52.459 ms - 14 p16-7-0-0.r02.asbnva01.us.bb.verio.net
(129.250.2.83) 52.859 ms 51.897 ms 51.353 ms - 15 ge-1-1.a00.asbnva01.us.ra.verio.net
(129.250.26.97) 52.701 ms 55.343 ms 52.642 ms - 16 ge-3-2.a00.asbnva01.us.ce.verio.net
(168.143.105.58) 52.092 ms 52.513 ms 52.877 ms - 17 216.239.48.77 (216.239.48.77) 51.610 ms
52.824 ms 52.076 ms
5Reading the Output
- Line number the TTL value
- rDNS or machine that responds If it fails, the
IP address is used. (this may be left out with
the -n option). - IP address that responds
- RTTs Three round trip times in milliseconds -
how long it took a packet to get from me to that
system and back again, called the latency between
the two systems.
6Partial Timeouts
- A timeout is indicated with an asterisk ()
- Sometimes, a line in the output may have one or
two of the times missing, with an asterisk where
it should be. For example, you might get a line
like this - sl-bb21-chi-4-0.sprintlink.net (144.232.26.29)
244.835 ms - In this case, the machine is up and responding,
but for whatever reason it did not respond to the
second and third packets. This is quite common.
Responses like this are most often computers,
rather than dedicated routers. For example,
systems running Solaris routinely show an
asterisk instead of the second RTT. - It's important to remember that timeouts are not
necessarily an indication of packet loss. This is
a common misconception, but since there are only
three probes, dropping one response is no big
deal.
7Full Timeouts
- 18 216.239.47.113 (216.239.47.113) 51.621 ms
52.797 ms 52.635 ms - 19 216.239.47.102 (216.239.47.102) 53.621 ms
52.013 ms 52.143 ms -
-
- etc.
- If your trace ends in all timeouts, this means
that the target system could not be reached. - The packets could not make it there and back
- May not be reaching the target
- May be reaching, but not returning
- May be an intentional block due to a firewall or
other security measures, and the block may affect
traceroute but not actual server connections.
8Error Indicators
- A trace can end with one of several error
indications indicating why the trace cannot
proceed. In this Linux example, the router is
indicating that it has no route to the target
host - 4 rbrt3.exit109.com (208.225.64.50) 35.931 ms
!H 39.970 ms !H - The !H is a host unreachable error message (it
indicates that an ICMP error message was
received). The trace will stop at this point.
Possible ICMP error messages of this nature
include - !H Host unreachable. The router has no route to
the target system. - !N Network unreachable.
- !P Protocol unreachable.
- !S Source route failed. You tried to use source
routing, but the router is configured to block
source-routed packets. - !F Fragmentation needed. This indicates that the
router is misconfigured. - !X Communication administratively prohibited.
The network administrator has blocked traceroute
at this router.
9TTL Warnings
- Sometimes, with some versions of traceroute, you
will see TTL warnings after the times - 6 qwest-nyc-oc12.above.net (208.185.156.26)
90.0 ms (ttl251!) 90.0 ms (ttl251!) 90.0 ms
(ttl251!) - This merely indicates that the TTL (time-to-live)
value on the reply packet was different from what
was expected. - This probably means that your route is
asymmetric. This is not shown by all versions of
traceroute, and can be safely ignored.
10Windows Output
- C\tracert www.google.com
- Tracing route to www.google.com 216.239.51.101
- over a maximum of 30 hops
- 1
- 2
- 3
- 4
- 5 10 ms 10 ms 10 ms 10.0.1.125
- 6 10 ms 10 ms 10 ms
lou-belknap-1-0-0-1-a.kec.net 199.120.154.161 - 7 20 ms 20 ms 30 ms
sl-gw31-chi-1-1.sprintlink.net 160.81.92.13 - 8 20 ms 30 ms 20 ms
sl-bb21-chi-4-0.sprintlink.net 144.232.26.29 - 9 20 ms 30 ms 20 ms
p4-0.sprint.chcgil06.us.bb.verio.net
144.232.19.234 - 10 20 ms 30 ms 20 ms
p16-3-0-0.r02.chcgil01.us.bb.verio.net
129.250.5.115 - 11 30 ms 30 ms 40 ms
p16-5-0-0.r01.chcgil01.us.bb.verio.net
129.250.2.220 - 12 50 ms 50 ms 50 ms
p16-1-0-1.r20.asbnva01.us.bb.verio.net
129.250.5.103 - 13 50 ms 50 ms 61 ms
p16-7-0-0.r02.asbnva01.us.bb.verio.net
129.250.2.83 - 14 50 ms 50 ms 50 ms
ge-1-1.a00.asbnva01.us.ra.verio.net
129.250.26.97 - 15 50 ms 50 ms 50 ms
ge-3-2.a00.asbnva01.us.ce.verio.net
168.143.105.58 - 16 50 ms 50 ms 50 ms 216.239.48.77
11Windows Output
- Different order of output fields
- RTTs first
- rDNS
- IP
- Notice that this trace was completed, unlike the
Linux traceroute above that never did complete.
This is due to the basic differences of how the
Linux and Microsoft trace programs work. If one
fails, it is probably a good idea to try the
other. - The Windows version does not show ICMP error
messages in the manner described above. Errors
are shown as (possibly ambiguous or confusing)
text. For example, a host unreachable error
will be shown as Destination net unreachable on
Windows.
12The Reverse Route
- Any connection over the Internet actually depends
on two routes the route from your system to the
server, and the route from that server back to
your system. These routes may be (and often are)
completely different (asymmetric). - Only the router interfaces on your side report
back to your traceroute - Unfortunately, there are no easy ways to check
the reverse route unless you have access to the
machine at the other end. - It is often helpful to try the trace from a
variety of network connections, if available. - Many systems have a web page where you can run a
traceroute from their system back to yours. In
combination with your trace to their system, this
can give you the other half of the picture. - For a list of web sites that allow a variety of
tracing facilities from their site (to yours or
other sites) see www.traceroute.org. - .
13Reverse Route Example
- The following is an example of one such site. In
the next two traces, I run a trace to University
of Maryland site, and then a reverse from that
site back to my machine
14Two Directions
205.168 - 205.171 Sprint, out of Denver,
CO 144.232 Sprint, out of Reston, VA
sl-bb23-rly-15-3.sprintlink.net (144.232.20.45)
Elkridge Maryland chi Chicago ewr Newark,
NJ dca Washington DC sl-st21-ash-6-0.sprintlin
k.net (144.232.19.17 Ashburn, VA
15Visual Route Site
- http//visualroute.visualware.com/
- a web site that allows you to trace from Dulles,
Washington DC to any place. It is an example of
the visualroute program which is shareware
and expires after 15 days, or costs 49 - http//www.jefferypsanders.com/traceit.html
- a site where you can download a program that will
let you build your own database. Freeware.
16Analyzing Timeouts
- Trace ends in Timeouts
- There could be a problem.
- A system is blocking traceroute attempts,
- The next step is to figure out where the problem
is. - The problem is no necessarily with the last hop
from which a reply was received - The source of the problem is either
- The connection between that system and the next
system on the route - The next system itself
- You may just have to wait for the problem to be
fixed, especially if the problem system is not at
your ISP and thus you aren't a paying customer of
that network. - The problem could lie somewhere on the return
route between the system giving the timeouts and
your own system, and that problem may not be
reflected in the previous parts of the trace
because the route may be entirely different.
17Example Timeout Problem
- 16 c1-pos5-3.snjsca1.home.net (24.7.66.77)
136.612 ms 129.795 ms 129.133 ms - 17 bb1-pos6-0-0.rdc1.sfba.home.net (24.7.72.18)
130.473 ms 137.609 ms 134.162 ms -
- The last reachable system on the route is at hop
17. - The problem may be with the system at hop 18
- The network connection between hops 17 and 18.
- The return route.
- The only way to tell is to see the reverse trace.
A reverse trace from hop 17 would be useful here
as well, to verify that the routes are indeed
different. - It may be difficult or impossible to obtain
traceroutes from those systems, because the
network administrator at home.net would have to
run them for you, and is probably too busy to
worry about such a request. - In this case, you can try running traces to the
target system from various other places. In the
above example, if you knew what router was
normally at hop 18 (from seeing it in previous
traces), you could try a trace to that router
from another site.
18Long Routes
- traceroute to 24.48.145.237 (24.48.145.237), 30
hops max, 40 byte packets - 1 main2-249-97.iad.above.net (209.249.97.3)
1.143 ms 0.559 ms 0.382 ms - 2 core1-main2-oc3-1.iad.above.net
(209.249.0.25) 0.574 ms 0.886 ms 0.429 ms - 3 sjc-iad-oc12-1.sjc.above.net (207.126.96.121)
82.134 ms 82.537 ms 82.158 ms - 4 sl-gw8-sj-0-1.sprintlink.net
(144.232.192.129) 82.523 ms 82.383 ms 82.949
ms - 5 sl-bb12-sj-6-0.sprintlink.net (144.232.3.109)
82.348 ms 82.762 ms 83.029 ms - 6 sl-bb10-sj-8-0.sprintlink.net (144.232.3.85)
83.346 ms 83.012 ms 83.006 ms - 7 sl-bb10-rly-6-0.sprintlink.net (144.232.9.13)
136.004 ms 135.804 ms 136.274 ms - 8 sl-bb6-dc-0-0-0.sprintlink.net
(144.232.7.170) 137.625 ms 137.204 ms 136.794
ms - 9 gip-dc-2-fddi1-0.gip.net (204.59.144.194)
137.344 ms 138.156 ms 139.390 ms - 10 gip-arch-1-atm2-0-0-132-atm.gip.net
(204.59.5.25) 311.850 ms 325.246 ms 285.607 ms - 11 gip-telehouse-1-atm0-0-0-333-atm.gip.net
(204.59.5.14) 281.472 ms 291.957 ms 314.661 ms - 12 gip-linx-fddi0.gip.net (204.59.2.198)
277.425 ms 297.364 ms 248.030 ms - 13 linx-gw1.UK.EU.net (195.66.224.90) 291.800
ms 213.447 ms 221.377 ms - 14 Nyk-nr01.NY.US.EU.net (134.222.228.158)
266.863 ms 301.220 ms 320.008 ms - 15 nyc-core-02.inet.qwest.net (205.171.17.9)
206.191 ms 233.207 ms - 16 nyc-core-03.inet.qwest.net (205.171.17.85)
235.085 ms 270.805 ms 252.668 ms - 17 nyc-core-01.inet.qwest.net (205.171.17.82)
281.931 ms 277.519 ms 278.152 ms - 18 wdc-core-02.inet.qwest.net (205.171.5.235)
265.548 ms 233.789 ms 219.698 ms
19High Latency
- As the latency of a connection increases
- Interactive response suffers
- Download speed can also suffer (remember TCP
windowing - Inherent Connection-type Latency
- Typical Modem latency 120-130ms.
- ISDN line 40-45ms.
- If you use a connection of this type, you won't
see any better than these numbers. - If you see a large jump in latency from one hop
to the next, that could indicate a problem. - It could be a saturated (overused) network link
- A slow network link
- An overloaded router
- A problem anywhere on the return route from the
high-latency hop as well. - A long hop, such as a cross-country link or one
that crosses an ocean - You can use ping to get a better idea of the
latency as well as the packet loss to a given
site or router - Traceroute only does three probes per router (by
default), which isn't a very good sample on its
own.
20Long Route Example
- C\tracert core1.London1.Level3.net
- Tracing route to core1.London1.Level3.net
212.113.2.65 - over a maximum of 30 hops
- 1 10 ms
- 2
- 3
- 4
- 5 10 ms 10 ms 10 ms 10.0.1.125
- 6 10 ms 10 ms 20 ms
lou-belknap-1-0-0-1-a.kec.net 199.120.154.161 - 7 20 ms 30 ms 20 ms
sl-gw31-chi-1-1.sprintlink.net 160.81.92.13 - 8 20 ms 20 ms 30 ms
sl-bb23-chi-4-0.sprintlink.net 144.232.10.49 - 9 20 ms 20 ms 20 ms
sl-bb22-chi-13-0.sprintlink.net 144.232.26.49 - 10 20 ms 30 ms 20 ms 144.232.8.186
- 11 20 ms 30 ms 20 ms
so-4-0-0.mp2.chicago1.level3.net 209.247.10.169 - 12 50 ms 60 ms 60 ms
unknown.level3.net 64.159.0.238 - 13 121 ms 120 ms 120 ms
so-2-0-0.mp2.london1.level3.net 212.187.128.154 - 14 120 ms 120 ms 120 ms
loopback0.core1.london1.level3.net 212.113.2.65 - Trace complete.
- Note the jump in delay time in line 13 for the
transatlantic hop.
21Trace to China
- C\tracert www.linkwan.com
- Tracing route to www.linkwan.com 202.96.137.89
- over a maximum of 30 hops
-
- 10 70 ms 60 ms 60 ms
sl-bb20-sea-8-2.sprintlink.net 144.232.18.49 - 11 60 ms 60 ms 70 ms
sl-bb21-tac-10-0.sprintlink.net 144.232.18.86 - 12 180 ms 190 ms 190 ms
sl-bb20-tok-15-3.sprintlink.net 144.232.19.242 - 13 180 ms 190 ms 190 ms
sl-bb21-tok-15-0.sprintlink.net 203.222.36.34 - 14 240 ms 241 ms 240 ms
sl-bb20-hk-14-2.sprintlink.net 203.222.33.6 - 15 230 ms 240 ms 241 ms 203.222.38.54
- 16 431 ms 440 ms 441 ms
sla-cnatel-1-0.sprintlink.net 203.222.39.38 - 17 481 ms 451 ms 491 ms 202.97.33.153
- 18 1282 ms 1292 ms 1292 ms
pos9-0-r1-c-gz-b.gd.cn.net 61.140.0.2 - 19 441 ms 440 ms 431 ms
pos1-0-r1-c-sz-c.gd.cn.net 202.105.1.246 - 20 430 ms 451 ms 431 ms 61.140.1.18
- 21 410 ms 431 ms 431 ms 61.144.236.34
-
22Tracing Private IP Ranges
- Private addresses should never be visible over
the Internet. But, sometimes you will see them in
traceroute output. If they appear within your
local network, this is okay private addresses
inside your own network can be visible to you.
If, however, they appear within someone else's
network, this can be problematic - 10 ebay-2-gw.customer.ALTER.NET (157.130.197.90)
114.204 ms 123.232 ms 120.957 ms - 11 10.1.2.5 (10.1.2.5) 110.693 ms 114.475 ms
107.747 ms - 12
- 13
- The private address 10.1.2.5 within another
network should not be visible to us. In this
case, though, it is the last visible address
before the trace ends in timeouts. - Visibility of private IP addresses doesn't
necessarily mean that the route does not work. It
is often simply the way the administrators of the
target network have set up their system. In fact,
the output above, despite the private IP address
and the timeouts, shows a route that works
perfectly well for web access. - However, a route which includes private addresses
is difficult to troubleshoot. You can't ping the
private routers to see if there is any packet
loss. You can't trace directly to them from other
sites. And in general, they show a certain level
of cluelessness in how the network is set up
23Route Looping
- 14 hou-core-03.inet.qwest.net (205.171.5.146)
165.484 ms 164.335 ms 175.928 ms - 15 hou-core-02.inet.qwest.net (205.171.23.5)
162.291 ms 172.713 ms 171.532 ms - 16 kcm-core-01.inet.qwest.net (205.171.5.201)
212.967 ms 193.454 ms 199.457 ms - 17 dal-core-01.inet.qwest.net (205.171.5.203)
206.296 ms 212.383 ms 189.592 ms - 18 kcm-core-01.inet.qwest.net (205.171.5.201)
210.201 ms 225.674 ms 208.124 ms - 19 dal-core-01.inet.qwest.net (205.171.5.203)
189.089 ms 201.505 ms 201.659 ms - 20 kcm-core-01.inet.qwest.net (205.171.5.201)
334.19 ms 320.39 ms 245.182 ms - dal-core-01.inet.qwest.net (205.171.5.203)
218.519 ms 210.519 ms 246.635 ms - We have learned about routing loops they are
the result of incomplete routing tables and
packets that try to come through before complete
convergence has occurred.
24Odd Traceroute Results
- 11 USW-phx-gw.customer.ALTER.NET (137.39.162.10)
142.840 ms 151.245 ms 129.564 ms - 12 206.80.192.221 (206.80.192.221) 127.569 ms
vdsla121.phnx.uswest.net (216.161.182.121)
185.214 ms - 13 vdsla121.phnx.uswest.net (216.161.182.121)
442.912 ms 205.956 ms 221.537 ms - 14 vdsla121.phnx.uswest.net (216.161.182.121)
164.728 ms 186.997 ms 190.414 ms - vdsla121.phnx.uswest.net (216.161.182.121)
306.964 ms 189.152 ms 221.288 ms - Hop 12 Responses from two different machines.
- Hops 13, 14, and 15 are all showing the same
address! Since the response times are actually
different, though, we can guess that they are, in
reality, different systems. The trace ends
normally at hop 15. - So what the heck is going on here? US West says
this is a security measure, to hide the details
of their internal network. The last few hops all
return the address of the end-user's ADSL line,
rather than their actual address. I'm not
entirely sure what kind of security this is
meant to provide. - Obviously, this makes any kind of troubleshooting
of this connection next to impossible. If you
encounter problems in this situation, the best
you can do is contact the network provider and
let them deal with it.
25How the Trace Program Works
- Send consecute IP packets with incrementing
time-to-live (TTL) values - Each time a packet passes through a router, its
TTL value is decremented by one when it reaches
zero, the packet is dropped, and an ICMP
Time-To-Live Exceeded error message is returned
to the sender. - Eventually either the final destination is
reached, or the maximum value (default is 30) is
reached and the traceroute ends. - At the final destination, a different error is
returned. Most traceroute programs work by
sending UDP datagrams to some random
high-numbered port where nothing is likely to be
listening. When that final system is reached,
since nothing is answering on that port, an ICMP
Port Unreachable error message is returned, and
we are finished. - The Windows version of traceroute uses ICMP Echo
Request packets (ping packets) rather than UDP
datagrams. - A few versions of traceroute, such as the one on
Solaris, allow you to choose either method
(high-port UDP or ICMP echo requests).
26Source Routing
- Normally IP routing is dynamic with each router
making a decision about which next-hop router to
send the datagram to. Applications have no
control of this, and are normally not concerned
with it. It takes tools such as Traceroute to
figure out what the route really is. - The idea behind source routing is that the sender
specifies the route. Two forms are provided - Strict source routing. The sender specifies the
exact path that the IP datagram must follow. If a
router encounters a next hop in the source route
that isn't on a directly connected network, an
ICMP "source route failed" error is returned. - Loose source routing. The sender specifies a list
of IP address that the datagram must traverse,
but the datagram can also pass through other
routers between any two addresses in the list. - Traceroute provides a way to look at source
routing, as we can specify an option allowing us
to force a source route, and see what happens. - The source route options are actually called
"source and record route" (LSRR and SSRR, for
loose and strict) since the list of IP addresses
is updated as the datagram passes along the path.
27How Source Routing Works
- 1. The sending host takes the source route list
from the application, removes the first entry (it
becomes the destination address of the packet),
moves all the remaining entries left by one
entry, and places the original destination
address as the final entry in the list. The
pointer is set to 1. - 2. Each router that handles the packet checks
whether it is the destination address of the
packet. If not, the packet is forwarded as
normal. In other words, this particular router
is not one of the ones in the loose source list
of IPs. - 3. If the router is the destination, and the
pointer is not yet greater than the length, then - 4. The next address in the list now becomes the
destination address, - 5. The IP address corresponding to the outgoing
interface replaces the source address just used,
and - 6. The pointer is incremented by 1.
28Source Routing Operation
29Loose-Source Routing Example
- First look at a regular trace between two lab
machines - root_at_Outside root traceroute 216.249.144.202
- traceroute to 216.249.144.202 (216.249.144.202),
30 hops max, 50 byte packets - 1 216.249.144.205 (216.249.144.205) 2.616 ms
2.058 ms 2.178 ms - 2 216.249.144.202 (216.249.144.202) 5.908 ms
!S 3.743 ms !S 3.684 ms !S - Now, lets force it to pass through the .210
machine - root_at_Outside root traceroute -g
216.249.144.202 216.249.144.210 - traceroute to 216.249.144.210 (216.249.144.210),
30 hops max, 50 byte packets - 1 216.249.144.210 (216.249.144.210) 1.256 ms
1.191 ms 1.177 ms - 2 216.249.144.205 (216.249.144.205) 2.616 ms
2.058 ms 2.178 mx - 3 216.249.144.202 (216.249.144.202) 5.908 ms
!S 3.743 ms !S 3.684 ms !S
30Source Routing in Real Life
- This is most likely what you will see these days
- root_at_Outside root traceroute -g 216.239.33.100
216.249.144.230 -n - traceroute to 216.249.144.230 (216.249.144.230),
30 hops max, 46 byte packets - 1 216.249.144.230 1.264 ms 1.207 ms 1.195 ms
- 2 216.249.153.254 2.597 ms !S 1.818 ms !S
1.804 ms !S - In this example I tried to specify a loose source
route and trace to www.google.com. You will
notice the !S on the first router leaving the
labs network it is configured like most
routers these days to not permit source routing
of any kind. The specified intermediate router
has already been tagged (the .230 machine), but
that doesnt matter to the .254 router. It is set
up to disallow any source routing, so it will not
forward any packets that have the IP option of
source routing enabled.
31Our Tracing Lab
- Compare Windows vs Linux traceroute utilities.
- How NATing effects tracing
- Look at how a firewall might effect tracing
both inbound and outbound - Take a look at source routing
- Learn a few tracing tricks
32Research/Presentation Opportunity
- Talk about the visual trace programs. What do
they offer? What are their differences? How are
they able to provide the geographical
information? - How are IPs related to geography? What changes
in this topic (if any) will be implemented with
the adoption of the IPv6 standards?