CCNP 4

1
CCNP 4

• Layer 3 Troubleshooting

2
Common OSPF Problems

• Hello/Dead Interval MismatchAuthentication type
  and key
• Mismatch in area ID
• debug ip ospf adj
• show ip ospf neighbor

3
Mismatched Stub/Transit/NSSA Area Options

• When OSPF exchanges Hello packets with a
  neighbor, one of the things that it exchanges is
  an optional capability represented by 8 bits.
• One of the option fields is for the E bit, which
  is the OSPF stub flag. When the E bit is set to
  0, the area with which the route is associated is
  a stub area, and no external LSAs are allowed in
  this area.
• If the E bit does not match on both sides, an
  adjacency will not be formed. This is called an
  optional capabilities mismatch.
• debug ip ospf adj

4
OSPF Stuck in ATTEMPT

• This problem is only valid for NMBA networks in
  which neighbor statements have been defined.
• Stuck in ATTEMPT means that a router is trying to
  contact a neighbor by sending its Hello, but it
  has not received a response. The output of show
  ip ospf neighbor indicates this problem exists.
• The most common possible causes of the problem
  are
• Misconfigured neighbor statement
• Unicast connectivity is broken on NBMA, which
  could be due to a wrong DLCI, an access list, or
  NAT translating the unicast.

5
Show ip ospf neighbor
6
adjacencies

• Routers being neighbors does not guarantee an
  exchange of link state updates.
• Neighbor routers must form adjacencies in order
  for this to happen. Interface type plays a major
  role in how the adjacencies are formed. For
  example, neighbors on point-to-point links always
  become adjacent. Once a router decides to form an
  adjacency with a neighbor, it starts by
  exchanging a full copy of its link state
  database. The neighbor follows suit.

7
adjacencies

• After passing through several neighbor states,
  the routers will become adjacent. The following
  is a list of the different states that OSPF can
  be stuck in and some of the possible causes.

8
OSPF Stuck in ATTEMPT

• This problem is only valid for NMBA networks in
  which neighbor statements have been defined.
  Stuck in ATTEMPT means that a router is trying to
  contact a neighbor by sending its Hello, but it
  has not received a response.
• The output of show ip ospf neighbor indicates
  this problem exists.
• The most common possible causes of the problem
  are next slide

9

• Misconfigured neighbor statement
• Unicast connectivity is broken on NBMA, which
  could be due to a wrong DLCI, an access list, or
  NAT translating the unicast.

10

• OSPF Stuck in INITThe INIT state indicates that
  a router sees hello packets from the neighbor,
  but two-way communication has not been
  established. A Cisco router includes the router
  IDs of all neighbors in the INIT (or higher)
  state in the neighbor field of its hello packets.
  For two-way communication to be established with
  a neighbor, a router also must see its own router
  ID in the neighbor field of the hello packets
  coming from the neighbor router. The output of
  the show ip ospf neighbor command indicates that
  the router is stuck in INIT.

11

• The most common possible causes of this problem
  are
• An access list on one side is blocking OSPF
  Hellos.
• Multicast capabilities are broken on one side
  (switch problem).
• Authentication is enabled on only one side.
• The frame-relay map/dialer map statement on one
  side is missing the broadcast keyword.
• Hellos are getting lost on one side at Layer 2

12

• OSPF Stuck in 2-WAYThe 2-WAY state is an
  indication that the router has seen its own
  Router ID in the neighbor field of the neighbor
  hello packet. The reception of a Database
  Descriptor (DBD) packet from a neighbor in the
  INIT state will also cause a transition to 2-way
  state. The OSPF neighbor 2-WAY state is not a
  cause for concern.
• On networks that require a DR/BDR election to
  take place, if all routers are configured with
  priority 0, there will be no election process.
  All routers will remain in 2-WAY state, as the
  show ip ospf neighbor command will indicate.
  The solution is to make sure at least one router
  has an ip ospf priority of at least 1.

13
Show ip ospf neighbor
14
OSPF Stuck in EXSTART/EXCHANGE

• OSPF neighbors that are in EXSTART or EXCHANGE
  state are in the process of trying to exchange
  DBD packets. The router and its neighbor form a
  master/slave relationship. The adjacency should
  continue past this state. If it does not, there
  is a problem with the DBD exchange, such as MTU
  mismatch, or the receipt of an unexpected DBD
  sequence number.

15

• The most common possible causes of this problem
  are
• Mismatched interface MTU.
• Duplicate Router IDs on neighbors.
• Inability to ping across with more than certain
  MTU size.
• Broken unicast connectivity, which could be due
  to a wrong DLCI, an access list, or NAT
  translating the unicast.
• The debug ip ospf adj command can help diagnose
  these problems, as shown in Figure , which
  indicates an MTU interface mismatch.

16

• OSPF Stuck in LOADINGIn the LOADING state,
  routers send link-state request packets. During
  the adjacency, if a router receives an outdated
  or missing link-state advertisement (LSA), it
  requests that LSA by sending a link-state request
  packet. Neighbors that do not transition beyond
  this state are most likely exchanging corrupted
  LSAs. This problem is usually accompanied by a
  "OSPF-4-BADLSA" console message.
• Since this is not a common problem, it is
  recommend that the network administrator contacts
  the Cisco TAC.

17

• If a neighbor does not reply or a neighbor reply
  never reaches the local router, the router will
  also be stuck in the LOADING state. The most
  common possible causes of the problem are
• Mismatched MTU
• Corrupted link-state request packet
• The output of show ip ospf neighbor indicates
  that the R2 neighbor is stuck in LOADING.
• The debug ip ospf adj command can help diagnose
  these problems. Command output can indicate an
  MTU interface mismatch. Command output can also
  indicate a possible problem due to packet
  corruption.

18
debug ip ospf adj
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
unnumbered interfaces

• When OSPF creates a router LSA for point-to-point
  links, it adheres to the following rule to fill
  the Link ID and Link Data fields within the
  router LSA. The Link Data field for unnumbered
  point-to-point links should have a MIB-II Index
  value. Because the link data value of the
  numbered interface does not match that of the
  unnumbered interface, this creates a discrepancy
  in the OSPF database.

30
Area 0

• One of the functions of a Type 4 summary LSA is
  to announce the reacheability of an ASBR to other
  areas. The Type 4 LSA is not required if the ASBR
  exists in the same area. The ASBR does not
  generate the Type 4 summary LSA if it is not
  connected to area 0. To generate a summary LSA of
  Type 3 or Type 4, a router must have a connection
  to area 0. As a result, the external routes will
  not be installed in the network.

31

• Debugs and VerificationThe output of show ip
  ospf database external command in Figure shows
  that the route exists in the external OSPF
  database of router R1. However, the output of
  show ip ospf database asbr-summary
• The next logical step is to go on the ABR and see
  if it is indeed an ABR. If it is an ABR it will
  generate a summary LSA of Type 3 or Type 4. If it
  is not an ABR, it will not generate that summary.
  The output of the show ip ospf command in Figure
  shows that router R2, which is between two areas
  and may look like an ABR, is not identifying
  itself as an ABR. If R2 were an ABR, the output
  would display that is an Area Border Router.

32

• Note All areas in an OSPF autonomous system must
  be physically or virtually connected to the
  backbone area (area 0). For a router to be an
  ABR, one of its interfaces must be part of area
  0.
• SolutionIn this example, router R2 does not
  generate the Type 4 summary LSA because it is not
  connected to area 0. To generate a summary LSA of
  Type 3 or Type 4, a router must have a connection
  into area 0.
• To solve this problem, router R2 must be
  connected to area 0, either physically or
  virtually by creating a virtual link, as shown in
  Figure . By configuring a virtual link on R2,
  the router is now virtually connected to area 0
  therefore, it now considers itself an ABR. After
  connecting R2 to area 0, the output of show ip
  ospf shows that it is now an ABR.

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Route summarization

• Route summarization allows a group of contiguous
  networks to be summarized as fewer addresses in
  order to help reduce the size of the routing
  table, thus increasing the performance of OSPF
  and other routing protocols. OSPF can use two
  types of summarization
• Interarea route summarization that can be done on
  the ABR.
• External route summarization that can be done on
  the ASBR.

37
Neighbor formation process

• When OSPF forms an adjacency, it floods all of
  its link-state packets to its neighbor. This
  flooding sometimes takes a lot of CPU processing.
  Starting with Cisco IOS Software 12.0T, packet
  pacing was included to help prevent CPU
  processing problems from occurring. Prior to
  12.0T, the IOS did not support packet pacing.
• During link-state flooding a router will try to
  send data as fast as it can over a link. If a
  link is slow or the router on the other side is
  slow in responding, this results in
  retransmission of the LSA and eventually leads to
  CPUHOG messages. Packet pacing adds a pacing
  interval between the LS updates. Instead of
  flooding everything at once, packet pacing sends
  the packet with a gap of a few milliseconds in
  between

38
(No Transcript)
39

• LSA refresh processStarting with Cisco IOS 12.0,
  the LSA group pacing feature was introduced to
  eliminate a CPU processing problem that can occur
  every 30 minutes. Prior to IOS 12.0, all LSAs are
  flooded every 30 minutes to refresh the MAXAGE
  times in the link state database. Without this
  flooding, the LSAs would expire after 60 minutes.
  This flooding is also known as a paranoid update.
  
• This flooding causes CPUHOG messages every 30
  minutes, especially in cases where a couple of
  thousand LSAs are all being flooded at the same
  time. The CPUHOG messages appear in the router
  log every 30 minutes.

40

• LSA group packing looks at the LSA periodic
  interval (every 4 minutes, by default) and
  refreshes only those LSAs that are past their
  refresh time. This is an efficient way of
  reducing a large flood by chopping it down to
  smaller LSA floods. No extra configuration is
  required for this feature, but for large numbers
  of LSAs (generally 10,000 or more), it is
  recommended to use small intervals (for example,
  every 2 minutes) for a few hundred LSAs, use a
  large interval, such as 20 minutes.

41

• If 10,000 LSAs need to be refreshed, keeping the
  refresh interval smaller will check the LSA every
  2 or 4 minutes to see how many LSAs have reached
  the refresh interval, which is 30 minutes. The
  advantage of checking this frequently is that few
  LSAs would need to be refreshed every 2 or 4
  minutes, and this will not cause a huge storm of
  LSA updates. If the number of LSAs is small, it
  really does not matter whether the refresh occurs
  at 2 minutes or 20 minutes. This is why it is
  better to increase the timer so that all the
  LSAs, which are few in number, can be refreshed
  at once. Figure shows how to configure the LSA
  refresh interval.

42

• Whenever there is a change in topology, OSPF runs
  the SPF algorithm to compute the Shortest Path
  First tree again. Unstable links existing within
  the OSPF network could cause constant SPF
  calculations. Some of the most common reasons for
  SPF running constantly are
• Interface flaps within the area
• Neighbor interface flaps within the area
• Duplicate router ID

43

• Topology changes in an OSPF network cause SPF
  calculations within that area. The SPF is not
  recalculated if the topology change is in another
  area. Actually, OSPF distributes interarea
  (between areas) topology information using a
  distance-vector method. The ABRs forward routing
  information between areas using a distance vector
  technique similar to that of RIP or IGRP.
• The following is an example of SPF running
  constantly due to an interface flap within the
  network. This is a common problem in OSPF.
  Whenever there is a link flap in an area, OSPF
  runs SPF within that area. So, if a network has
  unstable links, it can cause constant SPF
  calculations. SPF itself is not a problem because
  OSPF is just adjusting to the change in the
  database through calculating SPF.

44

• Debugs and VerificationA link flap in an area
  causes SPF to run. If a link is flapping
  constantly, this can increase the number of SPF
  calculations in an area. A constant number of SPF
  calculations in not a problem, but if the number
  is incrementing constantly, it is an indication
  of a problem.
• The output of show ip ospf, shows that there is a
  huge counter for SPF in area 0.
• The easiest way to find out which particular LSA
  is flapping is to turn on debug ip ospf monitor.
  This debug command in Figure shows that a
  router LSA is flapping in area 0.
• The next step is to determine which router LSA is
  flapping and check the log for any interface
  flap. The show log command in Figure shows the
  log of the router with router ID 192.168.1.129,
  the ID displayed in the output of the debug ip
  ospf monitor command. The log shows that a serial
  link keeps going up and down. Whenever there is
  an interface flap, it causes SPF to run.

45

• SolutionThere are two solution to this problem
• Fix the link that is flapping.
• Redefine the area boundaries.
• Sometimes, the first solution might not be
  manageable because the link is flapping as the
  result of a telco outage that is outside the
  network boundary. One way to fix this temporarily
  is to manually shut down the interface.
• The second solution requires some redesigning. If
  the link flap is happening too often, it might be
  possible to redefine the area, exclude this
  router from the area, and make it a member of a
  totally stubby area. Sometimes, this is also
  difficult to implement, depending upon the
  physical location of this link.
• In short, link flaps are realities if there are
  too many link flaps, the number of routers in an
  area should be decreased so that fewer routers
  are affected.

46
BGP Neighbor Relationships

• Common Problems
• Directly connected external BGP neighbors not
  initializing
• Non-directly connected external BGP neighbors not
  initializing
• Internal BGP neighbors not initializing
• BGP neighbors (external and internal) not
  initializing

47
Directly connected external BGP neighbors not
initializing

• The autonomous system (AS) will not send or
  receive any IP prefix updates to or from a
  neighboring AS unless the neighbor relationship
  reaches the Established state, which is the final
  stage of BGP neighbor establishment.
• When an AS has a single EBGP connection, no IP
  connectivity can occur until BGP finalizes its
  operation of sending and receiving IP prefixes.

48

• The most common possible causes of directly
  connected external BGP neighbors not initializing
  are
• Layer 2 is down, preventing communication with
  directly connected EBGP neighbor.
• An incorrect neighbor IP address is in the BGP
  configuration.

49
BGP FSM

• BGP FSM includes six states
• Idle
• Connect
• Active
• OpenSent
• Open Confirm
• Established

Note These arrows should show pointing back to
the same state.
50
Idle State

• BGP always begins in the Idle state, in which it
  refuses all incoming connections.
• It is normally initiated by an administrator or a
  network event.
• When Start event occurs, the BGP process
• Initializes all BGP resources
• Starts the ConnectRetry timer
• Initializes a TCP connection the the neighbor
• Listens for a TCP initialization from the
  neighbor
• Changes its state to Connect

51
Connect State

• In this state, the BGP process is waiting for the
  TCP connection to be completed.
• If the connection is successful, the BGP process
• Clears the ConnectRetry timer
• Completes initialization
• Sends an Open message to the neighbor
• Transitions to the OpenSent state

52
Connect State

• If the connection is unsuccessful, the BGP
  process
• Continues to listen for a connection to be
  initiated by the neighbor
• Resets the ConnectRetry timer
• Transitions to the Active state

53
Active State

• In this state, the BGP process is trying to
  initiate a TCP connection with the neighbor.
• If the TCP connection is successful
• Clears the ConnectRetry timer
• Completes initialization
• Sends an Open message to the neighbor
• Transitions to the OpenSent state

54
Active State

• If the ConnectRetry timer expires while BGP is in
  the Active State, the BGP process
• Transitions back to the Connect state
• Resets the ConnectRetry timer
• In general, a neighbor state that is switching
  between "Connect" and "Active" is an indication
  that something is wrong and that there are
  problems with the TCP connection.
• It could be because of many TCP retransmissions,
  or the incapability of a neighbor to reach the IP
  address of its peer.

55
OpenSent State
In this state an Open message has been sent and
BGP is waiting to hear an Open message from its
neighbor.
errors
No errors

• When an Open message is received, all its fields
  are checked.
• If errors exist, a Notification message is sent
  and the state transitions to Idle.
• If no errors exist, a Keepalive message is sent
  and the Keepalive timer is set, the peer is
  determined to be internal or external, and state
  is changed to OpenConfirm.

56
OpenConfirm State
error
No errors

• In this state, the BGP process waits for a
  Keepalive or Notification message.
• If a Keepalive message is received, the state
  transitions to Established.
• If a Notification message is received, or a TCP
  disconnect is received, the state transitions to
  Idle.

57
Established State

• In this state, the BGP connection is fully
  established and the peers can exchange Update,
  Keepalive and Notification messages.
• If an Update or Keepalive message is received,
  the Hold timer is restarted.
• If a Notification message is received, the state
  transitions to Idle.

58
Bgp commands

• Show ip bgp summary
• Sho ip bgp neighbors
• Show ip bgp summary
• Sho ip bgp flap-statistics
• Debug ip bgp dampeningeventskeepalivesupdates
  

59
Nondirectly connected external BGP neighbors not
initializing

• In some cases, EBGP neighbors are not directly
  connected.
• BGP neighbor relationships can be established
  between routers trying to make an EBGP neighbor
  relationship that are separated by one or more
  routers. Such a neighbor relationship is termed
  EBGP multihop in Cisco IOS Software.

60
Non directly Connected

• Peering between loopbacks between EBGP typically
  is done when multiple interfaces exist between
  the routers, and IP traffic needs to be
  load-shared among those interfaces. Such a
  connection is considered nondirectly connected.
• The most common possible causes for nondirectly
  connected external BGP neighbors not initializing
  are

61

• The route to the nondirectly connected peer
  address is missing from the routing table.
• The ebgp-multihop command is missing in BGP
  configuration.
• The update-source interface command is missing.
• The show ip bpg summary and show ip bgp neighbors
  commands can be used to show that the neighbor
  relationship is in Active state.
• .

62

• In the case of ebgp-multihop command missing in
  the BGP configuration, these commands will show
  that the BGP neighbor is in Idle state, because
  no resources are allocated to the BGP neighbor.
  This might be because the other side has not
  received any BGP negotiation

63

• The solutions for the route to the nondirectly
  connected peer address is missing from the
  routing table is to either use a static route
  (common practice) to the connected peer address
  or a use an IGP dynamic routing protocol such as
  OSPF. This is usually an issue when peering is
  done between peers using loopback addresses.

64

• Internal BGP neighbors not initializingIBGP can
  experience similar issues to EBGP in neighbor
  relationship. IBGP is an important piece of
  BGP-running networks. The causes of IBGP neighbor
  relationship issues are identical to those of
  EBGP
• The route to the nondirectly connected IBGP
  neighbor is missing.
• The update-source interface command is missing in
  BGP configuration.
• The same troubleshooting and configuration
  techniques can be used for IBGP neighbor issues
  that are used for EBGP neighbor issues.

65

• BGP neighbors (external and internal) not
  initializingInterface access list/filters are
  another common cause of BGP neighbor activation
  problems. If an interface access list
  unintentionally blocks TCP packets that carry BGP
  protocol packets, the BGP neighbor will not come
  up.

66

• BGP requires the IP routing table to have an
  exact matching entry for the prefix that BGP is
  trying to advertise using the network and
  redistribute commands. The prefix and mask of the
  network that BGP is trying to advertise must be
  identical in the IP routing table and in the BGP
  configurations. Another cause to consider is BGP
  autosummarization of classful networks at major
  network boundaries. BGP might be trying to
  redistribute 100.100.100.0/24, but only
  100.0.0.0/8 gets advertised.
• In most cases it would be desirable to turn off
  autosummarization with the no auto-summary
  command, so that proper mask routes can be
  advertised to BGP neighbors

67

• Problem in propagating/originating a BGP route to
  IBGP/EBGP neighborsA scenario might arise in
  which the BGP configuration to originate and
  propagate routes looks good, but BGP neighbors
  are not receiving the routes. The originators
  BGP table shows all the routes. There is a
  possibility that configured distribute-list
  filters are the cause of the problem or a problem
  in the policy routing.

68

• Problem in propagating a BGP route to an IBGP
  neighbor but not to an EBGP neighborIn some
  cases, certain routes are not propagated to IBGP
  neighbors, but are propagated only to EBGP
  neighbors.
• When IBGP speakers in an AS are not fully meshed
  and have no route reflector or confederation
  configuration, any route that is learned from an
  IBGP neighbor will not be given to any other IBGP
  neighbor. Such routes are advertised only to EBGP
  neighbors.

69

• It is essential that IBGP-learned routes are
  propagated to other BGP speakers. BGP operators
  can use three methods to address this problem
• Use IBGP full mesh.
• Design a route-reflector model.
• Design a confederation model.

70

• Problem in propagating an IBGP route to an
  IBGP/EBGP neighborA problem might arise in which
  an IBGP learned route is not propagated to any
  BGP neighbor, whether it is an IBGP or EBGP
  neighbor.
• One case could be that when an IBGP-learned
  route is not synchronized, that route is not
  considered as a candidate to advertise to other
  BGP neighbors. A BGP route is synchronized only
  if it has been first learned through an IGP or
  static route.

71

• In Cisco IOS Software, BGP advertises only what
  it considers the best path to its neighbors. If
  an IBGP path is not synchronized, it is not
  included in the best path calculation. The output
  of the show ip bgp command will show
  unsynchronized routes in the BGP table.
• The solution would be to either turn off
  synchronization or make the routes synchronized
  by redistributing them in the IGP at the router
  that first introduced this route into the IBGP
  domain.

72

• A common problem is with BGP routes not getting
  installed into the IP routing table. If a router
  must forward an IP packet by looking at the IP
  destination address of the IP packet, the router
  must have an IP routing table entry for the
  subnet of the IP destination address. If the BGP
  process fails to create an IP routing table
  entry, all traffic destined for missing IP
  subnets in the routing table will be dropped.
  This is generic behavior of hop-by-hop IP packet
  forwarding done by routers.

73

• The most common causes of IBGP-learned routes not
  getting installed in the IP routing table are
• IBGP routes are not synchronized.
• The BGP next hop is not reachable.
• The most common causes of EBGP-learned routes not
  getting installed in the IP routing table are
• BGP routes are dampened.
• The BGP next hop is not reachable in the case of
  multihop EBGP.
• The multiexit discriminator (MED) value is
  infinite.
• .

74

• A common problem in BGP occurs when the next hop
  is not reachable. The cause of this problem,
  IBGP-learned route is not getting installed into
  the IP routing table, is most common in
  IBGP-learned routes where BGP next-hop address
  should have been learned through an IGP. Failure
  to reach the next hop is an IGP problem, and BGP
  is merely a victim. With BGP, when IP prefixes
  are advertised to an IBGP neighbor, the NEXT-HOP
  attribute of the prefix does not change. The IBGP
  receiver must have an IP route to reach this next
  hop.

75

• The real power of BGP is in managing IP traffic
  flows coming in and going out of the AS. BGP in
  general and Cisco IOS Software in particular
  offer a great deal of flexibility in manipulating
  BGP attributes LOCAL_PREFERENCE, MED, and so
  forth to control BP best-path calculation. This
  best-path decision determines how traffic exits
  the AS. With the large size of BGP networks
  today, it is crucial that BGP operators
  understand how BGP attributes should be managed.

76

• The real power of BGP is in managing IP traffic
  flows coming in and going out of the AS. BGP in
  general and Cisco IOS Software in particular
  offer a great deal of flexibility in manipulating
  BGP attributes LOCAL_PREFERENCE, MED, and so
  forth to control BP best-path calculation. This
  best-path decision determines how traffic exits
  the AS. With the large size of BGP networks
  today, it is crucial that BGP operators
  understand how BGP attributes should be managed.
• Some of the most common problems encountered in
  managing outbound traffic flow

77

• Multiple exit points exist, but traffic goes out
  through one or a few exit routers.
• Traffic takes a different interface from what is
  shown in the routing table.
• A multiple BGP connection exists to the same BGP
  neighbor, but traffic goes out through only one
  connection.
• Asymmetrical routing occurs and it causes a
  problem especially when NAT and time-sensitive
  applications are used.

78

• Just as in managing outbound IP traffic from an
  AS, Cisco IOS Software offers BGP operators
  configuration options to manage inbound traffic
  in an AS. It is important that inbound traffic
  from other autonomous systems be managed as well.
  If this does not happen, capacity of the network
  will not be fully utilized. This causes
  congestion in one part of the network while other
  parts are underutilized. The end result of this
  mismanagement of inbound traffic flow is sluggish
  throughput, slow round-trip times, and delays in
  IP traffic. Therefore, it is essential that all
  inbound BGP policies are check and configured
  correctly.

79

• Some of the most common problems in managing
  inbound IP traffic in an AS using BGP are
• Multiple connections exist to an AS, but all the
  traffic comes in through one BGP neighbor, in the
  same AS.
• A BGP neighbor in an AS should just be a backup
  provider, but some traffic from the Internet
  still comes through that AS.
• Asymmetrical routing occurs.
• Traffic to a certain subnet should come through a
  particular connection, but it is coming from
  somewhere else.
• Each of these problems, both with outbound and
  inbound policy

80
ip traffic

• Sho ip traffic
• Statistics such as format errors, bad hops,
  unknown routes
• Debug ip icmp
• Icmp transactions
• Debug ip packet
• General ip debugging

81

• Ip access-lists
• Ip helper-address

82
End system commands

• route add
• arp d
• ipconfig
• ifconfig
• winipcfg