The Art of Network Troubleshooting

1
The Art of Network Troubleshooting

• How to Fix Any Problem

2
Two Dozen Simple Rules

• Were all smart people
• But we sometimes repeat old mistakes
• Or forget old lessons
• So lets examine and review some old
  troubleshooting techniques.
• Lest we forget again

3
Isolate the Problem

• C
• Si

4
Write Things Down

• Keep a notebook or written log
• Electronic PDA logs work well
• When you attack a problem, start an entry
• Values of logging
• You have a record when the problem reoccurs
• You buy yourself some time
• You wake up other parts of your brain when you
  write it down
• It forces you to express the problem

5
What Changed??

• What have you installed lately?
• What did your network admin change?
• What updates have you loaded?
• What AV software have you loaded?
• What anti-Spyware have you installed?
• What has been downloaded?
• Any hardware been added?
• Any software installations or upgrades of any
  kind?

6
Use Your References

• Odie
• Microsoft Knowledge Base
• Google It! / Google groups.
• www.eventid.net
• Other Vendors support sites
• Online forums
• www.windowsitpro.com

7
Record the Exact Error Message

• Look at Event Logs
• System and Application
• Write down Event ID and description
• Write down any error codes in message
• Use for knowledge base queries

8
Double-check Antivirus

• Users shut them off and forget to turn them back
  on
• Check DAT file dates
• http//housecall.antivirus.com can be used to
  scan hardware
• Ultimately the best AV tool is. . .
• A working brain

9
Wait 15 Minutes

• Microsoft favorite time interval
• WINs
• Server AD replication
• Group policies

10
Double check the obvious

• Are things plugged in?
• I mean really plugged in?
• Network cables, I/O, power
• Switches, hubs, routers

11
Assemble your Toolkit

• Software tools
• Hardware tools
• Vendor phone numbers
• List of URLs
• Service packs, patches
• Resource kit tools / Support tools
• BartPE
• Linux rootkit Virus checkers
• Always cold boot when using a bootable CD

12
Check IP Connectivity

• Ping
• Pathping
• Tracert
• Use tracert d ipnumber to avoid DNS
• If ICMP is filtered use portqry (KB 310099)
• Always ping the IP address instead of the DNS
  name.

13
Portqry Syntax

• Portqry n targetsystem options
• Options
• -e n try port number n
• -p tcp, -p udp, -p both protocol to use
• -r nm range of ports
• -o a,b,c,d list of ports
• -i dont reverse-resolve

14
Portqry Examples

• Check for a web server
• portqry n 10.0.0.2 e 80 -i
• portqry n 10.0.0.2 o 80,443 I
• Scan TCP ports 130-139
• portqry 10.0.0.2 r 130139 -i

15
No-Ping Ping Test

• C\gtping n 1 207.46.134.222
• Pinging 207.46.134.222 with 32 bytes of data
• Request timed out
• C\gtportqry n 207.46.134.222 e 80 I
• Querying target system called 207.46.134.222
• TCP Port 80 (http service) LISTENING

16
Separate the Name Resolution

• WINS vs. DNS
• For DNS use nslookup or dnslint
• For WINS use nblookup
• Dont forget about the local lookup files
• HOSTS
• LMHOSTS

17
Check the Logs

• Windows does not necessarily write error messages
  to the console
• Look at all the logs
• Consider enabling the security logs
• Search Microsoft for eventcombmt tool

18
Simplify the Problem

• Remove in between components
• Firewalls, AV, Anti-spyware
• Remove extraneous components
• extra protocols?
• Investigate binding order
• Remove name resolution by trying to access by IP
  number

19
Simplify the Problem (cont)

• Does turning something off make the problem go
  away?
• Whats the client scope? Single client? Group
  of clients? All clients?
• Whats the server scope? One server? All
  servers? Internet access?
• Are the failing machines related by network
  segment? by physical location? by Active
  Directory tree or subtree?

20
Hardware Breaks

• We tend to blame the software.
• Lightning, surges, heat, etc. can cause flaky
  hardware problems
• Corollary if something is going to break itll
  probably break when its new.
• Power issues can be back-breakers.
• Power bricks, low voltage, power cables, etc.

21
Best way to fix hardware

• All too often hardware doesnt die, it just gets
  a little sickly.
• The best and often only way to diagnose this is
  to swap it out.
• Spare switches, cables, are essential

22
Reboot!

• If you make a change and it doesnt take
• REBOOT
• For network devices like routers, modems,
  switches
• Turn them off, count to five, turn them on.
• To force GPOs it may take two or even three
  reboots.
• Windows Updates often require a reboot

23
Know Your Network

• Map and list IP addresses, MAC addresses, OSes,
  software revision levels
• Document WAPs, hubs, switches, routers
• Build the map when the network is
  functioning---BEFORE it breaks

24
Know What Normal Is

• Observe the network devices when things are good.
• What indicators are on?
• What should they look like?
• Take a digital picture of a working network
  device and keep them somewhere handy.

25
Make One Change at a Time

• The possibilities grow exponentially with
  multiple changes.
• With one change (A), it could only be due to A or
  something that was going to happen anyway (status
  quo (S))
• With two (A,B) it could be A, B, an AB
  interaction, or S
• With three A, B, C, AB, AC, CB, ABC, S
• And so forth. . .

26
Consider using a Network Monitor

• Windows Network Monitor
• Ethereal (www.ethereal.com)
• WildPackets
• You dont need to be an expert

27
Keep an External Address

• Ultimate test is whether you can reach the
  outside and they can reach you.
• An outside email address can also be helpful
  (hotmail, yahoo, gmail, etc.)

28
Check Security and Permissions

• Windows Rights as well as permissions
• Consider auditing processes to see if something
  cant run because of permissions.

29
Walk around the block and/or explain the problem
to someone.