Network Diagnostics on vxWorks

1
Network Diagnostics on vxWorks

• SLS Controls Pikett Training

2
What if an IOC does not boot?

• Is the link on the boot PC wrong or not set?
• ll /ioc/XXX-VME-XXX
• Are the boot parameters wrong?
• Compare with cabling -_at_ XXX-VME-XXX
• Also compare with bootinfo XXX-VME-XXX
• Is the network connection (hardware) broken?
• Try ping ipaddress from the same subnet.
• No reply? Try to replace cable, or IOC or change
  port on switch.
• Maybe the error messages on boot PC help
• tail f /var/log/messages and reboot IOC

3
What if all CA clients disconnect?

• Does the IOC still react on keyboard input?
• For beamline IOC first login to beamline ssh
  beamline-gw
• Get IOC console rmc XXX-VME-XXX
• Hit ENTER to see a prompt XXX-VME-XXX gt
• If not, try CTRL-X to reboot, but first check
  with control room and/or beamline crew.
• Is the network connection (hardware) broken?
• Try ping ipaddress from the same subnet.
• No reply? Try to replace cable, or IOC or change
  port on switch.
• Use vxWorks network debug functions (next slides).

4
How to know what debug functions exist?

• Network debug functions
• netHelp
• NFS debug functions
• nfsHelp
• Other help functions
• help
• ioHelp
• dbgHelp

5
inetstatShow network connection status

• If Send-Q or Recv-Q stays at 4096, something
  hangs, maybe the partner on the other side of the
  network (Foreign Address) can't consume monitor
  data as fast as they come.
• On Linux, the equivalent command is netstat -tp
  (as root).
• Compare "Foreigen Address" on IOC with "Local
  Address" on partner.
• Try to find and kill the partner program.

6
mbufShow network buffer statistics

• number of times failed to find space should be
  0.
• Is tNetTask still alive?
• Look with i
• SUSPEND means dead
• Do other tasks (prio lt 50) block tNetTask?
• Look with spy
• Is any task using all CPU time?

7
What can I do on the IOC?

• Reboot.
• But first
• Look if the IOC can be rebooted safely (Pikett
  folder)
• Inform the control room (5503)
• Copy results of casr , inetstatShow, mbufShow in
  a file
• How many clients does casr report?
• How many full Send-Qs does inetstatShow report?
• To which PC? (Foreign Address)
• To which program? (netstat -tp as root on PC)
• How many times failed to find space does
  mbufShow report?

8
casr channel access server report

• Each client uses resources (memory, CPU time,
  bandwidth)
• Try to exit some clients.
• Look for redundant panels on other workspaces.

9
What if file I/O performance is bad?

• Look if the the directory is mounted (should be)
• nfsDevShow lists all mounted directories
• While doing on the server tail f
  /var/log/messagestry on vxWorks nfsMount
  "server","/directory"
• Check /etc/exports on the server
• Example Allow all IOCs with address 172... to
  mount /home/ioc (read and write)/home/ioc
  172.0.0.0/255.0.0.0(rw)
• After fixing /etc/exports, do service nfs reload
• Test performance
• timex copy lt/work/bench/big10 gt/nullshould show
  less than 7 seconds execution time

10
IOCMON IOC monitoring

• Shows potential future problems (when CA still
  works)
• medm -x -macro IOCXXX-VME-XXX
  G_IOCMON_ioc_status.adl
• or launcher?Generic Tools?VME Status?XXX-VME-XXX

• "No. of caClinets" raising?
• Maybe a problem with one client.

• " Times failed to find space" gt 0?
• Maybe tNetTask blocked.
• Check also "CPU Load".

• "Output Errors" gt 0?
• Old Tornado 1 bug. Should be fixed.