Troubleshooting in the Check Point Environment Part II When the going gets tough

1
Troubleshooting in the Check Point Environment
Part IIWhen the going gets tough
Tobias Lachmann
2
Agenda

• How to approach troubleshooting
• quick and dirty debug
• fw monitor
• general kernel debug
• fwm and GUI clients
• VPN
• ClusterXL
• security server
• desktop policy server / desktop log server
• SecurePlatform
• UTM-1 Edge

3
(No Transcript)
4
How to approach troubleshooting

• Troubleshooting is not about resolving the
  reason!
• Its about identifying the problem!

5
How to approach troubleshooting

• Collect information
• What is the problem? What are the symptoms?
• Can the problem be replicated?
• Random occurence?
• Anything changed in the setup?
• User-related or machine-related?
• List systems that are part of the conversation

6
How to approach troubleshooting

• Use upgrade_export to take dump of configuration,
  CA etc. and try to replicate in lab environment.
• VMware can be used for this purpose easily, but
  unfortunately UTM-1 starting with NGX R70 dont
  work with VMware Workstation or VMware ESX Server.

7
fw ctl zdebug drop

• Replicate the problem and have a look at the
  gateway
• fw ctl zdebug drop
• lists all dropped packets in realtime
• gives an explanation why the packet is dropped

8
fw monitor

• What is it?
• fw monitor command is a Check Point kernel module
  that is used to capture packets.
• What makes it different?
• Packet capture at multiple positions within the
  kernel module chain, both for inbound and
  outbound packets. It doesnt work on Layer-2, so
  no MAC addresses are shown in the output.
• fw monitor is available on all platforms.

9
fw monitor

• What makes it different?
• filters packets using INSPECT code
• UUID of connection can be displayed
• SSUID can be displayed
• sees packets with the eyes of the gateway
• Shows flow of packets through the gateway
• No Layer-2 information in capture files

10
fw monitor
App.
App.
TCP
TCP
IP
Routing
IP
pre-outbound (o)
post-inbound (I)
VM
VM
pre-inbound (i)
post-outbound (O)
NIC
NIC
11
fw monitor

• Expert_at_fw1 fw monitor -e "accept
  (src212.1.52.68 or dst212.1.52.68)"
• monitor getting filter (from command line)
• monitor compiling
• monitorfilter
• Compiled OK.
• monitor loading
• monitor monitoring (control-C to stop)
• eth3.7i52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth3.7I52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0o52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0O52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0i52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e
• eth0I52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e

12
fw monitor

• eth3.7O52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e

13
fw monitor

• fw monitor options overview
• -u s Shows UUID or SUUID for every packet
• -i write data to STDOUT
• -d D debug / more debug output
• -e ltexprgt filter for expression (CLI mode)
• -f ltfilegt read filter expression from file
• -l ltlengt limit length of captured packet
• -m ltmaskgt which positions should be shown
• -x print raw packet data
• -o ltfilegt write packet into file
• -px ltposgt insert fw monitor at specific chain
  position
• -p all insert fwmonitor between all kernel
  modules
• -ci ltcountgt stop capture after count incoming
  packets
• -co ltcountgt stop capture after count outgoing
  packets

14
fw monitor
15
fw monitor
Capture only ICMP packets
fw monitor -e "accept 911
16
fw monitor
Capture only packets from a special host
fw monitor -e "accept 12,b192.168.1.1
17
fw monitor

• Filtering will be easier for you if you use
  macros.
• Macros for fw monitor are defined in
  FWDIR/lib/fwmonitor.def which references
  FWDIR/lib/tcpip.def, where the actual expression
  is located.
• Example filter for source IP
• fwmonitor.def macro src
• tcpip.def macro ip_src
• expression 12,b

18
fw monitor

• Use macros together with operators to add
  complexity
• accept (srcx.x.x.x or dstx.x.x.x)
• accept ((srcx.x.x.x, dsty.y.y.y) or
  (srcy.y.y.y, dstx.x.x.x))
• accept not (sport22 or dport22)
• accept sport21 and not (srcx.x.x.x)

19
fw monitor

• Use fw monitor to see if packets are translated
• fw monitor -e accept (src212.1.56.151 or
  dst212.1.56.151)
• eth0i48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth0I48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth1o48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth1O48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• fw monitor -e accept (src212.1.56.151 or
  dst212.1.56.151)
• eth0i48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id31171
• eth0I48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171
• eth1o48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171
• eth1O48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171

20
fw monitor

• Common expressions for fw monitor
• fw monitor e accept (srcx.x.x.x or
  dstx.x.x.x)
• fw monitor m iO e accept host(x.x.x.x)
• fw monitor e accept ((srcx.x.x.x, dsty.y.y.y)
  or (srcy.y.y.y, dstx.x.x.x))
• fw monitor e accept (ip_px)
• Combine with o ltfilegt for output into a file.

21
fw monitor

• Read complex expressions from a filter file
• fw monitor f ltfilenamegt
• If you use macros in a filter file, make sure to
  include the appropriate definition file.
• include fwmonitor.def
• accept ((sport22 or dport22) and not
  (host(x.x.x.x))

22
fw monitor

• Use for better
  analysis of capture files.
• Preferences ? Protocols ? Ethernet ? Check box
  Attempt to interpret as Firewall-1 monitor file
• Preferences ? Protocols ? FW-1 ? Activate UUID,
  chain position, summary in protocol tree
• Add column fw1 chain of format FW-1 monitor
  if/direction
• Add coloring rules
• preIn ? Filter String fw1.direction i
• postIn ? Filter String fw1.direction I
• preOut ? Filter String fw1.direction o
• postOut ? Filter String fw1.direction O

23
fw monitor

• On UTM-1 Edge
• Setup ? Tools ? Packet Sniffer
• two modes normal sniffer or fw monitor
• On SecuRemote/SecureClient
• srfw monitor o ltfilenamegt

24
General kernel debug

• fw ctl debug
• Allocation of a buffer for the debug logsfw ctl
  debug buf size in kb
• The main debug commandfw ctl debug m ltmodulegt
  ltoptiongt
• Writing the debug logs into a filefw ctl kdebug
  f o ltfilenamegt
• Stop debuggingfw ctl debug 0

25
General kernel debug

• Some examples for modules and options
• Module fw
• Options error warning cookie crypt domain ex
  driver filter hold if install ioctl kbuf ld log
  machine memory misc packet q xlate xltrc conn
  synatk media align balance chain bridge tcpstr
  scv ndis packval sync ipopt link nat cifs drop
• Module vpn
• Options driver err packet policy sas rdp clear
  cipher init sr comp xl counters mspi cphwd ref
  vin cluster nat l2tp warn
• ? Refer to ATRG NGX for complete list.

26
General kernel debug, new in R70

• Filter debug, only lines with ltstringsgt in it are
  written to the output (best practice error,
  failed)
• fw ctl debug d ltstringsgt
• Filter debug, only lines that dont contain
  ltstringgt in it are written to the output
• fw ctl debug d ltstringsgt
• Can be combined
• fw ctl debug d error,failed,packet

27
General kernel debug, new in R70

• Stop debug messages when a certain string is
  issued.
• fw ctl debug s ltstringgt
• Example
• fw ctl debug s error

28
fwm debug

• To debug fwm do the following
• fw debug fwm on TDERROR_ALL_ALL5
• fw debug fwm on OPSEC_DEBUG_LEVEL9
• To stop debug run
• fw debug fwm off TDERROR_ALL_ALL0
• fw debug fwm off OPSEC_DEBUG_LEVEL0
• Logs are written to FWDIR/log/fwm.elg

29
Debugging GUI clients

• Debug GUI clients under NGX R65
• Dashboard ? fwpolicy.exe d o fwp_debug.txt
• Tracker ? cplgv.exe d o cplgv_debug.txt
• Monitor ? smartcons.exe d o smartcons_debug.txt
• general syntax ltexecutablegt -d o ltfile_namegt
• Output is in specified directory or local
  directory,if directory is omitted.

30
Debugging GUI clients

• Debug GUI clients under NGX R70
• Dashboard ? fwpolicy.exe d o fwp_debug.txt
• Tracker ? cplgv.exe d o cplgv_debug.txt
• Monitor ? smartcons.exe d o smartcons_debug.txt
• general syntax ltexecutablegt -d o ltfile_namegt
• Output is in specified directory or in
  C\Programme\CheckPoint\SmartConsole\R70\PROGRAM\
  data
• if directory is omitted.

31
Debugging GUI clients

• Trace utility provided
• by Check Point RD.

32
VPN debug

• Best practice before starting debug
• Compare configuration on both ends
• often Phase I / Phase II parameters are not equal
  which causes the VPN to fail
• take special notice of networks and subnet masks
• carefully compare Pre-Shared-Secrets
• Have a close look at the logs in SmartView
  Tracker
• Most informations can be found in the logs

33
VPN debug

• To determine status of VPN tunnels, use menu
  based
• vpn tunnelutil ? vpn tu
• To shutdown all VPN operation, use
• vpn drv off
• To enable VPN again, use
• vpn drv on
• install policy

34
VPN debug

• VPN debugging events can be logged on the gateway
• vpn debug on
• Debug output is written to FWDIR/log/vpnd.elg
• More details can be logged using the command
• vpn debug on TDERROR_ALL_ALL5
• Turn off debugging with
• vpn debug off

35
VPN debug

• IKE negotiations during VPN tunnel establishment
  can be logged in ike.elg
• On the gateway
• vpn debug ikeon / vpn debug ikeoff
• Debug output is written to FWDIR/log/ike.elg
• On SecuRemote/SecureClient
• Kill SecurRemote/SecureClient
• Create the file fwike_debug.all in C\
• Launch SecureRemote/SecureClient
• Debug output is written to SRDIR\log\IKE.elg

36
VPN debug

• On UTM-1 Edge appliance
• WebUI -gt Reports -gt Tunnels -gt save IKE trace
• Click Save IKE Trace, which creates ike.elg

37
VPN debug

• Capture traffic using fw monitor
• fw monitor e accept port(500) or port(4500)
  o monitor.out
• Output file is monitor.out, IKE payloads are
  encrypted.
• Capture traffic using vpn debug
• vpn debug mon
• Output file is ikemonitor.snoop, IKE payloads are
  in clear.
• Turn off with vpn debug moff.

38
VPN debug

• Initiate VPN and IKE debug together
• vpn debug trunc
• Disable VPN and IKE debug
• vpn debug off
• vpn debug ikeoff

39
cpd debug

• To debug cpd do the following
• cpd_admin debug on TDERROR_ALL_ALL5
• To stop debug run
• cpd_admin debug off TDERROR_ALL_ALL5
• Logs are written to CPDIR/log/cpd.elg

40
ClusterXL

• Status information
• fw hastat
• HOST NUMBER HIGH AVAILABILITY STATE
  MACHINE STATUS
• localhost 2 stand-by OK
• cphaprob state
• Cluster Mode New High Availability (Primary
  Up)
• Number Unique Address Assigned Load State
• 1 192.168.55.202 100 Active
• 2 (local) 192.168.55.201 0 Standby

41
ClusterXL

• cphaprob ia list
• cphaprob a if
• fw ctl pstat
• Statistics of ClusterXL sync
• cphaprob syncstat
• Reset statistics of ClusterXL sync
• cphaprob reset syncstat

42
ClusterXL

• Debugging
• fw ctl debug buf 8192
• fw ctl debug m fw conn drop packet if sync
• fw ctl debug m cluster all
• fw ctl kdebug f o ltfilenamegt

43
Security servers debug

• Some examples for security servers
• FTP security server in.aftpd
• Telnet security server in.atelnetd
• HTTP security server in.ahttpd
• SMTP security server in.asmtpd
• ClientAuth (900) in.ahclientd
• ClientAuth (259) in.aclientd
• AntiSpam security server in.msd
• URL filtering security server in.aufpd

44
Security servers debug

• Verify that security server process exists. Check
  FWDIR/tmp for existing PID files.
• Start debugging (example for FTP security
  server)fw debug in.aftpd on FWAFTPD_LEVEL3
• Stop debuggingfw debug in.aftpd off
  FWAFTPD_LEVEL3

45
Security servers debug

• Verify that security server process exists. Check
  FWDIR/tmp for existing PID files.
• Start debugging (example for AntiSpam security
  server)fw debug in.msd on TDERROR_ALL_ALL5
• Stop debugging fw debug in.msd on
  TDERROR_ALL_ALL0

46
Desktop policy server debug

• To debug dtps do the following
• dtps debug on
• To stop debug run
• dtps debug off
• Logs are written to FWDIR/log/dtpsd.elg

47
Desktop log server debug

• To debug dtls do the following
• fw debug dtls on
• To stop debug run
• fw debug dtls off
• Logs are written to FWDIR/log/dtlsd.elg

48
Secure Platform debug

• Sometimes it is useful to verify file integrity
  and version against a test
• environment, for example after installation of
  ad-hoc fixes or HFA.
• Use md5sum for creating hashes.
• Expert_at_fwm md5sum upgrade_import
• e6c6417cca9db098b94673dd420a4903 upgrade_import
• Use cpvinfo for displaying version information.
• Expert_at_fwm cpvinfo upgrade_import
• Build Number 620650003
• Major Release NGX
• Minor Release amfi_hfa
• Release Number 5.0.5
• Version Name NGX

49
Secure Platform debug

• For some problems with processes a core dump can
  be usefull.
• A core dump is a disk file that contains an image
  of the processs memory at the time of
  termination.
• Core dumps are mainly used by Check Point RD for
  fixing a specific problem.

50
Secure Platform debug

• To enable core dumps do the following
• ulimit c unlimited
• um_core enable
• Reboot
• Check that /etc/sysconfig/enable_cores exist
  after Reboot.
• Dumps will be in /var/log/dump/usermode

51
Troubleshooting UTM-1 Edge

• Sofaware Management Server Console
• http//ltip SmartCentergt9283/
• restart SMS
• reload SMS settings
• force policy update
• reboot
• reset local (Edge) password
• view status information

52
Troubleshooting UTM-1 Edge
53
Troubleshooting UTM-1 Edge

• Is the SMS process running on SmartCenter?
• ps aux grep sms
• Is traffic reaching the SmartCenter?
• fw monitor
• libsw must be current, at least same version as
  latest firmware installed on a Edge.
• Check /opt/CPEdgecmp-R65/libsw/version.txt
• Expert_at_fwm head -n1 version.txt
• libsw built with version 8.0.39

54
Troubleshooting UTM-1 Edge

• Debugging Sofaware Management Server
• Edit FWDIR/conf/sofaware/SWManagement.ini
• Change in line containing LogPolicy1 the value
  Info to Debug
• smsstop
• sms confdir FWDIR/conf/sofaware
• Replicate the problem and watch for console
  output.
• Terminate programm and restart SMS afterwards
• smsstart

55
Troubleshooting UTM-1 Edge

• Configuration for Edge Devices on SPLAT under
• /opt/CPEdgecmp-R65/tmp
• ltname of Edge objectgt.pf ? ruleset
• ltname of Edge objectgt.pfz ? compressed ruleset
• ltname of Edge objectgt.topo ? topology for VPN
• ltname of Edge objectgt.tpz ? compressed topology
• ltname of Edge objectgt.p12 ? PKCS12 certificate
• Delete files. Install policy again to re-generate
  them.
• Make sure, that the files are compiled and the
  Edge gets the latest version.

56
Troubleshooting Edge

• Analyse local policy
• Run info fw rules on command line
• or WebUI ? Setup ? Tools ? Command Line
• Analyse NAT policy
• Run info nat on command line
• or WebUI ? Setup ? Tools ? Command line

57
Troubleshooting UTM-1 Edge

• Create diagnostics file
• Log into WebUI
• ? Setup ? Tools ? Diagnostics

58
Resources

• fw monitor
• http//www.checkpoint.com/techsupport/downloads/h
  tml/ethereal/fw_monitor_rev1_01.pdf
• Troubleshooting and Debugging Tools for Faster
  Resolution
• http//www.checkpoint.com/services/enterprise/doc
  s/Troubleshooting_and_Debugging.pdf
• The CPinfo utility
• https//supportcenter.checkpoint.com/supportcente
  r/portal?eventSubmit_doGoviewsolutiondetailssolu
  tionidsk30567

59
Questions?
60
Still got a question?

• Tobias Lachmann
• Technical Consultant
• MCS Moorbek Computer Systeme GmbH
• Essener Bogen 17
• 22419 Hamburg
• tobias.lachmann_at_mcs.de
• Telefon 040 / 53773 - 160