Troubleshooting in the Check Point Environment Part I Houston, we have a problem

1
Troubleshooting in the Check Point Environment -
Part IHouston, we have a problem
Tobias Lachmann
2
Agenda

• How to approach troubleshooting
• Network interfaces
• Firewall status
• fw commands
• VPN debug
• Troubleshooting UTM-1 Edge
• Licenses
• Opening a service request
• Check Point TAC

3
(No Transcript)
4
How to approach troubleshooting

• Troubleshooting is not about resolving the
  reason!
• Its about identifying the problem!

5
How to approach troubleshooting

• Collect information
• What is the problem? What are the symptoms?
• Can the problem be replicated?
• Random occurence?
• Anything changed in the setup?
• User-related or machine-related?
• List systems that are part of the conversation

6
How to approach troubleshooting

• Bug or configuration problem?
• Common configuration problems
• Firewall rule prevents traffic
• SmartDefense / IPS blade prevents traffic
• Antispoofing
• misconfigured routing
• wrong encryption domain
• wrong username / password

7
How to approach troubleshooting

• Any reference for problem or error message?
• official documentation
• SecureKnowledge
• CPUG forum
• Check Point forum
• Google

8
Network interfaces
9
Network interfaces

• are the needed interfaces up and running?
• is ARP working?
• are the needed routes present?
• is the next hop alive?
• which network interfaces do we have?
• cpstat os f ifconfig

10
Network interfaces

• show status
• ethtool ltifnamegt
• show statistics
• ethtool S ltifnamegt
• show driver version
• ethtool i ltifnamegt

11
Network interfaces

• show configuration and status information
• ifconfig ltifnamegt
• show routing table
• netstat rn
• show arp table
• arp -a

12
Network interfaces

• Is proxy ARP working?
• fw ctl arp
• Is manual proxy ARP configured?
• Check FWDIR/conf/local.arp

13
Firewall status

• Status of processes monitored by CPwatchdog
• cpwd_admin list
• Current connections?
• fw tab t connections s

14
Firewall status

• (Which) policy installed?
• fw stat l
• Show policy statistics
• cpstat fw f policy

15
Firewall status

• Gateway busy? Memory exhausted? Disc full?
• vmstat 10
• top
• df -h

16
Firewall status

• Display overall statistics every 5 seconds
• cpstat -o 5 os -f statistics
• Display detailed statistics
• cpstat os -f cpumemorydisk

17
fw ctl zdebug drop

• Replicate the problem and have a look at the
  gateway
• fw ctl zdebug drop
• lists all dropped packets in realtime
• gives an explanation why the packet is dropped

18
fw log f t

• Replicate the problem and have a look at the
  management
• fw log f t
• lists all logged packets in real-time
• Use SmartView Tracker for better viewing of log
  entries

19
fw monitor

• What is it?
• fw monitor command is a Check Point kernel module
  that is used to capture packets.
• What makes it different?
• Packet capture at multiple positions within the
  kernel module chain, both for inbound and
  outbound packets. It doesnt work on Layer-2, so
  no MAC addresses are shown in the output.
• fw monitor is available on all platforms.

20
fw monitor
App.
App.
TCP
TCP
IP
Routing
IP
pre-outbound (o)
post-inbound (I)
VM
VM
pre-inbound (i)
post-outbound (O)
NIC
NIC
21
fw monitor

• Replicate the problem and have a look at the
  gateway
• fw monitor e accept
• lists all packets reaching and leaving the
  gateway in real-time
• shows how the packets are processed

22
fw monitor

• Expert_at_fw1 fw monitor -e "accept
  (src212.1.52.68 or dst212.1.52.68)"
• monitor getting filter (from command line)
• monitor compiling
• monitorfilter
• Compiled OK.
• monitor loading
• monitor monitoring (control-C to stop)
• eth3.7i52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth3.7I52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0o52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0O52 212.1.56.233 -gt 212.1.52.68 (TCP)
  len52 id18406
• TCP 56661 -gt 22 .S.... seqb2f3509d ack00000000
• eth0i52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e
• eth0I52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e

23
fw monitor

• Interpreting the output
• eth3.7O52 212.1.52.68 -gt 212.1.56.233 (TCP)
  len52 id0
• TCP 22 -gt 56661 .S..A. seq68a919c9 ackb2f3509e

24
fw monitor

• Common expressions for fw monitor
• fw monitor e accept (srcx.x.x.x or
  dstx.x.x.x)
• fw monitor e accept (srcx.x.x.x,
  dsty.y.y.y)
• fw monitor e accept ((srcx.x.x.x, dsty.y.y.y)
  or (srcy.y.y.y, dstx.x.x.x))
• Combine with o ltfilegt for output into a file.

25
fw monitor

• Use fw monitor to see if packets are translated
  (NAT)
• fw monitor -e accept (src212.1.56.151 or
  dst212.1.56.151)
• eth0i48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth0I48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth1o48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• eth1O48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id27053
• fw monitor -e accept (src212.1.56.151 or
  dst212.1.56.151)
• eth0i48 212.1.56.151 -gt 195.244.116.166 (TCP)
  len48 id31171
• eth0I48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171
• eth1o48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171
• eth1O48 212.1.56.151 -gt 192.168.199.2 (TCP)
  len48 id31171

26
fw monitor

• On UTM-1 Edge
• Setup ? Tools ? Packet Sniffer
• two modes normal sniffer
• or fw monitor
• On SecuRemote/SecureClient
• srfw monitor o ltfilenamegt

27
fw monitor

• Use for better
  analysis of capture files.
• Preferences ? Protocols ? Ethernet ? Check box
  Attempt to interpret as Firewall-1 monitor file
• Add column fw1 chain of format FW-1 monitor
  if/direction
• Add coloring rules
• preIn ? Filter String fw1.direction i
• postIn ? Filter String fw1.direction I
• preOut ? Filter String fw1.direction o
• postOut ? Filter String fw1.direction O

28
VPN debug

• Best practice before starting debug
• Compare configuration on both ends
• often Phase I / Phase II parameters are not equal
  which causes the VPN to fail
• take special notice of networks and subnet masks
• carefully compare Pre-Shared-Secrets
• Have a close look at the logs in SmartView
  Tracker
• Most information can be found in the logs

29
VPN debug

• To determine status of VPN tunnels, use menu
  based
• vpn tunnelutil ? vpn tu
• To shutdown all VPN operation, use
• vpn drv off
• To enable VPN again, use
• vpn drv on
• install policy

30
Troubleshooting UTM-1 Edge

• Analyse local policy
• Run info fw rules on command line
• or WebUI ? Setup ? Tools ? Command Line
• Analyse NAT policy
• Run info nat on command line
• or WebUI ? Setup ? Tools ? Command line

31
Troubleshooting UTM-1 Edge

• Create diagnostics file
• Log into WebUI
• ? Setup ? Tools ? Diagnostics

32
Licenses

• Limited number of hosts?
• fw lichosts
• Count of used hosts
• fw lichosts wc l
• SecureClient licenses used
• dtps lic

33
Licenses

• Show license
• cplic print
• Compare to SmartUpdate / SmartView Monitor output
• Especially UTM products sometimes tend to mess up
  with licenses which can cause Antivirus, Antispam
  or URL filtering to stop working.
• You need to keep contracts updated.

34
Opening a Service Request

• Two support programs with different flavors
• Direct Enterprise Support
• Standard
• Premium
• Diamond
• Collaborative Enterprise Support
• Co-Standard
• Co-Premium

35
Opening a service request

• Who is handling your service request
• Direct Enterprise Support
• user has direct contact to TAC
• SLA from 5x12 to 24x7
• Support Engineer / Premium Support Engineer /
  Designated Diamond Support Engineer
• Access to SecureKnowledge
• Standard Advanced
• Premium Diamond Expert

36
Opening a service request

• Who is handling your service request
• Collaborative Enterprise Support
• user has contact with CCSP,CCSP has contact with
  TAC
• SLA 24x7
• Support Engineer / Premium Support Engineer
• Access to SecureKnowledge
• Standard Advanced
• Premium Expert

37
Opening a Service Request

• Submit info to Check Point TAC or your CCSP/CSP
• provide contact info
• describe Check Point environment
• list used gateway hardware
• provide info about network topology and hardware
• describe the problem / the symptoms in detail
• what kind of business impact has this problem
• recommendation be available for remote session

38
Opening a Service Request

• Which Check Point versions are you using?
• Version of SPLAT
• ver
• version of gateway
• fw ver k
• version of SmartCenter
• fwm ver

39
Opening a Service Request

• Create compressed CPInfo diagnostic file
• /opt/CPinfo-10/bin/cpinfo z ltfilenamegt
• Create compressed CPInfo diagnostic file
  including logs
• /opt/CPinfo-10/bin/cpinfo l z ltfilenamegt
• CPInfo files can be viewed using InfoView

40
TAC organisation
Customer Focus Programmers
41
TAC escalation
Support desk
Product team
Escalations
Customer focus programmer
42
TAC escalation path

• http//www.checkpoint.com/services/contact/esc
  alation.html

43
Resources

• fw monitor
• http//www.checkpoint.com/techsupport/downloads/h
  tml/ethereal/fw_monitor_rev1_01.pdf
• Troubleshooting and Debugging Tools for Faster
  Resolution
• https//sc1.checkpoint.com/uc/htmls/SstTroublesho
  oting_and_Debugging.pdf
• The CPinfo utility
• https//supportcenter.checkpoint.com/supportcente
  r/portal?eventSubmit_doGoviewsolutiondetailssolu
  tionidsk30567

44
Questions?
45
Still got a question?

• Tobias Lachmann
• Technical Consultant
• MCS Moorbek Computer Systeme GmbH
• Essener Bogen 17
• 22419 Hamburg
• tobias.lachmann_at_mcs.de
• Telefon 040 / 53773 - 160