Packeteer PacketShaper Update

1
Packeteer PacketShaper Update

• OARNET April 2005

Sean ApplegateMid-Atlantic Territory
Engineer(703) 801-0413Sapplegate_at_packeteer.com
2
Agenda

• New Features in 7.X
• DoS Tuning
• Performance Tuning
• General Q A

3
Whats new in 7.X

• Several Minor UI Improvements
• WUI Plugin Architecture
• Quick Commands
• IPFilters
• Flow Detail Records (FDR)
• Adaptive Response Technology (ART)

4
User Interface Improvements
5
Quick Commands Making Many Changes Quickly
2
3
4
1
5
6
IPFilter Command

• PacketShaper ipfilter
• show Show IP info
• clear Clear IP filters
• onlyaccept Only accept specified packets
• passthrough Passthrough specified packets
• discard Discard specified packets
• iponly Relay only IP trafic onoff.
• Is applied in the NIC so it is more efficient
  than the Ignore/Discard policies.

7
IPFilter Command
Usage ipfilter passthrough ltdevicegt srcdst
ltipaddrgt ltmaskgt PacketShaper ip passthrough
main outside src 192.168.32.1 PacketShaper ip
passthrough inside dest 192.168.32.1
8
IPFilter Command
PacketShaper ip show Relay all traffic.
Exclude Filters total 2 19 hits 0 Inside
dst 192.168.32.1 (ffffffff) --gt passthru
20 hits 3 Outside src 192.168.32.1
(ffffffff) --gt passthru Include Filters total
0 PacketShaper ip clear clearing all filters...
9
Flow Detail Records (FDR)

• Off-box per-flow records
• Architecture is Similar to NetFlow but w/more
  application performance data points
• Typically increases CPU usage by 5-10
• Sends UDP Packet with up to 32 Flow Records to
  Flow Collector
• Collectors
• ReportCenter
• Linux OpenSource (Perl/PostgreSQL)
• Any NetFlow-5 Collector

• Formats
• NetFlow-5
• Packeteer-1
• Packeteer-2

Go to Documentation
10
FDR Architecture
Third_Party Billing Software
Cisco Collector
ReportCenter
NetFlow v5
NetFlow v5
Packeteer-1/2
11
Packeteer-1
12
Packeteer-2
13
Packeteer-2 cont.
14
Implementing FDR
PacketShaper setup flowrecords id 1 packeteer-2
192.168.32.25 9292 on lt Linux PacketShaper
setup flowrecords id 2 packeteer-2 192.168.32.50
9800 on lt RC3 PacketShaper set flowrecords
show ID RecordType CollectorIP Port
Enabled 1 packeteer-2 192.168.32.25
9292 on 2 packeteer-2 192.168.32.50
9800 on 3 4 PacketShaper setup
flowrecords id 2 off lt Turn off entry
2 PacketShaper setup flowrecords id 2 none lt
Remove entry 2
Go to Documentation
15
AR Overview - What is Adaptive Response?

• Adaptive Response is Packeteer technology that
  leverages its network visibility to inform,
  alert, or implement practical controls in
  response to a change in network conditions
• AR uses information available in the Measurement
  Engine, Hostdb and specific commands to trigger
  on a selected condition.
• AR has several components
• Templates
• Agents
• Action Files
• Incident Reports
• Goal Enable the design of a Smarter Network that
  can respond automatically to common issues.

Go to Documentation
16
AR Overview Templates

• Template building blocks for AR
• Each template identifies the metrics that can be
  monitored for a specific resource.
• Options
• Resource that is monitored.
• Metric(s) that are available to measure
  thresholds against.
• Variables available to reference in action files.
• General
• Template Specific

17
AR Agents

• Agent The Glue
• Multiple agents can be based on a single template
• Specifies the exact metric to monitor
• Specifies the resource to monitor (link, class,
  host, etc)
• Specifies Green, Yellow and Red Thresholds
• References optional Action files which implement
  alerts or actions.
• Creates Incident Report when triggered

Go to Documentation
18
AR Overview Action Files Drilldowns

• Action files (also called command or .cmd files)
• Provides the commands to run when an agent
  triggers
• Example - Send an email when inbound/default
  becomes greater than 15 of the partition
  bandwidth. The action file may look something
  like this for the Inbound Default Traffic agent
• partition apply ClassName 15 fixed
• send email dblanco_at_packeteer.com agentname
  agent The ClassName class has exceeded 15 of
  the link for more than 60 minutes. It has been
  limited to 15 of the link. Go to the Info tab to
  view the Incident Report for agentname agent to
  see more details.
• Drilldown (also called Incident Report)
• Produces detailed information on the events that
  caused an agent to trigger
• On the WUI, holding the mouse over the AR
  category produces a window with a list of links
  to drilldown files or graphs clicking on the
  link shows the drilldown in a separate window
• Example - High Bandwidth Host drilldown looks
  like this
• "time","agent3","avg-bps","host-ip","direction"
• "06-Jan-2005 141000","High Bandwidth
  Host",13816,10.10.253.99,outbound
• "06-Jan-2005 141000","High Bandwidth
  Host",7816,10.1.1.27,inbound

Go to Documentation
19
Default ART Templates

• App Health
• Default Traffic
• New Application
• High Bandwidth New App
• Host
• High Bandwidth Host
• Host Info Variables
• Failed Flow Ratio
• New Flows Per Minute (NFPM) Failed Flow
• New Flows Per Minute (NFPM) Side Unknown
• General
• Class ME Variables
• Partition Variables
• Link ME Variables

• Unit Health
• Unit Limits
• System Load
• Memory Allocation
• Network Health
• Traffic Performance
• Partition Utilization

20
AR Dashboard
21
AR Page
22
Editing an Agent
23
Adding an Action File
5
4
Go to Documentation
24
DoS Tuning

• What to Look For?
• Load Shedding
• Policy Flow Limit
• AR Agents
• NFPM Failed w/discard policy (500 failed
  flows/min)
• Class ME Variables
• TCP Inits (/inbound /outbound)
• TCP Server Ignores (link inbound outbound)
• TCP Server Ignores by class (wild card /inbound/
  /outbound/)
• Actions Enable FDR, Block Hosts, Enable
  PacketCapture, Run Forensic Commands
• Forensic Analysis
• Hostdb info sp n 25
• Hostd info sp ltnetworkgt ltmaskgt
• Hostdb info sf n 25
• Hostdb info sr n 25
• Tr h f lthostgt or Tr h r ltclass pathgt
• Flow Detail Records

25
What to Look For During a DoS Attack

• Host flow failures
• Hostdb info sp n 10
• Tcp-inits
• Tcp-server-ignores
• Tcp-server-ignores
• Client-floodblocks
• Server-floodblocks
• Tcpallocfailures
• Hostdballocfailures
• ReportPortal has quick access to all these
  metrics
• Analysis Matrix gt Failed Flows, Traffic History
• Report Portal gt Advanced gt I/O Troubleshooting
  Reports
• Configure Events or AR Agents to monitor above
  normal hits.

26
Load Shedding

• Drop new and/or existing connections on a per
  host basis that exceed a define threshold
• White List for Exception Hosts

Go to Documentation
27
Implementing Load Shedding
PacketShaper setup loadshedding exception add
listexceptionhosts PacketShaper setup
loadshedding clientFPM 2000 PacketShaper setup
loadshedding serverFPM 1000000 PacketShaper
setup loadshedding FailedFPM 1000 PacketShaper
setup loadshedding enable PacketShaper setup
loadshedding show Load Shedding
Enabled Shed New Flows Only
Enabled
Load Shedding
Parameters Current Default Min
Max
Client FPM
5000 1000000 10 1000000
Server FPM 1000000
1000000 10 1000000 Failed FPM
1000 1000000 10
1000000 TCBConn 100
100 5 1000000 UCBConn
100 100 5
1000000 Load Shedding Host Exception List
exceptionhosts
28
Policy Flowlimit

• Similar to Load Shedding but done in PacketWise
  Software and takes into account all flows and can
  only effect new flows.
• The limits are set to default values of 10,000
  flows per minute on client hosts and 100,000
  flows per minute on servers.
• Flows limits include flows of any type, not types
  of just that traffic type.
• policy flowlimit lttclassgt noneltclient-fpmgt
  ltserver-fpmgt

Go to Documentation
29
Hostdb Info

• PacketShaper hostdb info -sp -n 5
• IP Address Conn RTT Cur 1 Min
  Peak --- New Flows Per Minute ---
• to PS rate avg
  rate Client Server Failed
• --------------------------------------------------
  ------------------------------
• 137.45.62.218 I 1054 1ms 10k 13k
  1.3M 248 6 174
• 137.45.86.84 I 104 1ms 4248 4552
  1.2M 165 7 140
• 137.45.68.31 I 145 1ms 4209 7057
  1.4M 166 6 128
• 219.133.4.180 0 --- 0 20k
  1.6M 113 0 105
• 137.45.68.40 I 25 1ms 3672 2928
  1.8M 106 68 101

Go to Documentation
30
Tr h r lthostgt / Tr h f ltclassgt

• PacketShaper tr h r
• -----( /Inbound/Denied/Soulseek )-----
• 12-Apr-2001 221240 Soulseek
• 137.45.62.218 4979
• 68.36.231.123 1235 pcp09971203pcs.narlington.nj.c
  omcast.net

Go to Documentation
31
Performance Tuning

• The Basics
• Hard Code Interfaces
• Prune Your Traffic Tree smaller trees are
  faster
• Increase discovery thresholds (Setup Tab gt System
  Variables)
• Turn Discovery on Sparingly (schedule it)
• Classify/Shape Resnet, Admin, Library, Labs
  Separately
• Use Partitions and Policies Together
• Use Rate Policies on Medium to Long Lived Flows
• Dont Use Rate Policy on /__bound/Default classes
• Protect and Limit Traffic all too often higher
  ed just limits traffic
• Use Scheduling as Appropriate gt Backup Configs,
  ToD Shaping, etc.
• Dont run your device at Red Line thresholds
• Advanced
• Configure AR System Health Action Files and ME
  Failure Metrics
• Create a Customer Portal with Key Performance
  Reports
• System Health
• System Limits
• Sys kmem ltpkt comgt types

32
System Health
Current Idle 100 Average Idle 99 (last
25 secs) Minimum Idle 0 Vector 1 (
25958, 6126, 77697, 102998) Vector 2 (
548994, 243126, 184608) Vector 3 ( 1037898,
830514, 987846) Load 0 (1000
ticks) Random Ppt 0 0 Hard Ppt
0 0 Shedding Ppt 0 0 No Buf
Ppt 0 0 TxDone Queue 0
0 TxPend Queue 0 0 Rx Queue
0 0 TxDone Bkup 0 0 ticks Rx
Queue Dly 0 0 ticks Overhead
0 0 ticks
33
System Limits
Statically allocated objects Current
Remaining Total -----------------------------
-------------------------------------- Traffic
classes 201 1847
2048 Partitions
24 1000 1024 Dynamic Partitions
13504 6496
20000 Policies
71 1977 2048 Matching rules
509 4613
5122 Classes with worst clients/servers
0 16 16 Classes with top
talkers/listeners 11 1
12 TCP flows 34845
365155 400000 Other IP flows
14923 185077 200000 Legacy
flows 8 19992
20000 Concurrent Hosts
200000 0 200000
Represents entries in hostdb, not active hosts.
We over-write old hosts so once it is has seen
the max hosts it will stay there until reset.
34
Sys kmem commands packet engine / common
processes
Idx Type Blocks Allocs
Fails Bytes HiWater Limit Pcnt 34 magic
8959 729016725 0 322524
1433160 12582912 3.88 35 appness
5510 443470843 0 286520 307944
12582912 3.45 47 discovery 0
0 0 0 0 6291456
0.00 50 congestion 74897 -1260946167
641326241 6291348 6291516 6291456 75.70 52
appness-tracking 0 0 0
0 0 6291456 0.00 53 classThingy
0 0 0 0 0
6291456 0.00 55 dshash 0
0 0 0 0 6291456
0.00 56 crumbs 1790 -1934452076
0 161176 796176 6291456 1.94 59
dynamic-ptn 0 0 0
0 0 6291456 0.00 61 logger
0 0 0 0 0
6291456 0.00 80 appness hash 49
280865 0 20436 68568 6291456
0.25 81 appness peeker 5510 443305107
0 198360 213192 12582912 2.39 82
appness data 2051 153351844 0
302188 409728 12582912 3.64 83 appness
criteria 5510 1299790183 0 693888
906096 18874368 8.35 84 appness utils
149 11687607 0 6548 162836
6291456 0.08 85 appness misc 366
4026244 0 26984 244636 6291456
0.32 86 flow records 0 0
0 0 0 6291456
0.00
0
104795 267180072 641326241 8310976
10850116 0 100.00
35
General Q A

• Performance
• Sizing
• Feature Requests
• Support / Resources
• Training
• Futures
• Etc