Network Performance Tuning

1
Network Performance Tuning

• Art Houle
• Hayes Computer Systems
• http//www.hcs.net

2
Throughput Issues
3
Causes of the Problem

• Lost packets
• Line errors
• Poor connections
• Electrical interference
• Dirty optical connections
• Dropped packets
• Busy switch
• Underpowered hardware
• Link bandwidth
• Excessive traffic
• Hacker attacks
• SPAM
• User abuse

4
Effect of Lost Packets

• Lost packets are retransmitted after timeout
• TCP retransmitted by session layer
• UDP
• Video/voice is lost
• Other may retransmit by application layer
• FTP example
• 5ms between packets
• 2 second timeout
• 4001 lost throughput
• 0.25 packet loss 50 throughput loss

5
Link Throughput Issues
6
Tune Switch and Router Links

• Match duplex (Prefer full-duplex)
• Vendor issues (auto vs. manual)
• Adjust buffer size
• Hold queue 2048 in
• Hold queue 2048 out
• Implement flow-control on GigE links
• Monitor interfaces for errors
• Monitor CPU level to be less than 50 on 5-minute
  average

7
Hardware Capacity

• 1 Gbps to servers
• 1 Gbps to power users
• 100 Mbps to general users
• WAN link
• Monitor and graph usage
• Look for flat-top on graph
• Monitor servers
• Munin freeware
• Disk, CPU, processes, etc.

8
Monitor Switches and Routers

• Manageable switch
• Telnet or ssh (preferably)
• SNMP for automatic monitoring
• Syslog to a server that is monitored
• Monitor
• Port utilization
• Port errors
• Box CPU usage
• Netflow
• Know the traffic on your network

9
Look For Interface Errors

• Sho int gi 1/1
• GigabitEthernet1/1 is up, line protocol is up
  (connected)
• Hardware is C6k 1000Mb 802.3, address is
  0013.5f1e.59c0 (bia 0013.5f1e.59c0)
• MTU 1550 bytes, BW 1000000 Kbit, DLY 10 usec,
  reliability 255/255, txload 48/255, rxload 32/255
• Encapsulation 802.1Q Virtual LAN, Vlan ID 1.,
  loopback not set
• Keepalive set (10 sec)
• Full-duplex, 1000Mb/s, media type is T
• Input flow-control is off, output flow-control
  is off
• Clock mode is auto
• ARP type ARPA, ARP Timeout 040000
• Last input 000000, output 000000, output
  hang never
• Last clearing of "show interface" counters 11w0d
• Input queue 0/75/162/0 (size/max/drops/flushes)
  Total output drops 0
• Queueing strategy fifo
• Output queue 0/40 (size/max)
• 5 minute input rate 126165000 bits/sec, 36998
  packets/sec
• 5 minute output rate 189014000 bits/sec, 27700
  packets/sec
• L2 Switched ucast 667206 pkt, 42720388 bytes -
  mcast 309876 pkt, 21531924 bytes
• L3 in Switched ucast 0 pkt, 0 bytes - mcast 0
  pkt, 0 bytes mcast

10
Network Topology

• Segment the LAN
• reduce broadcast traffic
• Replace ATM with Serial or Metro-E
• 10 overhead for ATM cells
• Recurring cost
• Separate servers and clients for security

11
Reduce Non-business Traffic

• SPAM
• Netflow monitoring summarize SMTP by local IP
• Black-list worst sources
• Attacks internal and external
• Hacked computers source traffic
• Scan regularly and unplug compromised systems
• Nitro Security
• NetExpose
• Snort Linux freeware
• Then rebuild the OS and applications from scratch
• User web surfing
• Filter inappropriate sites
• P2P (Kazaa) and Video (YouTube)
• IP to user documentation
• Locate the abuser and correct bad habits

12
NetFlow

• All traffic between one source and destination
  address and port is a flow
• Netflow J-Flow(RFC-3917, RFC-3954)
• S-flow (RFC-3176) samples of flows
• Analysis Software examples
• S-flow Trend
• Flow-Scan
• Flow-tools
• Netflow-tracker
• Q-Radar
• Stager
• NFSEN
• Cisco CNS NetFlow

• Caligore Flow-Inspector
• InMon
• NetQOS
• Net I Monitor
• NetDector
• NetIntercepy
• Many others
• Packeteer stand alone in-line box

13
QOS

• WRED at the WAN link outbound
• Weighted Random Early Drop
• When WAN congestion occurs
• Preferentially drop less important traffic
• QOS strategy
• Mark packets inbound to the edge router by
  traffic type
• E-mail is not time critical
• Drop at the outbound link

14
Tune Servers for Faster Response

• More RAM
• Turn off DNS lookups on syslog
• Monitor processes, disk, CPU, etc
• munin freeware
• Backups
• Schedule off hours to avoid conflicts
• Consider jumbo-frames (may need upgrade)
• Off-load SSL
• Multiple servers
• Load distribution
• DNS round-robin
• Content switch (persistance)

15
Summary

• Restructure
• Smaller LANs
• Bigger links
• Bigger boxes
• Replace ATM
• Monitor
• Link errors
• Link usage
• CPU
• Traffic usage
• Reduce abuse
• SPAM
• Users
• Hacked systems
• Rework servers

16
Hayes Solutions

• Network engineering consultation
• Cisco and 3Com reseller
• Managed network solutions
• Line speed hardware
• Security components
• WAN bandwidth upgrade
• E-mail systems to contain abuse
• Content filters
• WebSense

17
Thank You

• Hayes Computer Systems
• 2473 Care Drive, Ste 201, Tallahassee, FL 32308
• www.hcs.net
• 850-297-0551
• 800-825-9390
• Art Houle
• 850-297-0551x183
• ahoule_at_hcs.net