Network Application Performance

1
Network Application Performance

• Deke Kassabian and Shumon Huque
• ISC Networking Telecommunications
• February 2002 - Super Users Group

2
Introduction

• What this talk is all about
• Network performance on the local area network and
  around campus
• Network performance in the wide area and for
  advanced applications
• Goal acceptable performance, positive user
  experience

3
Who needs to be involved?

• End Users
• Researchers
• Local Support Providers
• Application Developers
• System Programmers/Administrators
• Network Engineers

4
What is performance?

• Performance might mean
• Elapsed time for file transfers
• Packet loss over a period of time
• Percentage of data needing retransmission
• Drop outs in video or audio
• Subjective feeling that feedback is on time

5
Throughput

• Throughput is the amount of data that arrives per
  unit time.
• Goodput is the amount of data that arrives per
  unit time, minus the amount of that data that was
  retransmitted.

6
Delay

• Delay is a time measurement for data transfer
• One way network delay for a bit in transit
• Delay for a total transfer
• Time from mouse click to screen message that the
  operation is complete

NIC to NIC
Stack to Stack
Eyeball to eyeball
7
Jitter

• Variation in delay over time
• Non-issue for non-realtime applications
• May be problematic for some applications with
  real-time interactive requirements, such as video
  conferencing
• E2E delay of 70 ms /- 5 ms -gt low jitter
• E2E delay of 35 ms /- 20 ms -gt higher jitter

8
Some Contributors to Delay

• Slow networks
• Slow computers
• Poor TCP/IP stacks on end-stations
• Poorly written applications

9
Analysis of Delay
A
B
(2) Propagation Delay

1. Insertion time

(3) Processing Delay
10
Analysis of Delay
From Deke To Ira Date Mon Feb 12, 2002,
1100AM EST Subject Lunch Hey Ira, Meet you at
the food trucks at noon! Deke
Send 1,000 bits from A to B, With an
acknowledgement, Over 100 meters of fiber
A
B
(2) Propagation Delay
0.0000004 sec

1. Insertion time

(3) Processing Delay
0.0001 sec
0.01 sec
11
Analysis of Delay
Send 1,000 bits from A to B, With an
acknowledgement, Over 100 meters of fiber
A
B
Data Insertion 0.0001 sec
Propagation 0.0000004 sec
Processing (B) 0.01 sec
Ack Insertion 0.001 sec
Propagation 0.0000004 sec
Processing (A) 0.01 sec
Total Elapsed Time 0.0211008 seconds
12
Analysis of Delay
A
Add 2 switches and a router to the path
B
S
S
R
Add 0.00002 sec
Add 0.00002 sec
Add 0.002 sec
New Total Elapsed Time 0.0231408 seconds
13
Summary of Delay Analysis

• Propagation delay is of little consequence in
  LANs, more of an issue for high bandwidth WANs.
• Queueing delays are rarely major contributors.
• Processing delay is almost always an issue.
• Retransmission delays can be major contributors
  to poor network performance.

14
Speaker Change
15
What Im going to talk about

• More on delay contributors, their causes and how
  to minimize them
• Protocol Stack behavior tuning
• Quality of Service (QoS)
• Performance measurement tools
• Operating System tuning examples
• General comments about things you can do

16
Recap Delay Contributors

• Processing Delay
• Retransmission Delay
• Queueing Delay
• Propagation Delay

17
Processing Delay

• Time it takes to process a packet at an
  end-station or network node. Depends on
• Network protocol complexity, application code,
  computational power at node, NIC efficiency etc
• Endstation Tuning
• Application Tuning

18
Endstation Tuning

• Good network hardware/NICs
• Correct speed/duplex settings
• Auto-negotiation problems
• Sufficient CPU
• Sufficient Memory
• Network Protocol Stack tuning
• Path MTU discovery, Jumbo Frames, TCP Window
  Scaling, SACK etc

19
Ethernet Bandwidth/Duplex mode

• Ethernet bandwidth 10, 100, 1000
• 10 Gigabit Ethernet soon
• Duplex modes half-duplex, full-duplex
• Auto-Negotiation
• Mismatch Detection
• CRC/Alignment errors
• Late Collisions

20
Application Tuning

• Optimize access to host resources
• Pay attention to Disk I/O issues
• Pay attention to Bus and Memory issues
• Know what concurrent activity may be interfering
  with performance of app
• Tuning application send/receive buffers
• Efficient application protocol design
• Positive end user feedback
• Subjective perception of performance

21
Retransmission Delay

• Causes
• Packet loss
• Bad hardware NICs, switches, routers,
  transmission lines
• Congestion and Queue drops
• Out of order packet delivery
• May be considered packet loss from applications
  perspective if it cant re-order packets
• Untimely delivery (delay)
• Some apps may consider a packet to be lost if
  they dont receive it in a timely fashion

22
Retransmission Delay (cont)

• Mitigating retransmission delay
• Ensure working equipment
• Although some packet loss is unavoidable eg.
  most transmission lines have a BER (Bit Error
  Rate)
• Reduce time to recover from packet loss
• Eg. Highly tuned network stack with more
  aggressive retransmission and recovery behavior
• Forward Error Correction (FEC)
• Very useful for time/delay sensitive applications
• Also, for cases when its expensive to retransmit
  data

23
Bit Errors on WAN paths

• Bit Error Rate (BER) specs for networking
  interfaces/circuits may not be low enough
• 1 bit-error in 10 billion bits
• Assuming 1500 byte packets
• Packet error rate 1 in 1 million
• 10 hops gt 1 in 100,000 packet drop rate

24
Queueing Delay

• Long queueing delays could be caused by lame
  hardware (switches/routers)
• Head of line blocking
• Insufficient switching fabric
• Insufficient horse power
• Unfavorable QoS treatment

25
Queueing Delay (cont)

• How to reduce
• Use good network hardware
• Improved network architecture
• Reduce number of switching/routing elements on
  the network path
• Richer network topology, more interconnections
• End user may not have influence over architecture
• Employ preferential queue scheduling algorithms
• Will discuss later in QoS section of talk

26
Propagation Delay

• Restricted by speed of light through transmission
  medium
• Cant be changed, but rarely a concern in the
  campus/LAN environment
• A concern in long distance paths (WAN), but
• Some steps can be taken to increase performance
  (throughput) on such paths

27
Other delays and bottlenecks

• Intermediary systems
• DNS
• Routing issues
• Route availability, asymmetric routing, routing
  protocol stability and convergence time
• Firewalls
• Tunnels (IPSec VPNs, IP in IP tunnels etc)
• Router hardware poor at encap/decap

28
Throughput

• Influenced by a number of variables
• All the delay factors we discussed
• Window size (for TCP)
• Bottleneck link capacity
• End station processing and buffering capacity

29
What Im going to talk about next

• Brief description of TCP/IP protocol
• How to improve TCP/IP performance

30
Transport TCP vs UDP

• Network apps use 2 main transport protocols
• TCP (Transmission Control Protocol)
• Connection oriented (telephone like service)
• Reliable guarantees delivery of data
• Flow control
• Examples Web (HTTP), Email (SMTP, IMAP)
• UDP (User Datagram Protocol)
• Connectionless (postal system like)
• Unreliable no guarantees of delivery
• Examples DNS, various types of streaming media

31
When to use TCP or UDP?

• Many common apps use TCP because its convenient
• TCP handles reliable delivery, retransmissions of
  lost packets, re-ordering, flow control etc
• You may want to use UDP if
• Delays introduced by ACKs are unacceptable
• TCP congestion avoidance and flow control
  measures are unsuitable for your application
• You want more control of how your data is
  transported over the network
• Highly delay/jitter sensitive apps often use UDP
• Audio-video conferencing etc

32
Network Stack Tuning

• Jumbo Frames
• Path MTU Discovery
• TCP Extensions
• Window Scaling - RFC 1323
• Fast Retransmit Fast Recovery
• Selective Acknowledgements

33
Jumbo Frames

• Increase MTU used at link layer, allowing larger
  maximum sized frames
• Increases Network Throughput
• Fewer larger frames means
• Fewer CPU interrupts and less processing overhead
  for a given data transfer size
• Some studies have shown Gigabit Ethernet using
  9000 byte jumbo frames provided 50 more
  throughput and used 50 less CPU!
• (default Ethernet MTU is 1500 bytes)

34
Jumbo Frames (cont)

• Pitfalls
• Not widely deployed yet
• Many network devices may not be capable of jumbo
  frames (theyll look like bad frames)
• May cause excessive IP fragmentation
• BER may have more impact on jumbo frames
• Eg. A single bit-error can cause a large amount
  of data to be lost and retransmitted
• May have negative impact on host processing
  requirements
• More memory for buffering, newer NICs

35
Path MTU Discovery

• MTU (Max Transmission Unit)
• Max sized frame allowed on the link
• Path MTU
• Min MTU on any network in the path between 2
  hosts
• IP Fragmentation Reassembly
• Path MTU Discovery
• MSS (Max Segment Size)
• What happens without PMTU discovery?
• Might select wrong MTU and cause fragmentation
• Suboptimal selection of TCP MSS (536 default?)

36
Path MTU Discovery (cont)
R1
MTU9000
IP fragmentation may occur
MTU4474
R2
A
MTU1500
R3
B
Path MTU is 1500
MTU9000
37
TCP Sliding Window

• TCP uses a flow control method called Sliding
  Window
• Allows sender to send multiple segments before it
  has to wait for an ACK
• Results in faster transfer rate coz sender
  doesnt have to wait for an ACK each time a
  packet is sent
• Receiver advertises a window size that tells the
  sender how much data it can send without waiting
  for ACK

38
TCP Sliding Window (cont)
39
Slow Start

• In actuality, TCP starts with small window and
  slowly ramps it up (upto rwin)
• Congestion Window (cwnd)
• controls startup and limits throughput in the
  face of congestion
• cwnd initialized to 1 segment
• cwnd gets larger after every new ACK
• cwnd gets smaller when packet loss is detected
• Slow Start is actually exponential

40
Congestion Avoidance

• Assumption packet loss is caused by congestion
• When congestion occurs, slow down transmission
  rate
• Reset cwnd to 1 if timeout
• Use slowstart until we reach the half way point
  where congestion occurred.
• Then use linear increase
• Increase cwnd by 1 segment/RTT

41
TCP Behavior

• Recovery after a loss can be very slow on todays
  high delay/bandwidth links
• (graph from Peter ONeill, NCAR)

42
TCP Throughput Acceleration
rtt (msec) 0-100Mbps (sec)
5 0.216
10 0.864
20 3.45
50 21.6
100 86.4
200 345
(From Phil Dykstra)
43
TCP Window Size Tuning

• TCP performance depends on
• Transfer rate (bandwidth)
• Round trip time
• BWDelay product
• TCP Window should be sized to be at least as
  large as the BWDelay product

44
BWDelay Product

• BWDelay product measures
• Amount of data that would fill the network pipe
• Buffer space required at sender and receiver to
  achieve the max possible TCP throughput
• Amount of unacknowledged data that TCP must
  handle in order to keep pipe full

45
BWDelay example

• A path from Penn to Stanford has
• Round trip time 60 ms
• Bandwidth 120 Mbps
• BW Delay
• 60/1000 sec 120 1000000 bits/sec
• 7200000 bits 7200 Kbits
• 900 Kbytes
• So TCP window should be at least 900KB

46
TCP Window Scaling

• RFC 1323 TCP Extensions for High Performance
• Allows scaling of TCP window size beyond 64KB (16
  bit window field)
• Introduces new TCP option
• Note In previous example, TCP needs to support
  Window Scaling to use 900KB window

47
Window Scaling Pitfalls

• Why not use large windows always?
• Might consume large memory resources
• May not be useful for all applications
• Isnt useful in the campus/LAN environment

48
Fast Retransmit Fast Recovery

• TCP required to send immediate D-ACK when
  out-of-order packet received
• After 3 D-ACKs, sending TCP retransmits only one
  segment
• Also perform congestion avoidance but not slow
  start

1
2
7
3
4
5
6
Packet loss, causing D-ACK
49
TCP Selective Acks (SACK)

• RFC 2018
• Allows TCP to efficiently recover from multiple
  segment losses within a window
• Without retransmitting entire window

50
Enough about TCP
51
Performance depends on App

• So, understand applications requirements (high
  throughput, low latency, low jitter), eg
• File Transfer using TCP
• Needs high throughput
• Intolerant of packet loss
• May be more tolerant of delay
• Interactive Video Conferencing application
• Tolerant of some loss
• More intolerant of delay and jitter

52
Quality of Service (QoS)

• A method to selectively allocate scarce network
  resources
• A mechanism to offer varying degrees of service
  to varying classes of traffic
• Service delay, jitter, proportion of link
  bandwidth etc

53
Quality of Service (QoS) cont

• Requires deployed QoS infrastructure
• Might require
• Traffic marking capabilities in hosts and network
  hardware
• Traffic classification and identification
  capabilities
• Multiple traffic queues with different service
  characteristics
• Different queue servicing algorithms
• Mechanisms to specify and enforce QoS policy
• Signalling mechanisms
• IEEE 802.1p, IP precedence, IntServ/RSVP,
  DiffServ, MPLS

54
Performance Measurement Tools

• To measure real performance of an app, you need
  to instrument the app with measurement code!
• However, independent measurement of some common
  network perf metrics can be done
• Two kinds
• Active and Passive measurement

55
Active Measurement

• Ping
• Traceroute
• Netperf http//www.netperf.org/
• Iperf http//dast.nlanr.net/Projects/Iperf/
• Pathchar ftp//ftp.ee.lbl.gov/pathchar/
• Pathrate http//www.pathrate.org/
• Mping

56
Passive Measurement

• OCxMON/PCMon
• Router/switch stats collected via
• SNMP
• Netflow, etc
• tcpdump, snoop, etherfind

57
Some tuning examples

• Microsoft Windows
• Newer versions Win98, Win2K, WinXP support many
  of the features (window scaling, PMTU discovery,
  SACK etc)
• May require registry tweaks to turn some of them
  on
• TCPTune A TCP Stack Tuner for Windows
• http//moat.nlanr.net/Software/TCPtune/

58
More tuning examples

• MacOS X
• need to find out more, who knows?
• Supports window scaling
• sysctl net.inet.tcp.rfc1323
• net.inet.tcp.rfc1323 1
• Socket buffer raising
• Kernel tunable kern.ipc.maxsockbuf
• TCP send/receive buffer tuning
• Tunables supported
• net.inet.tcp.sendspace
• net.inet.tcp.recvspace

59
More tuning examples

• Linux
• In /proc/sys/net/core/ set
• rmem_default
• rmem_max
• wmem_default
• wmem_max
• In /proc/sys/net/ipv4 set
• tcp_windowscaling
• tcp_sack

60
More tuning examples

• Solaris 2.x - 8
• ndd -set /dev/tcp tcp_max_buf xxx
• ndd -set /dev/tcp tcp_xmit_hiwat xxx
• ndd -set /dev/tcp tcp_recv_hiwat xxx
• ndd -set /dev/ip ip_path_mtu_discovery 1
• ndd -set /dev/tcp tcp_sack_permitted 2

61
Web100 Project

• http//www.web100.org/
• Enhance TCP capabilities with
• Better (finer grain) kernel instrumentation
• Automatic controls
• Availability
• Today Linux (patches for 2.4.16 kernel)
• Being ported to other operating systems.

62
Things you can do (WAN)

• Make sure app offers adequately sized receive
  windows and send buffers
• But dont run your system out of memory
• Find out your path RTT with ping
• Check your path with traceroute
• Determine bottleneck capacity and available
  bandwidth on path
• Make sure your OS uses Path MTU discovery
• Make sure your OS uses TCP Large Windows, Fast
  Retransmit, SACK

63
Things you can do (Campus)

• Check your host
• (80 of the problems)
• Check your host
• Bandwidth/Duplex problems
• Network stack tuning
• Application tuning
• Talk to campus networking folks

64
Conclusion

• Understand performance requirements of your
  application
• What are the issues?
• Campus/LAN environment
• WAN environment
• What can you do to ask for help?

65
Any Questions?

• Deke Kassabian
• deke_at_isc.upenn.edu
• Shumon Huque
• shuque_at_isc.upenn.edu