Flow Control

1
Flow Control
With an Application to Storage Networks

• Ben Eckart
• ECE 6160
• 11-13-07

2
Outline

• Overview of Flow Control
• Flow Control Paradigms
• On/Off
• Window-based
• Rate-based
• Applications of Flow Control
• TCP windows, High Speed protocols
• QoS for differentiated services
• QoS for storage networks
• Research Topic
• Flow control for iSCSI storage network
  environments

3
What is Flow Control?

• Reliable communication needs both error
  correction, and buffer overflow protection
• Flow control mechanisms throttle source bandwidth
  or data flow to prevent buffer overflow
• Flow control is also used to shape network
  traffic to meet QoS needs

4
Flow Control vs. Congestion Control

• Flow control different than congestion control
  though terms sometimes used interchangeably
• Congestion control is reactive to packet loss and
  indicators of congestion
• Flow control is contingent only on single sender
  and receiver, depending on the constraints of the
  hosts, or the QoS considerations of the network

5
Flow Control Overview

• What situations call for flow control?
• Discrepancy in send and receive rates
• Discrepancy in sending rate and application
  retrieval rate
• Overflow prevention with bursty rates

6
Three Main Methods of Flow Control

• On/off
• Serial communication
• Ethernet has optional flow control with PAUSE
  frames
• Window-based
• TCPs advertised receive window
• TCP variants for High-Speed networks, or storage
  networks
• Rate-based
• Application level protocols for high-speed
  networks
• PA-UDP, Tsunami, applications for QoS for
  differentiated services or multimedia applications

7
On/Off
Data frame
Transmitter
Receiver
Buffer
On/Off Command
Sender
On
On
Off
Off
Time
Receiver
8
On/Off

• Implementation Serial communication
• On/Off can be implemented either in hardware or
  software
• In software, XON/XOFF characters are used
• Simple
• Inefficient
• In hardware, it is known as RTS/CTS
• Requires extra lines
• Does not reduce bandwidth on data lines

9
On/Off Ethernet Flow Control

• Implementation Ethernet Flow Control (802.3x)
• PAUSE frames, meant to be used temporarily for
  overloaded switches
• Standard is optional, not used much
• Potentially harmful to network 5
• Can break other QoS services
• No standard for initiating frames
• Point-to-Point Switches are already designed such
  that line cannot be overprovisioned

10
On/Off - Ethernet Flow Control

• It is actually more detrimental to flow control
  in the core than helpful. Flow control in the
  core can cause congestion in sections of the
  network that otherwise would not be congested. So
  how do you decide which segments to hold off when
  one of the segments gets congested? Even a
  network manager familiar with the traffic
  patterns of his/her network would, in many cases,
  be hard pressed to answer this question due to
  the dynamic nature of network traffic. And if
  particular links are constantly in a congested
  state, there is most likely a problem with the
  current implementation of the network. -Hewl
  ett Packard 5

11
Window-based Flow Control
Sender
ACK
ACK
Time
Receiver

• Packets are sent in bursts according to window
  size
• ACK is required before next window-sized burst
  can be sent
• This allows explicit buffer control since ACK and
  windows determine flow characteristics

12
Window-based Flow Control

• Sometimes called a sliding-window protocol

Segmented Data for Network
Window Size
ACKed
Not yet sent, but within window
Blocked from being sent
Sent, but no ACK received
13
TCP Flow Control

• TCPs window-based flow control is used in many
  different applications
• Window-based flow control has the benefit of
  being able to control data flow while maintaining
  data integrity and ensuring reliable communication

14
TCP Flow Control

• Advertised window is the minimum of the
  congestion window and flow control window.
• Window field is included in the TCP packet header

15
TCP Flow Control

• Receive Window How much free buffer space is
  available at receiver?
• Flow control works to throttle send rate so
  buffer doesnt overflow. Thus, the following must
  be true of the receive window 7
• LastByteRcvd LastByteRead RcvBuffer
• RcvWindow RcvBuffer LastByteRecvd
  LastByteRead

RecvWindow
Receiver
Sender
Data
Empty Buffer
Data
Advertised Window
16
Flow Control

• Sender must now keep
• LastByteSent LastByteAcked RcvWindow
• This is done with sliding window protocol
  discussed earlier
• r(source) windowSize packetSize / rtt
  

17
Tricky situations with TCP windows

• What happens when a receiver sits on a full
  buffer?
• Advertised window drops to zero, and sender has
  all sent packets ACKed.
• Deadlock!

18
Flow Control for High-Speed TCP

• High-speed networks have characteristically large
  bandwidth delay products
• These latencies cause havoc on normal flow
  control schemes which depend on synchronous
  communication of state information
• TCP set to BDP is optimal, but it still cannot
  respond to student changes in the network,
  optimal when network is relatively stable
• Survey of HS solutions in 6

19
Rate-based Flow Control

• Protocols for high-speed networks
• Window-based, On/Off are dependent on RTT
• Rate-based provides for fine tuning of flow, but
  protocol other considerations are involved which
  increase complexity

20
Rate-based Flow Control

• Feedback loop is usually maintained
• Receiver sends sender feedback
• Feedback can be explicit or implicit
• Explicit feedback tells sender rate to send at
• Implicit allows for sender to calculate best rate
  given received parameters

21
Rate-based Flow Control

• Application PA-UDP for file transfers
• Flow control using monitored system performance
• Used to prevent buffer overflows
• Flow control is related to disk performance, CPU
  performance, memory buffer sizes, and size of
  file to transfer

22
Rate-based Flow Control

• Application PA-UDP for file transfers

Sender
RATE/NACK
RATE/NACK
RATE/NACK
Time
Receiver
23
ARQ (Automatic Repeat-reQuest) for Flow Control

• What happens when ACK is not received? How can we
  ensure reliability with flow control mechanism?
• Stop and Wait (with timer)
• Go back N (TCP window)
• Selective Repeat (NACK)

24
Flow Control for QoS

• QoS is important in many applications
• Streaming multimedia
• VoIP, Video conferencing
• Storage networks
• Examples include
• Leaky bucket
• token bucket
• WFQ
• Round-robin

25
Leaky Bucket

• Leaky bucket can be implemented via a queue which
  sends packets from the queue in a fixed rate
• Leaky buckets can shape and smooth traffic that
  is often bursty in nature
• Image source wikipedia.com

26
Token Bucket

• Arriving packets require token to move into
  network
• Less restrictive than leaky bucket

Bucket Fills with Tokens at Fixed Rate
Requests arrive in bursts
Remove Token
Network
27
Methods of Queuing for Flow Control

• Another form of flow control occurs at the
  queuing level
• Switches and routers can decide to prioritize
  certain data flows given QoS agreements
• Methods
• Weighted Fair Queuing
• Priority Queuing
• Round Robin

28
Research in Flow Control for QoS

• How can flow control techniques be used to ensure
  QoS for network storage systems?
• Controlling Network Bandwidth to Support Storage
  QoS in International Workshop on Storage Network
  Architecture and Parallel I/Os (SNAPI), San
  Diego, CA, 2007

29
Flow Control, QoS and Storage Networks

• QoS for storage is much different than for
  storage networks because bandwidth comes into
  play
• Current QoS solutions for storage networks only
  consider IOPS, and not bandwidth requirements

30
Flow Control, QoS and Storage Networks

• Research by Junkil Ryu and Chanik Park propose
  QoS based on request size and network throughput
• Why does this work? Because request sizes change,
  and IOPS cannot account for these changes

31
Flow Control for Storage Networks

• Ryu and Park noted
• IOPS is related to network bandwidth and request
  size.
• Why do QoS solutions ignore these two parameters
  and focus only on IOPS?
• IOPS only can overprovision because of changing
  request sizes

32
Request Size Distributions
33
Relation between IOPS and BW

• IOPS r(stream) / avg(request size)
• Higher bandwidth allows for higher IOPS
• Lower request size with same bandwidth gives
  higher IOPS
• Common QoS parameter, IOPS, does not take this
  into account, and thus changing request sizes can
  overload storage network

34
System Overview
35
Flow Control, QoS and Storage Networks

• Network and Storage Resource Monitor (NSRM)
• Monitors IOPS and response time
• If service over QoS threshold, NSRM notifies QoS
  enforcer

36
Flow Control, QoS and Storage Networks

• QoS enforcer
• Finds the bottleneck resource
• Breaks QoS if necessary though tries to select
  victim that will not have QoS be affected
• Uses network bandwidth allocator to enforce QoS

37
Flow Control, QoS and Storage Networks

• Network Bandwidth Allocator (NBA)

38
Flow Control, QoS and Storage Networks

• Implementation
• Implemented on TCP layer
• Uses TCPs flow control by manipulating window
  and ACK rates
• NSRM and QoS enforcer implemented in iSCSI server

39
Performance Evaluation

• Controlling Bandwidth via window size and delay

40
Unfair allocation
32 KB to 64 KB
41
3MBPS 2KB
3MBPS 32KB
To 64KB
Possible overload, over 250 IOPS, To 16 KB
4MBPS 32KB
3MBPS 32KB
42
Results
43
Conclusion

• Flow Control can be used to set rates optimally
  among network given QoS constraints, speed
  limitations or buffer constraints
• Flow control applications exist for
• QoS guarantees in differentiated services
  networks
• QoS implementations for storage networks
• Optimal bandwidth considerations with high-speed
  protocols
• Bandwidth throttling for mismatched speeds
  between hosts communicating via Ethernet, Serial,
  other media

44
References

• J. Ryu and C. Park, Controlling Network Bandwidth
  to Support Storage QoS in International Workshop
  on Storage Network Architecture and Parallel I/Os
  (SNAPI), 2007, San Diego, CA
• V. Jacobson, Congestion avoidance and control,
  ACM SIGCOMM Proceedings, p.314-329, August 16-18,
  1988, Stanford, CA
• J. M. Jaffe, Bottleneck flow control. IEEE
  Transactions on Communications 29, 7 (July 1981),
  954-962
• M. Gerla and L. Kleinrock, Flow control A
  comparative survey. IEEE Trans. Commun. COM-28, 4
  (Apr. 1980), 553-574
• Vendors on flow control. Network World, 1999.
  online http//www.networkworld.com/netresources/
  0913flow2.html
• S. Keshav, "Flow control in high-speed networks
  with long delays", Proc. INET '92, Kobe, Japan,
  pp. 461-470, May 1992.
• J. Kurose and K. Ross. Computer Networking A
  Top-Down Approach. Addison Wesley, 2008

45
Questions?
?