Hybrid Reliable Multicast with TCP-XM

1
Hybrid Reliable Multicast with TCP-XM
Karl Jeacle

• Jon Crowcroft

Marinho P Barcellos, UNISINOS Stefano Pettini, ESA
2
Rationale

• Would like to achieve high-speed bulk data
  delivery to multiple sites
• Multicasting would make sense
• Existing multicast research has focused on
  sending to a large number of receivers
• But Grid is an applied example where sending to a
  moderate number of receivers would be extremely
  beneficial

3
Multicast availability

• Deployment is a problem!
• Protocols have been defined and implemented
• Valid concerns about scalability much FUD
• chicken egg means limited coverage
• Clouds of native multicast
• But cant reach all destinations via multicast
• So applications abandon in favour of unicast
• What if we could multicast when possible
• but fall back to unicast when necessary?

4
Multicast TCP?

• TCP
• single reliable stream between two hosts
• Multicast TCP
• multiple reliable streams from one to n hosts
• May seem a little odd, but there is precedent
• TCP-XMO Liang Cheriton
• M-TCP Mysore Varghese
• M/TCP Visoottiviseth et al
• PRMP Barcellos et al
• SCE Talpade Ammar

5
ACK implosion
6
Building Multicast TCP

• Want to test multicast/unicast TCP approach
• But new protocol kernel change
• Widespread test deployment difficult
• Build new TCP-like engine
• Encapsulate packets in UDP
• Run in userspace
• Performance is sacrificed
• but widespread testing now possible

7
TCP/IP/UDP/IP
8
TCP engine

• Where does initial TCP come from?
• Could use BSD or Linux
• Extracting from kernel could be problematic
• More compact alternative
• lwIP Lightweight IP
• Small but fully RFC-compliant TCP/IP stack
• lwIP multicast extensions TCP-XM

9
TCP-XM overview

• Primarily aimed at push applications
• Sender initiated advance knowledge of receivers
• Opens sessions to n destination hosts
  simultaneously
• Unicast is used when multicast not available
• Options headers used to exchange multicast info
• API changes
• Sender incorporates multiple destination and
  group addresses
• Receiver requires no changes
• TCP friendly, by definition

10
TCP
SYN
SYNACK
ACK
Sender
Receiver
DATA
ACK
FIN
ACK
FIN
ACK
11
TCP-XM
Receiver 1
Sender
Receiver 2
Receiver 3
12
TCP-XM connection

• Connection
• User connects to multiple unicast destinations
• Multiple TCP PCBs created
• Independent 3-way handshakes take place
• SSM or random ASM group address allocated
• (if not specified in advance by user/application)
• Group address sent as TCP option
• Ability to multicast depends on TCP option

13
TCP Group Option
kind50
len6
Multicast Group Address
1 byte
1 byte
4 bytes

• New group option sent in all TCP-XM SYN packets
• Non TCP-XM hosts will ignore (no option in
  SYNACK)
• Presence implies multicast capability
• Sender will automatically revert to unicast

14
TCP-XM transmission

• Data transfer
• Data replicated/enqueued on all send queues
• PCB variables dictate transmission mode
• Data packets are multicast (if possible)
• Retransmissions are unicast
• Auto fall back/forward to unicast/multicast
• Close
• Connections closed as per unicast TCP

15
TCP-XM protocol states
16
Fall back / fall forward

• TCP-XM principle
• Multicast if possible, unicast when necessary
• Initial transmission mode is group unicast
• Ensures successful initial data transfer
• Fall forward to multicast on positive feedback
• Typically after 75K unicast data
• Fall back to unicast on repeated mcast failure

17
Multicast feedback Option
kind51
len3
Multicast Packets Received
1 byte
1 byte
1 byte

• New feedback option sent in ACKs from receiver
• Only used between TCP-XM hosts
• Indicates of last n packets received via
  multicast
• Used by sender to fall forward to multicast
  transmission

18
TCP-XM reception

• Receiver
• No API-level changes
• Normal TCP listen
• Auto-IGMP join on TCP-XM connect
• Accepts data on both unicast/multicast ports
• tcp_input() accepts
• packets addressed to existing unicast
  destination
• but now also those addressed to multicast group
• Tracks how last n segs received (u/m)

19
API changes

• Only relevant if natively implemented!
• Sender API changes
• New connection type
• Connect to port on array of destinations
• Single write sends data to all hosts
• TCP-XM in use
• conn netconn_new(NETCONN_TCPXM)
• netconn_connectxm(conn, remotedest, numdests,
  group, port)
• netconn_write(conn, data, len, )

20
PCB changes

• Every TCP connection has an associated Protocol
  Control Block (PCB)
• TCP-XM adds
• struct tcp_pcb
• struct tcp_pcb firstpcb/ first of the mpcbs
  /
• struct tcp_pcb nextm / next tcpxm pcb /
• enum tx_mode txmode / unicasting or
  multicasting /
• u8_t nrtxm / number of
  retransmits for multicast /
• u32_t nrtxmtime / time since last
  retransmit /
• u32_t mbytessent / total bytes sent
  via multicast /
• u32_t mbytesrcvd / total bytes
  received via multicast /
• u32_t ubytessent / total bytes sent
  via unicast /
• u32_t ubytesrcvd / total bytes
  received via unicast /
• struct segrcv msegrcv128 / ismcast boolean
  for last n segs /
• u8_t msegrcvper / of last segs
  received via mcast /
• u8_t msegrcvcnt / counter for segs
  recvd via mcast /
• u8_t msegsntper / of last segs
  delivered via mcast /

21
Linking PCBs

• next points to the next TCP session
• nextm points to the next TCP session thats part
  of a particular TCP-XM connection
• Minimal timer and state machine changes

22
What happens to the cwin?

• Multiple receivers
• Multiple PCBs
• Multiple congestion windows
• Default to min(cwin)
• i.e. send at rate of slowest receiver
• Is this really so bad?
• Compare to time taken for n unicast transfers

23
LAN speed
24
LAN efficiency
25
WAN speed
26
WAN efficiency
27
Multiple TCP-XM flows
28
TCP-XM vs TCP flows
29
Protocol performance
30
Protocol efficiency
31
Grid multicast?

• How can multicast be used in Grid environment?
• TCP-XM is new multicast-capable protocol
• Globus is de-facto Grid middleware
• Would like TCP-XM support in Globus

32
Globus XIO

• eXtensible Input Output library
• Allows i/o plugins to Globus
• API
• Single POSIX-like API / set of semantics
• Simple open/close/read/write API
• Driver abstraction
• Hides protocol details / Allows for extensibility
• Stack of 1 transport n transform drivers
• Drivers can be selected at runtime

33
XIO architecture
34
XIO implementation
35
XIO/XM driver specifics

• Two important XIO data structures
• Handle
• Returned to user when XIO framework ready
• Used for all open/close/read/write calls
• lwIP netconn connection structure used
• Attribute
• Used to set XIO driver-specific parameters
• and TCP-XM protocol-specific options
• List of destination addresses

36
XIO code example

• // init stack
• globus_xio_stack_init(stack, NULL)
• // load drivers onto stack
• globus_xio_driver_load("tcpxm", txdriver)
• globus_xio_stack_push_driver(stack, txdriver)
• // init attributes
• globus_xio_attr_init(attr)
• globus_xio_attr_cntl(attr, txdriver,
  GLOBUS_XIO_TCPXM_SET_REMOTE_HOSTS, hosts,
  numhosts)
• // create handle
• globus_xio_handle_create(handle, stack)
• // send data
• globus_xio_open(handle, NULL, target)
• globus_xio_write(handle, "hello\n", 6, 1,
  nbytes, NULL)
• globus_xio_close(handle, NULL)

37
One-to-many issues

• Stack assumes one-to-one connections
• XIO user interface requires modification
• Needs support for one-to-many protocols
• Minimal user API changes
• Framework changes more significant
• GSI is one-to-one
• Authentication with peer on connection setup
• But cannot authenticate with n peers
• Need some form of GSI-M

38
Driver availability

• Multicast transport driver for Globus XIO
• Requires Globus 3.2 or later
• Source code online
• Sample client
• Sample server
• Driver installation instructions
• http//www.cl.cam.ac.uk/kj234/xio/

39
mcp mcpd

• Multicast file transfer application using TCP-XM
• mcpd on servers
• mcp file host1 host2 hostN on client
• http//www.cl.cam.ac.uk/kj234/mcp/
• Full source code online
• FreeBSD, Linux, Solaris

40
All done!

• Thanks for listening!
• Questions?