A Multicast Transport Driver for Globus XIO

1
A Multicast Transport Driver for Globus XIO

• Karl Jeacle

karl.jeacle_at_cl.cam.ac.uk
2
Rationale

• Would like to achieve high-speed bulk data
  delivery to multiple sites
• Multicasting would make sense
• Existing multicast research has focused on
  sending to a large number of receivers
• But Grid is an applied example where sending to a
  moderate number of receivers would be extremely
  beneficial

3
Multicast availability

• Deployment is a problem!
• Protocols have been defined and implemented
• Valid concerns about scalability much FUD
• chicken egg means limited coverage
• Clouds of native multicast
• But cant reach all destinations via multicast
• So applications abandon in favour of unicast
• What if we could multicast when possible
• but fall back to unicast when necessary?

4
Multicast TCP?

• TCP
• single reliable stream between two hosts
• Multicast TCP
• multiple reliable streams from one to n hosts
• May seem a little odd, but there is precedent
• TCP-XMO Liang Cheriton
• M-TCP Mysore Varghese
• M/TCP Visoottiviseth et al
• PRMP Barcellos et al
• SCE Talpade Ammar

5
ACK implosion
6
Building Multicast TCP

• Want to test multicast/unicast TCP approach
• But new protocol kernel change
• Widespread test deployment difficult
• Build new TCP-like engine
• Encapsulate packets in UDP
• Run in userspace
• Performance is sacrificed
• but widespread testing now possible

7
TCP/IP/UDP/IP
8
TCP engine

• Where does initial TCP come from?
• Could use BSD or Linux
• Extracting from kernel could be problematic
• More compact alternative
• lwIP Lightweight IP
• Small but fully RFC-compliant TCP/IP stack
• lwIP multicast extensions TCP-XM

9
TCP-XM overview

• Primarily aimed at push applications
• Sender initiated advance knowledge of receivers
• Opens sessions to n destination hosts
  simultaneously
• Unicast is used when multicast not available
• Options headers used to exchange multicast info
• API changes
• Sender incorporates multiple destination and
  group addresses
• Receiver requires no changes
• TCP friendly, by definition

10
TCP
SYN
SYNACK
ACK
Sender
Receiver
DATA
ACK
FIN
ACK
FIN
ACK
11
TCP-XM
Receiver 1
Sender
Receiver 2
Receiver 3
12
TCP-XM connection

• Connection
• User connects to multiple unicast destinations
• Multiple TCP PCBs created
• Independent 3-way handshakes take place
• SSM or random ASM group address allocated
• (if not specified in advance by user/application)
• Group address sent as TCP option
• Ability to multicast depends on TCP option

13
TCP-XM transmission

• Data transfer
• Data replicated/enqueued on all send queues
• PCB variables dictate transmission mode
• Data packets are multicast (if possible)
• Retransmissions are unicast
• Auto fall back/forward to unicast/multicast
• Close
• Connections closed as per unicast TCP

14
Fall back / fall forward

• TCP-XM principle
• Multicast if possible, unicast when necessary
• Initial transmission mode is group unicast
• Ensures successful initial data transfer
• Fall forward to multicast on positive feedback
• Typically after 75K unicast data
• Fall back to unicast on repeated mcast failure

15
TCP-XM reception

• Receiver
• No API-level changes
• Normal TCP listen
• Auto-IGMP join on TCP-XM connect
• Accepts data on both unicast/multicast ports
• tcp_input() accepts
• packets addressed to existing unicast
  destination
• but now also those addressed to multicast group
• Tracks how last n segs received (u/m)

16
Grid multicast?

• How can multicast be used in Grid environment?
• TCP-XM is new multicast-capable protocol
• Globus is de-facto Grid middleware
• Would like TCP-XM support in Globus

17
Globus XIO

• eXtensible Input Output library
• Allows i/o plugins to Globus
• API
• Single POSIX-like API / set of semantics
• Simple open/close/read/write API
• Driver abstraction
• Hides protocol details / Allows for extensibility
• Stack of 1 transport n transform drivers
• Drivers can be selected at runtime

18
XIO architecture
19
XIO implementation
20
XIO/XM driver specifics

• Two important XIO data structures
• Handle
• Returned to user when XIO framework ready
• Used for all open/close/read/write calls
• lwIP netconn connection structure used
• Attribute
• Used to set XIO driver-specific parameters
• and TCP-XM protocol-specific options
• List of destination addresses

21
XIO code example

• // init stack
• globus_xio_stack_init(stack, NULL)
• // load drivers onto stack
• globus_xio_driver_load("tcpxm", txdriver)
• globus_xio_stack_push_driver(stack, txdriver)
• // init attributes
• globus_xio_attr_init(attr)
• globus_xio_attr_cntl(attr, txdriver,
  GLOBUS_XIO_TCPXM_SET_REMOTE_HOSTS, hosts,
  numhosts)
• // create handle
• globus_xio_handle_create(handle, stack)
• // send data
• globus_xio_open(handle, NULL, target)
• globus_xio_write(handle, "hello\n", 6, 1,
  nbytes, NULL)
• globus_xio_close(handle, NULL)

22
One-to-many issues

• Stack assumes one-to-one connections
• XIO user interface requires modification
• Needs support for one-to-many protocols
• Minimal user API changes
• Framework changes more significant
• GSI is one-to-one
• Authentication with peer on connection setup
• But cannot authenticate with n peers
• Need some form of GSI-M

23
LAN speed
24
LAN efficiency
25
WAN speed
26
WAN efficiency
27
Driver availability

• Multicast transport driver for Globus XIO
• Requires Globus 3.2 or later
• Source code online
• Sample client
• Sample server
• Driver installation instructions
• http//www.cl.cam.ac.uk/kj234/xio/

28
mcp mcpd

• Multicast file transfer application using TCP-XM
• mcpd on servers
• mcp file host1 host2 hostN on client
• http//www.cl.cam.ac.uk/kj234/mcp/
• Full source code online
• FreeBSD, Linux, Solaris

29
All done!

• Thanks for listening!
• Questions?