A Multicast Transport Driver for Globus XIO Karl Jeacle

ETNGRID 2005 Rationale  Would like to achieve high-speed bulk data delivery to multiple sites  Multicasting would make sense  Existing multicast research has focused on sending to a large number of receivers  But Grid is an applied example where sending to a moderate number of receivers would be extremely beneficial

ETNGRID 2005 Multicast availability  Deployment is a problem!  Protocols have been defined and implemented  Valid concerns about scalability; much FUD  “chicken & egg” means limited coverage  Clouds of native multicast  But can’t reach all destinations via multicast  So applications abandon in favour of unicast  What if we could multicast when possible…  …but fall back to unicast when necessary?

ETNGRID 2005 Multicast TCP?  TCP  “single reliable stream between two hosts”  Multicast TCP  “multiple reliable streams from one to n hosts”  May seem a little odd, but there is precedent…  TCP-XMO – Liang & Cheriton  M-TCP – Mysore & Varghese  M/TCP – Visoottiviseth et al  PRMP – Barcellos et al  SCE – Talpade & Ammar

ETNGRID 2005 ACK implosion

ETNGRID 2005 Building Multicast TCP  Want to test multicast/unicast TCP approach  But new protocol == kernel change  Widespread test deployment difficult  Build new TCP-like engine  Encapsulate packets in UDP  Run in userspace  Performance is sacrificed…  …but widespread testing now possible

ETNGRID 2005 TCP/IP/UDP/IP Sending Application TCP IP UDP IP Receiving Application TCP IP UDP IP If natively implemented test deployment

ETNGRID 2005 TCP engine  Where does initial TCP come from?  Could use BSD or Linux  Extracting from kernel could be problematic  More compact alternative  lwIP = Lightweight IP  Small but fully RFC-compliant TCP/IP stack  lwIP + multicast extensions = “TCP-XM”

ETNGRID 2005 TCP-XM overview  Primarily aimed at push applications  Sender initiated – advance knowledge of receivers  Opens sessions to n destination hosts simultaneously  Unicast is used when multicast not available  Options headers used to exchange multicast info  API changes  Sender incorporates multiple destination and group addresses  Receiver requires no changes  TCP friendly, by definition

ETNGRID 2005 TCP SYN SYNACK ACK FIN ACK DATA SenderReceiver

ETNGRID 2005 TCP-XM Sender Receiver 1 Receiver 2 Receiver 3

ETNGRID 2005 TCP-XM connection  Connection  User connects to multiple unicast destinations  Multiple TCP PCBs created  Independent 3-way handshakes take place  SSM or random ASM group address allocated  (if not specified in advance by user/application)  Group address sent as TCP option  Ability to multicast depends on TCP option

ETNGRID 2005 TCP-XM transmission  Data transfer  Data replicated/enqueued on all send queues  PCB variables dictate transmission mode  Data packets are multicast (if possible)  Retransmissions are unicast  Auto fall back/forward to unicast/multicast  Close  Connections closed as per unicast TCP

ETNGRID 2005 Fall back / fall forward  TCP-XM principle  “Multicast if possible, unicast when necessary”  Initial transmission mode is group unicast  Ensures successful initial data transfer  Fall forward to multicast on positive feedback  Typically after ~75K unicast data  Fall back to unicast on repeated mcast failure

ETNGRID 2005 TCP-XM reception  Receiver  No API-level changes  Normal TCP listen  Auto-IGMP join on TCP-XM connect  Accepts data on both unicast/multicast ports  tcp_input() accepts:  packets addressed to existing unicast destination…  …but now also those addressed to multicast group  Tracks how last n segs received (u/m)

ETNGRID 2005 Grid multicast?  How can multicast be used in Grid environment?  TCP-XM is new multicast-capable protocol  Globus is de-facto Grid middleware  Would like TCP-XM support in Globus…

ETNGRID 2005 Globus XIO  eXtensible Input Output library  Allows “i/o plugins” to Globus  API  Single POSIX-like API / set of semantics  Simple open/close/read/write API  Driver abstraction  Hides protocol details / Allows for extensibility  Stack of 1 transport & n transform drivers  Drivers can be selected at runtime

ETNGRID 2005 XIO architecture

ETNGRID 2005 XIO implementation

ETNGRID 2005 XIO/XM driver specifics  Two important XIO data structures 1.Handle  Returned to user when XIO framework ready  Used for all open/close/read/write calls  lwIP netconn connection structure used 2.Attribute  Used to set XIO driver-specific parameters…  … and TCP-XM protocol-specific options  List of destination addresses

ETNGRID 2005 XIO code example // init stack globus_xio_stack_init(&stack, NULL); // load drivers onto stack globus_xio_driver_load("tcpxm", &txdriver); globus_xio_stack_push_driver(stack, txdriver); // init attributes globus_xio_attr_init(&attr); globus_xio_attr_cntl(attr, txdriver, GLOBUS_XIO_TCPXM_SET_REMOTE_HOSTS, hosts, numhosts); // create handle globus_xio_handle_create(&handle, stack); // send data globus_xio_open(&handle, NULL, target); globus_xio_write(handle, "hello\n", 6, 1, &nbytes, NULL); globus_xio_close(handle, NULL);

ETNGRID 2005 One-to-many issues  Stack assumes one-to-one connections  XIO user interface requires modification  Needs support for one-to-many protocols  Minimal user API changes  Framework changes more significant  GSI is one-to-one  Authentication with peer on connection setup  But cannot authenticate with n peers  Need some form of “GSI-M”

ETNGRID 2005 LAN speed

ETNGRID 2005 LAN efficiency

ETNGRID 2005 WAN speed

ETNGRID 2005 WAN efficiency

ETNGRID 2005 Driver availability  Multicast transport driver for Globus XIO  Requires Globus 3.2 or later  Source code online  Sample client  Sample server  Driver installation instructions 

ETNGRID 2005 mcp & mcpd  Multicast file transfer application using TCP-XM  ‘ mcpd & ’ on servers  ‘ mcp file host1 host2… hostN ’ on client   Full source code online  FreeBSD, Linux, Solaris

ETNGRID 2005 All done!  Thanks for listening!  Questions?