1. TCP-XM: Unicast-enabled Reliable Multicast
Karl Jeacle

2. Rationale
Would like to achieve high-speed bulk data delivery to multiple sites
Multicasting would make sense
Existing multicast research has focused on sending to a large number of receivers
But Grid is an applied example where sending to a moderate number of receivers would be extremely beneficial
ICCCN 2005

3. Multicast availability
Deployment is a problem!
Protocols have been defined and implemented
Valid concerns about scalability; much FUD
“chicken & egg” means limited coverage
Clouds of native multicast
But can’t reach all destinations via multicast
So applications abandon in favour of unicast
What if we could multicast when possible…
…but fall back to unicast when necessary?
ICCCN 2005

4. Multicast TCP? TCP Multicast TCP
“single reliable stream between two hosts”
Multicast TCP
“multiple reliable streams from one to n hosts”
May seem a little odd, but there is precedent…
TCP-XMO – Liang & Cheriton
M-TCP – Mysore & Varghese
M/TCP – Visoottiviseth et al
PRMP – Barcellos et al
SCE – Talpade & Ammar
ICCCN 2005

5. ACK implosion
ICCCN 2005

6. Building Multicast TCP
Want to test multicast/unicast TCP approach
But new protocol == kernel change
Widespread test deployment difficult
Build new TCP-like engine
Encapsulate packets in UDP
Run in userspace
Performance is sacrificed…
…but widespread testing now possible
ICCCN 2005

7. If natively implemented
TCP/IP/UDP/IP
Sending
Application
TCP IP
UDP
Receiving
If natively implemented
test deployment
ICCCN 2005

8. TCP engine Where does initial TCP come from? Could use BSD or Linux
Extracting from kernel could be problematic
More compact alternative
lwIP = Lightweight IP
Small but fully RFC-compliant TCP/IP stack
lwIP + multicast extensions = “TCP-XM”
ICCCN 2005

9. TCP-XM overview Primarily aimed at push applications
Sender initiated – advance knowledge of receivers
Opens sessions to n destination hosts simultaneously
Unicast is used when multicast not available
Options headers used to exchange multicast info
API changes
Sender incorporates multiple destination and group addresses
Receiver requires no changes
TCP friendly, by definition
ICCCN 2005

10. TCP
SYN
SYNACK
ACK
Sender
Receiver
DATA
ACK
FIN
ACK
FIN
ACK
ICCCN 2005

11. TCP-XM
Receiver 1
Sender
Receiver 2
Receiver 3
ICCCN 2005

12. TCP-XM connection Connection
User connects to multiple unicast destinations
Multiple TCP PCBs created
Independent 3-way handshakes take place
SSM or random ASM group address allocated
(if not specified in advance by user/application)
Group address sent as TCP option
Ability to multicast depends on TCP option
ICCCN 2005

13. TCP Group Option New group option sent in all TCP-XM SYN packets
kind=50
len=6
Multicast Group Address
1 byte
1 byte
4 bytes
New group option sent in all TCP-XM SYN packets
Non TCP-XM hosts will ignore (no option in SYNACK)
Presence implies multicast capability
Sender will automatically revert to unicast
ICCCN 2005

14. TCP-XM transmission Data transfer Close
Data replicated/enqueued on all send queues
PCB variables dictate transmission mode
Data packets are multicast (if possible)
Retransmissions are unicast
Auto fall back/forward to unicast/multicast
Close
Connections closed as per unicast TCP
ICCCN 2005

15. TCP-XM protocol states
ICCCN 2005

16. Fall back / fall forward
TCP-XM principle
“Multicast if possible, unicast when necessary”
Initial transmission mode is group unicast
Ensures successful initial data transfer
Fall forward to multicast on positive feedback
Typically after ~75K unicast data
Fall back to unicast on repeated mcast failure
ICCCN 2005

17. Multicast feedback Option
kind=51
len=3
% Multicast Packets Received
1 byte
1 byte
1 byte
New feedback option sent in ACKs from receiver
Only used between TCP-XM hosts
Indicates % of last n packets received via multicast
Used by sender to fall forward to multicast transmission
ICCCN 2005

18. TCP-XM reception Receiver No API-level changes Normal TCP listen
Auto-IGMP join on TCP-XM connect
Accepts data on both unicast/multicast ports
tcp_input() accepts:
packets addressed to existing unicast destination…
…but now also those addressed to multicast group
Tracks how last n segs received (u/m)
ICCCN 2005

19. API changes Only relevant if natively implemented! Sender API changes
New connection type
Connect to port on array of destinations
Single write sends data to all hosts
TCP-XM in use:
conn = netconn_new(NETCONN_TCPXM);
netconn_connectxm(conn, remotedest, numdests, group, port);
netconn_write(conn, data, len, …);
ICCCN 2005

20. PCB changes
Every TCP connection has an associated Protocol Control Block (PCB)
TCP-XM adds:
struct tcp_pcb { …
struct tcp_pcb *firstpcb;/* first of the mpcbs */
struct tcp_pcb *nextm; /* next tcpxm pcb */
enum tx_mode txmode; /* unicasting or multicasting */
u8_t nrtxm; /* number of retransmits for multicast */
u32_t nrtxmtime; /* time since last retransmit */
u32_t mbytessent; /* total bytes sent via multicast */
u32_t mbytesrcvd; /* total bytes received via multicast */
u32_t ubytessent; /* total bytes sent via unicast */
u32_t ubytesrcvd; /* total bytes received via unicast */
struct segrcv msegrcv[128]; /* ismcast boolean for last n segs */
u8_t msegrcvper; /* % of last segs received via mcast */
u8_t msegrcvcnt; /* counter for segs recvd via mcast */
u8_t msegsntper; /* % of last segs delivered via mcast */ }
ICCCN 2005

21. Linking PCBs *next points to the next TCP session
nextm M1
next
nextm U1
next
nextm M2
next
nextm M3
next
nextm U2
*next points to the next TCP session
*nextm points to the next TCP session that’s part of a particular TCP-XM connection
Minimal timer and state machine changes
ICCCN 2005

22. What happens to the cwin?
Multiple receivers
Multiple PCBs
Multiple congestion windows
Default to min(cwin)
i.e. send at rate of slowest receiver
Is this really so bad?
Compare to time taken for n unicast transfers
ICCCN 2005

23. LAN speed
ICCCN 2005

24. LAN efficiency
ICCCN 2005

25. WAN speed
ICCCN 2005

26. WAN efficiency
ICCCN 2005

27. Multiple TCP-XM flows
ICCCN 2005

28. TCP-XM vs TCP flows
ICCCN 2005

29. Protocol performance
ICCCN 2005

30. Protocol efficiency
ICCCN 2005

31. mcp & mcpd Multicast file transfer application using TCP-XM
‘mcpd &’ on servers
‘mcp file host1 host2… hostN’ on client
Full source code online
FreeBSD, Linux, Solaris
ICCCN 2005

32. All done!
Thanks for listening!
Questions?
ICCCN 2005