Performance Evaluation of JXTA-* Communication Layers Mathieu Jan PARIS Research Group Paris, November 2004

2 Outline Overview of JXTA communications layers Experimental setup and benchmarks used Evaluation of JXTA-J2SE Fast-Ethernet Myrinet Evaluation of JXTA-C Fast-Ethernet Myrinet

3 Motivations: performance of JXTA? Papers from Emir Halepovic Pipe service only No in-depth explanations about the results No precise details about the experimental setup Mainly JXTA 1.x Complex topologies (relays, etc) Not clear what is benchmarked Results from the P3 project =~ XtremWeb over JXTA

4

5

6 Project JuxTest A set of benchmarks for JXTA-J2SE 2.x Focuses of JXTA-J2SE Communications layers Uses JXTA Distributed Framework project (JDF) Co-written with David A. Noblet Undergraduate at the University of New Hampshire Source code soon available! (22/11) Results already on-lineon-line

7 JXTA-J2SE communications layers Endpoint service Pipe service JXTA Socket TCP, HTTP, etc

8 JXTA-J2SE communication layers JXTA Socket Java Socket interface Data-stream interface Reliability Pipe service Pipe ID Unreliable Endpoint service Peer ID Unreliable

9 Experimental setup Nodes 2.4GHz Intel P4, 1GB of RAM, linux Networks Fast-Ethernet (MTU of 1500) Myrinet (GM , MTU of 9000) JVMs Sun 1.4.2_01-b06 (-server -Xms256M -Xmx256M) IBM (-Xms256M -Xmx256M) JXTA-J2SE peers via JDF 1 rdv peer and 2 edge peers Benchmarks used RTT and protocol efficiency

Performance Evaluation of JXTA-J2SE on Fast-Ethernet

11 Raw bandwidth results of JXTA Socket Java socket = MB/s JXTA 2.3 = 9.72 MB/s JXTA = 9.48 MB/s

12 Raw latency results of JXTA Socket Java socket = < 0.1 ms JXTA 2.3 =~ 3 ms JXTA =~ 4 ms

13 Protocol efficiency of JXTA Socket 4 message elements in a JXTA message ACK_NUMBER Message payload + reliability EndpointRouterMsg (XML doc) Used by the endpoint router service 565 bytes! Not even needed! EndpointSourceAddress TCP address EndpointDestinationAddress TCP address + EndpointService (Net group) + Endpoint Router 1 byte = 909 bytes sent on the wires!

14 Tunning the output buffer size of JXTA Socket output buffer size = size of a pipe message Java socket = MB/s JXTA (512KB) = MB/s

15 Raw bandwidth results of unicast pipe Java socket = MB/s JXTA 2.3 = 9.59 MB/s JXTA = MB/s

16 Raw latency result of unicast pipe Java socket = < 0.1 ms JXTA 2.3 =~ 2 ms JXTA =~ 35 ms

17 Protocol efficiency of unicast pipe 4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress 1 byte = 877 bytes sent on the wires!

18 JXTA Socket vs unicast pipe (bandwidth) JXTA Socket = 9.72 MB/s JXTA unicast pipe = 9.59 MB/s

19 JXTA Socket vs unicast pipe (latency) JXTA Socket =~ 3 ms JXTA unicast pipe =~ 2 ms

20 Removing limit on message size (unicast pipe) Java socket = MB/s JXTA = MB/s

21 IBM JVM vs Sun JVM on unicast pipe JXTA 2.3 IBM =~ 1.2 ms JXTA 2.3 SUN =~ 1.8 ms

22 Raw bandwidth results of endpoint service Java socket = MB/s JXTA 2.3 = MB/s JXTA = MB/s

23 Raw latency results of endpoint service Java socket = < 100 us JXTA 2.3 IBM =~ 480 us JXTA 2.3 SUN =~ 730 us Issue 1228: 2 TCP messages for 1 JXTA message!

24 Protocol efficiency of endpoint service 3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress 1 byte = only 300 bytes sent on the wires! If we can say only … No EndpointRouterMsg message element Removed 565 bytes! Removed XML parsing

25 Cost of each layer (bandwidth)

26 Cost of each layer (latency)

Performance Evaluation of JXTA-J2SE on Myrinet Or how to desperately reach the Gb/s!

28 Experimental setup « Ethernet emulation » feature of GM No OS-bypass mode :-( Same benchmarks A different IP address when configuring JXTA Focus on bandwidth Removed limit on message size

29 JXTA Socket bandwidth results Results are not very good! Java socket = MB/s JXTA 2.3 = MB/s JXTA = MB/s

30 Tuning the output buffer size of JXTA Socket Above 80 MB/s

31 Unicast pipe bandwidth results Above 1 Gb/s! :-) Java socket = MB/s JXTA 2.3 = MB/s JXTA = MB/s

32 Endpoint service bandwidth results Java socket = MB/s JXTA 2.3 = MB/s JXTA = MB/s

Performance Evaluation of JXTA-C on Fast-Ethernet

34 Short overview of JXTA-C Needs Apache Portable Runtime (APR) Threads, Network, etc Newly added support for TCP communications PeerCom, Seoul (South Korea) Does not support rdv peers Need to use a J2SE rdv peer No JXTA Socket layer Only pipe service and endpoint service Several bug fixes Mostly EndpointRouterMsg TCP_NODELAY on Several TCP messages

35 Raw bandwidth results of unicast pipe Similar results compared to JXTA-J2SE JXTA-J2SE 2.3 = 11.1 MB/s JXTA-J2SE = 11.1 MB/s JXTA-C = 11.1 MB/s

36 Raw latency results of unicast pipe JXTA-J2SE 2.3 =~ 1.8 ms JXTA-J2SE =~ 35 ms JXTA-C =~ 2 ms

37 Protocol efficiency of unicast pipe 4 message elements in a JXTA message PAYLOAD EndpointRouterMsg (XML doc) EndpointSourceAddress EndpointDestinationAddress 1 byte = 834 bytes sent on the wires! JXTA-J2SE = 877 bytes + GatewayForward tag in EndpointRouter Waiting for Bernard Traversat answear - charset tag for each message element

38 Raw bandwidth results of endpoint service JXTA-J2SE 2.3 = MB/s JXTA-J2SE = MB/s JXTA-C = MB/s

39 Raw latency results of endpoint service JXTA-J2SE 2.3 =~ 0.8 ms JXTA-J2SE =~ 1 ms JXTA-C =~ 0.8 ms

40 Protocol efficiency of endpoint service 3 message elements in a JXTA message PAYLOAD EndpointSourceAddress EndpointDestinationAddress 1 byte = only 239 bytes sent on the wires! If we can say only … JXTA-J2SE = 300 bytes No EndpointRouterMsg Removing 595 bytes! Removed XML parsing

41 Unicast pipe vs endpoint service

Performance Evaluation of JXTA-C on Myrinet Still using « Ethernet emulation » feature of GM

43 Unicast pipe bandwidth results JXTA-J2SE 2.3 = 106 MB/s JXTA-J2SE = 136 MB/s JXTA-C = 107 MB/s

44 Endpoint service bandwidth results JXTA-J2SE 2.3 = MB/s JXTA-J2SE = MB/s JXTA-C = 109 MB/s

45 JXTA-C on top of PadicoTM :-) JXTA-C is working on top of PadicoTM! (17/11) « sed » modifications in apr Using marcel threads instead of pthread JXTA-C configured to use GDBM instead of SDBM Goal: stop using « ethernet emulation » of GM … But JXTA-C is not yet zero-copy :-( Performance evaluation in progress …

46 Conclusion Performance evaluation of the communication layers of JXTA-* (J2SE and C) Fast-Ethernet Myrinet Much better results compared to other related work and explained results Known costs of JXTA JXTA-J2SE nearly saturate a Fast-Ethernet link JXTA-J2SE on Myrinet bandwidth greater than 1Gb/s JXTA-J2SE > JXTA-C on Myrinet JXTA-C is not zero-copy

47 Ongoing work Performance evaluation of JXTA-C on top of PadicoTM JuxMem client in C for DIET Better knowledge of JXTA-C tricks ;-) Write papers GP2PC (1/12) on JXTA raw performance Fast-ethernet ? on JXTA tuned performance for grid computing Myrinet

48 Future work Benchmark secure and propagate pipes Update JuxTest results JXTA Unidirectional throughput test Plug-in for Ethereal An easier analysis of JXTA protocols