1. An Evaluation of Contemporary Commercial SOAP Implementations Presented by: Alex Ng Alex Ng Department of Computing, Macquarie University Shiping Chen CSIRO ICT Centre Paul Greenfield CSIRO ICT Centre AWSA 2004, Melbourne, 13-14 April 2004

2. Outline Background Experimental design Result discussion Related work Future work

3. Web Services Definition (W3C) A Web service is a software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format (specifically WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP-messages, typically conveyed using HTTP with an XML serialisation in conjunction with other Web- related standards. W3C Web Services Architecture Working Draft (8 August 2003)

4. W3C Web Services Conceptual Stack Source: Ferguson D. Secure, Reliable, Transacted Web Services: Architecture and Composition. MSDN technical article, Sept. 2003.

5. SOAP (Simple Object Access Protocol) Simple Extensible Based on XML De facto standard for XML Messaging in supporting Web Services V1.1(May/00) V1.2 (June/03) Request POST /StockQuote HTTP/1.1 Host: www.stockquoteserver.com Content-Type: text/xml Content-Length: nnnn SOAPAction: “Some-URI” <SOAP:Envelope xmlns:SOAP="urn:schemas.xmlsoap.org:soap.v1 "> <t:Transaction xmlns:t=”URI” mustUnderstand=”1”>5 DIS

6. Other Competitors of SOAP Web Services Use different transport mechanism Microsoft DIME (Direct Internet Message Encapsulation) Work with or without SOAP & HTTP send attachments of various types BEEP (Blocks Extensible Exchange Protocol) Connection oriented, min 257 channels, asynchronous message exchange Work with or without HTTP Use different Web Architecture REST (Representational State Transfer) An architectural style (Roy Fielding) Use XML/HTML, no need for SOAP

7. Performance Issues of SOAP Web Services Serialisation and de-serialisation Message size ( 6-8 times larger than binary) Introduce latency in transmission, storing or retrieving XML data XML Processing Parsing (encode/decode XML elements) Validation (conformation to a pre-defined schema) Transformation (XML other format)

8. Network settings Issues HTTP 1.1 chunking (avoids content-length calculation) HTTP 1.1 Keep-Alive (reduces connection set up cost) HTTP 1.1 Pipelining (handles multiple requests/connection) TCP-Delayed-ACK (defers receiver acknowledgement) Nagle Algorithm (defers the sending of small packets)

9. Evaluation of contemporary commercial SOAP implementations (1) What level of performance is shown by major commercial SOAP Web Services implementations? (2) How does the performance of SOAP compare to that offered by conventional non- SOAP technologies? Document/LiteralRPC/Literal Document/EncodedRPC/Encoded SOAP Body DocumentRPC SOAP Parameter Literal Encoded

10. Experimental Design Test Driver Platform Specific Runner Property.xml Test Document Result Log Target Application Server Target Web Service 100Mbps Ethernet 4x1.6Ghz, 3.6Gb Invoice: One Customer, 50 product lines Complex Message 20 customers InquiryMedium Message Single customer inquiry/update Simple Message Implementation Doc/ Lit Doc/ Enc RPC/ Lit RPC/ Enc Product A  Product B  Product C 

11. Encoding Style Simple Message Medium Message Complex Message Product A Doc/Lit873758121392 Doc/Enc15931397338842 RPC/Enc15421396938838 Product B Doc/ Lit870751321566 RPC/Enc15971403538999 Product CRPC/Enc16641456640273 No.of Char. Result: Message Size

12. Result: Latency

13. Result: Simple Message Throughput

14. Result: Medium Message Throughput

15. Result: Complex Message Throughput

16. Result: Serialisation/De-serialisation 23% 13% 30% 10%

17. Evaluation Summary SOAP Performance is affected by: Encoding style (Document/Literal encoding is the choice) Product of implementation (a sizable gap between two products using the same document/literal encoding) Performance delivered by the majority SOAP implementations: Simple Message – able to deliver reasonably good performance (4 to 6ms vs 1ms non-SOAP) Medium Message – (10 to 22ms vs 7ms non-SOAP) Complex Message – (20 to 63ms vs 15ms non-SOAP) De-serialisation overhead is higher than that of serialisation.

18. Related Work (1) The Extreme Lab, Indiana University Govindaraju, et. al. sending different size of linked list of doubles using different implementations: SOAPRMI, SUNRMI, and NEXUSRMI, and TCP. TCP is two orders of magnitude better than SOAPRMI Serialisation and de-serialisation are the bottlenecks. Chiu, et.al. sending different arrays of mathematical doubles. Suggested Optimization: Pull Parser, schema-specific XML parser, trie and HTTP 1.1. Kohlhoff and Steele: Compares SOAP against FIX and CDR SOAP latency is 2 to 3 times worse than CDR FIX is a text based encoding and outperforms CDR and CDR

19. Related Work (2) Dhawan: Compares the performance of.NET Remoting against ASP.NET ASP.NET XML serialisation is more efficient than.NET Remoting SOAP serialisation Davis and Parashar Different SOAP Implementations: SOAPRMI/Java1.1, Microsoft SOAP Toolkit SP2 in VB, Perl SOAP::Lite, Apache SOAP 2.2, and Apache Axis Alpha 3. inappropriate TCP options had caused 200ms delays in some implementations (MS SOAP, Apache SOAP and SOAP:Lite ) Conclusion: SOAP is in the orders of magnitude slower than JavaRMI and CORBA

20. Related Work (3) Benkner, et. al.: Compares the performance of JAXM implementations against JAXRPC in different package sizes (1x1024 -> 8x128) Different Implementations: Apache Axis 1.1 RC2, Sun JWSDP 1.1, Mind Electric GLUE4.0, Systinet WASP 4.5, Novell jBroker Web 2.0, and X-Treme XSOAP 1.2.20. Conclusion: JAXM & JAXRPC produce similar results Best performance result is achieved when using large packages of data, i.e. splitting 1MB of source data into four packages of 256KB each for JAXM in Systinet WASP implementation and 2x 512 for JAXRPC in Apache implementation.

21. Future Work Extend Evaluation to other SOAP Implementations, such as Apache Axis Identify other factors when SOAP over HTTP is used across wide-area networks

22. Your Feedback

23. Reference Chiu K., et. al. (2002) Investigating the Limits of SOAP Performance for Scientific Computing In Proceedings of 11th. IEEE International Symposium on High Performance Distributed Computing HPDC-11 2002 (HPDC'02) Benkner, S., Brandic, I., Dimitrov, A., et al. Performance of Java Web Services Implementations. In Proceedings of ICWS'03. Las Vegas, 2003 Davis, D. and Parashar, M. Latency Performance of SOAP Implementations. In Proceedings of IEEE Cluster Computing and the GRID 2002 (CCGRID'02) Dhawan, P. Performance Comparison: Exposing Existing Code as a Web Service, October 2001 (on-line) Accessed Date: 10 October 2002 http://msdn.microsoft.com/library/default.asp?url=/library/en- us/dnbda/html/bdadotnetarch11.asp Kohlhoff, C. and Steele, R. (2003) Evaluating SOAP for High Performance Business Applications: Real – Time Trading Systems. In Proceedings of WWW2003. Budapest, Hungary. Govindaraju, M., A. Slominski, et al. Requirements for and Evaluation of RMI Protocols for Scientific Computing. Supercomputing. IEEE, 2000