1. Effect Of Message Size and Number of Clients on WS Frameworks For CIS*6650.01 Service Oriented Computing Dariusz Grabka Gerett Commeford Jack Cole

2. April 3, 2007 2Effects on WS Frameworks Background Web Service Frameworks provide functionality to applications that they host, including: Web Service Frameworks provide functionality to applications that they host, including: SOAP message handling SOAP message handling XML parsing XML parsing data marshalling into and out of application space data marshalling into and out of application space Performance doing these mundane but intensive chores can greatly affect the overall throughput of an application Performance doing these mundane but intensive chores can greatly affect the overall throughput of an application

3. April 3, 2007 3Effects on WS Frameworks Goals The purpose of this experiment is to compare major Web Service frameworks to better under- stand their performance under conditions of: The purpose of this experiment is to compare major Web Service frameworks to better under- stand their performance under conditions of: increasing number of clients increasing number of clients increasing XML message length increasing XML message length We were particularly interested in conditions leading to unacceptable service response (a denial-of-service situation) We were particularly interested in conditions leading to unacceptable service response (a denial-of-service situation)

4. April 3, 2007 4Effects on WS Frameworks WSTest WSTest is a Web Service benchmark developed by Sun and augmented by Microsoft WSTest is a Web Service benchmark developed by Sun and augmented by Microsoft Results published by both sides in 2004 Results published by both sides in 2004 Each showed results that demonstrated superior performance for their own platform Each showed results that demonstrated superior performance for their own platform http://msdn2.microsoft.com/en-us/vstudio/aa700840.aspx http://msdn2.microsoft.com/en-us/vstudio/aa700840.aspx http://msdn2.microsoft.com/en-us/vstudio/aa700840.aspx Tests results were published for just a couple of cases Tests results were published for just a couple of cases Tests were performed on a single client, and on a network of 50 clients, with similar results Tests were performed on a single client, and on a network of 50 clients, with similar results

5. April 3, 2007 5Effects on WS Frameworks WSTest Four tests Four tests EchoVoid – bi-directional empty messages EchoVoid – bi-directional empty messages EchoStruct – echo an array of structures containing an int, float, and string EchoStruct – echo an array of structures containing an int, float, and string EchoList – echo a linked list of structures EchoList – echo a linked list of structures GetOrder – receive a simulated purchase order GetOrder – receive a simulated purchase order We have implemented EchoStruct as it provides a suitable range of testing options We have implemented EchoStruct as it provides a suitable range of testing options

6. April 3, 2007 6Effects on WS Frameworks Test Environments Hardware: 3 identical 2.4 GHz Pentium 4 systems 3 identical 2.4 GHz Pentium 4 systems 512 Mb RAM 512 Mb RAM 1 Gigabit throughput switch 1 Gigabit throughput switch Software (running under Windows Server SE 2003): Each platform hosts a different framework Each platform hosts a different framework IBM WebSphere Application Server Express 6.1 IBM WebSphere Application Server Express 6.1 Microsoft.NET with IIS 6.0 Microsoft.NET with IIS 6.0 Sun JAVA Application Server PE 9 Sun JAVA Application Server PE 9

7. April 3, 2007 7Effects on WS Frameworks Test Conditions One server servicing one or two clients; no additional programs active One server servicing one or two clients; no additional programs active All systems were removed from external networks All systems were removed from external networks Benchmarked at 11MBps total throughput, 98% efficiency between two machines Benchmarked at 11MBps total throughput, 98% efficiency between two machines

8. April 3, 2007 8Effects on WS Frameworks Test Applications The EchoStruct service was implemented The EchoStruct service was implemented Servers were developed in the native language for the framework Servers were developed in the native language for the framework Java for WebSphere and Sun AS Java for WebSphere and Sun AS C# for.NET C# for.NET A framework-free gSoap server was also developed A framework-free gSoap server was also developed Two types of clients were developed Two types of clients were developed C# (using.NET), creating client threads C# (using.NET), creating client threads standard C (using gSoap), creating client processes standard C (using gSoap), creating client processes Document-literal style was used Document-literal style was used

9. April 3, 2007 9Effects on WS Frameworks Test Applications Timing performed using the Windows Multi-media library with 1 mS resolution Timing performed using the Windows Multi-media library with 1 mS resolution Two metrics: Two metrics: Response time (start to end) Response time (start to end) User's view of service response User's view of service response Latency (round-trip time) Latency (round-trip time) Removes client-side overhead from the measurement Removes client-side overhead from the measurement Care was taken to launch multiple clients simultaneously Care was taken to launch multiple clients simultaneously Multiple transactions were used to ensure we were measuring service time and not client deployment Multiple transactions were used to ensure we were measuring service time and not client deployment

10. April 3, 2007 10Effects on WS Frameworks Test Applications Detailed experiments of: Detailed experiments of: Variable number of clients from 1 to 1200 Variable number of clients from 1 to 1200 Variable number of array structures (and therefore message length) from 1 to 1000 Variable number of array structures (and therefore message length) from 1 to 1000 Variable number of iterations from 1 to 2000 Variable number of iterations from 1 to 2000 Single experiments to provide insight into possible future directions Single experiments to provide insight into possible future directions These will be discussed at the end of this report These will be discussed at the end of this report

11. April 3, 2007 11Effects on WS Frameworks Assumptions All hardware platforms and underlying software perform essentially the same All hardware platforms and underlying software perform essentially the same Benchmarks shows some small variation in hard- drive write speeds (not considered) Benchmarks shows some small variation in hard- drive write speeds (not considered) Windows client scheduling algorithms are reasonable Windows client scheduling algorithms are reasonable Frameworks are reasonably configured "out of the box" Frameworks are reasonably configured "out of the box"

12. April 3, 2007 12Effects on WS Frameworks Array Tests & Results EchoStruct Array Test Each structure contains an integer, a float, and a string (which we set to 30 characters) Each structure contains an integer, a float, and a string (which we set to 30 characters) Serializes to approximately 120 bytes Serializes to approximately 120 bytes Tests were run for array sizes of n=5, 50, 100, 200, 500, and 1000. Tests were run for array sizes of n=5, 50, 100, 200, 500, and 1000. In each trial 5 simultaneous clients submitted 30 repetitions of the array echo test => 150*n echoed transactions In each trial 5 simultaneous clients submitted 30 repetitions of the array echo test => 150*n echoed transactions 30 trials were performed and times gathered from each individual client 30 trials were performed and times gathered from each individual client

13. April 3, 2007 13Effects on WS Frameworks Array Test – Total Client Time

14. April 3, 2007 14Effects on WS Frameworks Array Test – Latency Time

15. April 3, 2007 15Effects on WS Frameworks Client Tests & Results EchoStruct Client Test Same structure as other tests Same structure as other tests Tests were run for n simultaneous clients, n = 1, 5, 10, 20, 40, 60 Tests were run for n simultaneous clients, n = 1, 5, 10, 20, 40, 60 In each trial, clients submitted 30 repetitions of the 50 element array echo test => 1500*n echoed transactions In each trial, clients submitted 30 repetitions of the 50 element array echo test => 1500*n echoed transactions Multiple trials were performed and times gathered from each individual client Multiple trials were performed and times gathered from each individual client

16. April 3, 2007 16Effects on WS Frameworks Client Tests

17. April 3, 2007 17Effects on WS Frameworks Client Test – 600 Clients

18. April 3, 2007 18Effects on WS Frameworks Client Test – 1200 Clients

19. April 3, 2007 19Effects on WS Frameworks Iteration Tests & Results EchoStruct Iteration Test Same structure as other tests Same structure as other tests Tests were run for n repetitions by each client, n = 5, 30, 100, 200, 500, 1000, 2000 Tests were run for n repetitions by each client, n = 5, 30, 100, 200, 500, 1000, 2000 In each trial, 5 simultaneous clients submitted n repetitions of the 5 element array echo test => 25*n echoed transactions In each trial, 5 simultaneous clients submitted n repetitions of the 5 element array echo test => 25*n echoed transactions Multiple trials were performed and times gathered from each individual client Multiple trials were performed and times gathered from each individual client

20. April 3, 2007 20Effects on WS Frameworks Iteration Test

21. April 3, 2007 21Effects on WS Frameworks Conclusions Performance of the 3 frameworks is similar Performance of the 3 frameworks is similar Contrary to expectations, service times increase ~ linearly with both number of clients and size of transaction within limits of our testing Contrary to expectations, service times increase ~ linearly with both number of clients and size of transaction within limits of our testing As expected, number of iterations also produces a linear effect As expected, number of iterations also produces a linear effect Some results show interesting anomalies Some results show interesting anomalies We believe that there is still much to be discovered by pushing the frameworks harder We believe that there is still much to be discovered by pushing the frameworks harder

22. April 3, 2007 22Effects on WS Frameworks Further Investigation 1. Comparison to non-Framework-based Web services By using a gSoap C/C++ server, we can perform the same tests with minimal overhead By using a gSoap C/C++ server, we can perform the same tests with minimal overhead This will give a near best-case performance metric This will give a near best-case performance metric Preliminary testing shows more than 50% improvement Preliminary testing shows more than 50% improvement

23. April 3, 2007 23Effects on WS Frameworks Further Investigation 2. Distribute the clients over multiple platforms Running multiple clients on a single platform will eventually lead to saturation of the client, and results will no longer be reflecting only the server load Running multiple clients on a single platform will eventually lead to saturation of the client, and results will no longer be reflecting only the server load We postponed this effect by developing very fast clients. Better to use multiple client platforms. We postponed this effect by developing very fast clients. Better to use multiple client platforms. In a single test (WebSphere, 50 clients, array size 50) we found that time was 5% worse when 2 platforms were used In a single test (WebSphere, 50 clients, array size 50) we found that time was 5% worse when 2 platforms were used Implies some optimization for single clients Implies some optimization for single clients Implies non-saturation to that point in our testing Implies non-saturation to that point in our testing

24. April 3, 2007 24Effects on WS Frameworks Further Investigation 3. Comparison with client and server on same platform Some network transfer time can be removed from the experiment and then used to characterize other results Some network transfer time can be removed from the experiment and then used to characterize other results Preliminary testing shows that this introduces interesting scheduling complications Preliminary testing shows that this introduces interesting scheduling complications

25. April 3, 2007 25Effects on WS Frameworks Further Investigation 4. Comparison of different message styles RPC-literal and Document literal styles involve different activities from the frameworks RPC-literal and Document literal styles involve different activities from the frameworks Additional clients and servers are required Additional clients and servers are required Versions have been created using gSoap Versions have been created using gSoap Operation has been verified but no comparisons have been performed Operation has been verified but no comparisons have been performed

26. April 3, 2007 26Effects on WS Frameworks Further Investigation 5. Comparison of message wrapping Even though the services are functionally identical, the wrappings may be slightly different Even though the services are functionally identical, the wrappings may be slightly different We have used EtherReal to snoop the SOAP messages, but have not pursued this investigation yet We have used EtherReal to snoop the SOAP messages, but have not pursued this investigation yet