1. A Case Study of Web Server Benchmarking Using Parallel WAN Emulation Carey Williamson Rob Simmonds Martin Arlitt University of Calgary

2. Web Benchmarking with IP-TNE 2 Network Emulation A hybrid performance evaluation methodology that combines aspects of implementation with simulation modeling A network emulator is a network simulator with an interface that allows client applications to interact with it in real-time “A simulator that talks back” (IP packets) Why? Provides a reliable, repeatable test environment for distributed applications Internet games, video conferencing, Web,...

3. Simulation Real World

4. Web Benchmarking with IP-TNE 4 IP-TNE Internet Protocol Traffic and Network Emulator The Internet Protocol Traffic and Network Emulator (IP-TNE) is a network emulator based on IP-TN (packet-level IP network simulator) Enables interaction between IP based clients via an IP-TN simulated network (in real time!) Distributed applications can interact with IP-TNE without modification

5. Web Benchmarking with IP-TNE 5 IP-TNE Overview

6. Web Benchmarking with IP-TNE 6 IP-TNE Overview (cont’d) CCTKit with real-time extensions provides an environment for fast network emulation (PDES) IP-TNE provides routing methods suitable for shared environments and dedicated test environments Now has HTTP and TCP client models that can be used for Web server benchmarking

7. Web Benchmarking with IP-TNE 7 So What? Flexible routing model support High-performance packet reading and writing via raw sockets (1 Gbps) Can model an arbitrary IP internetwork Detailed IP protocol models IPv4, ICMP, ping, traceroute, pchar, MTU,... Supports parallel execution on shared memory multiprocessors “Blazingly fast!” - CLW, 2002

8. Web Benchmarking with IP-TNE 8 Example: Web Benchmarking Web Server Client 1 Client 2 Client 3 Client C...

9. Web Benchmarking with IP-TNE 9 WAN Emulation (1 of 3) Web Server Client 1 Client 2 Client 3 Client C... “Centralized” Approach

10. Web Benchmarking with IP-TNE 10 WAN Emulation (2 of 3) Web Server Client 1 Client 2 Client 3 Client C... “Shim” Approach (NISTnet, DummyNet, WASP)

11. Web Benchmarking with IP-TNE 11 WAN Emulation (3 of 3) Web Server Client 1 Client 2 Client 3 Client C... Our IP-TNE Approach

12. Web Benchmarking with IP-TNE 12 Objectives of Case Study Evaluate new approach to WAN emulation, and demonstrate feasibility Confirm prior results by Nahum et al. on effects of WAN conditions on Web server performance How fast can Apache Web server go? How fast can IP-TNE go?

13. Web Benchmarking with IP-TNE 13 Experimental Setup IP-TNE on Compaq ES-40 (4 CPU) Apache (1.3.23) on another ES-40 Gigabit Ethernet (1 Gbps) in between OS is Compaq Tru64 (v5.1A) ANML for defining network model e.g., simple regular WAN topology

14. Web Benchmarking with IP-TNE 14

15. Web Benchmarking with IP-TNE 15

16. Web Benchmarking with IP-TNE 16 Web Workload Model Static content only Closed-loop workload generator Fixed-size Web objects Small (1 KB) Large (64 KB) Variable-size Web objects Median 3 KB Mean 9 KB Pareto heavy tail (alpha = 1.2) Zipf-like document popularity profile

17. Web Benchmarking with IP-TNE 17 Performance Metrics Two primary metrics HTTP transaction rate (trans/sec) Network throughput (Mbps) Several secondary metrics Response time Connection failure rate Packet loss rate...

18. Web Benchmarking with IP-TNE 18 Results (Fig. 4a) For 1 KB transfers with HTTP/1.0: Single client: 170 transactions/sec Transaction rate scales up with number of clients up to about H = 32 Transaction rate flattens, then drops sharply as num clients is increased more (closed loop) Peak rate achieved: 3800 trans/sec Peak throughput approximately 40 Mbps Transaction rate is (strongly) inversely related to the client round trip time (RTT)

19. Web Benchmarking with IP-TNE 19

20. Web Benchmarking with IP-TNE 20

21. Web Benchmarking with IP-TNE 21

22. Web Benchmarking with IP-TNE 22

23. Web Benchmarking with IP-TNE 23

24. Web Benchmarking with IP-TNE 24 Results (Fig. 4b) For 64 KB transfers with HTTP/1.0: Single client: 18 transactions/sec Transaction rate scales up with number of clients up to about H = 32 Transaction rate flattens, then drops slightly as num clients is increased more Peak rate achieved: 220 trans/sec Peak throughput approximately 115 Mbps Transaction rate is (weakly) inversely related to the client round trip time (RTT)

25. Web Benchmarking with IP-TNE 25

26. Web Benchmarking with IP-TNE 26 Results (Fig. 4c) For variable-size transfers with HTTP/1.0: Single client: 60 transactions/sec Transaction rate scales up with number of clients up to about H = 32 Transaction rate flattens, then drops as num clients is increased more Peak rate achieved: 1300 trans/sec Peak throughput approximately 90 Mbps Transaction rate is inversely related to the client round trip time (RTT) Behaviour is in between 1 KB and 64 KB results

27. Web Benchmarking with IP-TNE 27

28. Web Benchmarking with IP-TNE 28 Results (Fig. 5a) Concurrent connections with HTTP/1.0: Single client: 600 transactions/sec Qualitatively similar results to before, except that fewer clients are needed to drive the server to full load Conceptually concurrent connections are no different than adding more clients

29. Web Benchmarking with IP-TNE 29

30. Web Benchmarking with IP-TNE 30 Results (Fig. 5b) Persistent connections with HTTP/1.1: Single client: 300 transactions/sec Qualitatively similar results to before, except that transaction rate is about 70% higher than for HTTP/1.0 (since multiple HTTP req’s per TCP conn) Peak transaction rate 6500 trans/sec Much less dependency on RTT effects

31. Web Benchmarking with IP-TNE 31

32. Web Benchmarking with IP-TNE 32 Results (Fig. 5c) Pipelined persistent connections with HTTP/1.1: Single client: 800 transactions/sec Qualitatively similar results to before, except that transaction rate is about 100% higher than for HTTP/1.0 Peak transaction rate 7600 trans/sec Much less dependency on RTT effects

33. Web Benchmarking with IP-TNE 33

34. Web Benchmarking with IP-TNE 34 Results (Fig. 6a) Effect of WAN RTT delays: Increasing the per-link propagation delay increases the client RTT delay, which in turn reduces the transaction rate and throughput (as expected) As RTT increases, more and more clients are needed in order to drive the Web server to full load Similar to [Nahum et al. 2001]

35. Web Benchmarking with IP-TNE 35

36. Web Benchmarking with IP-TNE 36 Results (Fig. 6b) Effect of bandwidth asymmetry: For asymmetric access technologies such as ADSL (Asymmetric Digital Subscriber Line), the upstream link from the client to the server can sometimes be the bottleneck for TCP, even though it is primarily carrying ACKs only Depends on normalized bandwidth ratio Greater asymmetry, worse performance

37. Web Benchmarking with IP-TNE 37

38. Web Benchmarking with IP-TNE 38 Results (Fig. 6c) Effect of WAN packet losses: Decreasing the router queue size at the bottleneck link increases the packet loss ratio (as expected) As the level of packet loss increases, the HTTP transaction rate and the network throughput decrease Similar results to [Nahum et al. 2001]

39. Web Benchmarking with IP-TNE 39

40. Web Benchmarking with IP-TNE 40 Results (MTU Size) Effect of MTU size: The smaller the MTU size, the lower the transaction rate and network throughput Reasons: Smaller MTU size increases the relative header overhead per packet Smaller MTU size increases the number of packets that need to be sent Smaller MTU size (and MSS) increases the number of RTT’s required by TCP

41. Web Benchmarking with IP-TNE 41 Summary and Conclusions The IP-TNE is a useful tool for Web server benchmarking Demonstrates feasibility of WAN emulation using a single computer Confirms prior results by Nahum et al. studying the effects of WAN conditions on Web server performance Demonstrates performance advantages of HTTP/1.1

42. Web Benchmarking with IP-TNE 42 Future Work? Increase traffic to your Web site!!! Guarantee 20% increase in traffic! 1000’s of new clients per month!!!!! Send $$$ to carey@cpsc.ucalgary.ca

43. Web Benchmarking with IP-TNE 43 Future Work with IP-TNE Validation of IP-TNE (and IP-TN) Benchmarking IP-TNE vs IP-TNE Benchmarking Web caching appliances Evaluating SRPT scheduling in WAN setting Connection/packet-level scheduling algorithms Evaluating CATNIP approach to TCP/IP Evaluating portable (wireless) Web servers Workload sensitivities (Zipf, Pareto, corr, mods) Experiments with dynamic content (CGI, etc) Asymmetric networks, Ensemble-TCP Parallel TCP connections: friend or foe? Evaluating effect of TCP SACK in WAN