1. 4/6/01EMTM 5531 EMTM 553: E-commerce Systems Lecture 4A: Performance and Scalability Insup Lee Department of Computer and Information Science University of Pennsylvania lee@cis.upenn.edu www.cis.upenn.edu/~lee

2. 4/6/01EMTM 5532 System Architecture Client Tier 1 Web Server Tier 3Tier 2Tier N Application Server Database Server DMS

3. 4/6/01EMTM 5533 Goals and Approaches Internet-based E-Commerce has made system growth more rapid and dynamic. Improving performance and reliability to provide –Higher throughput –Lower latency (i.e., response time) –Increase availability Approaches –Scaling network and system infrastructure oHow performance, redundancy, and reliability are related to scalability –Load balancing –Web caching

4. 4/6/01EMTM 5534 Network Hardware Review Firewall –Restricts traffic based on rules and can “protect” the internal network from intruders Router –Directs traffic to a destination based on the “best” path; can communicate between subnets Switch –Provides a fast connection between multiple servers on the same subnet Load Balancer –Takes incoming requests for one “virtual” server and redirects them to multiple “real” servers SOURCE: SCIENT

5. 4/6/01EMTM 5535 Case Study: Consumer Retail eBusiness SOURCE: SCIENT

6. 4/6/01EMTM 5536 Scaling the existing network SOURCE: SCIENT

7. 4/6/01EMTM 5537 Initial Redesign SOURCE: SCIENT

8. 4/6/01EMTM 5538 The Unforseen Redesign: SOURCE: SCIENT

9. 4/6/01EMTM 5539 How to get rid of the last SPOF: SOURCE: SCIENT

10. 4/6/01EMTM 55310 Scaling Servers: Two Approaches Multiple smaller servers –Add more servers to scale –Most commonly done with web servers Fewer larger servers to add more internal resources –Add more processors, memory, and disk space –Most commonly done with database servers SOURCE: SCIENT

11. 4/6/01EMTM 55311 Where to Apply Scalability To the network To individual servers Make sure the network has capacity before scaling by adding servers SOURCE: SCIENT

12. 4/6/01EMTM 55312 Performance, Redundancy, and Scalability People scale because they want better performance But a fast site that goes down because of one component is a Bad Thing Incorporate all three into your design in the beginning - more difficult to do after the eBusiness is live SOURCE: SCIENT

13. 4/6/01EMTM 55313 Lessons Learned Don’t take the network for granted. Most people don’t want to think about the network - they just want it to work. You will never know everything up front so plan for changes. Warning: as you add redundancy and scalability the design becomes complicated. SOURCE: SCIENT

14. 4/6/01EMTM 55314 Approaches to Scalability Application Service Providers (sites) grow by –scale up: replacing servers with larger servers –scale out: adding extra servers Approaches –Farming –Cloning –RACS –Partitioning –RAPS Load balancing Web caching

15. 4/6/01EMTM 55315 Farming Farm - the collection of all the servers, applications, and data at a particular site. –Farms have many specialized services (i.e., directory, security, http, mail, database, etc.)

16. 4/6/01EMTM 55316 Cloning A service can be cloned on many replica nodes, each having the same software and data. Cloning offers both scalability and availability. –If one is overloaded, a load-balancing system can be used to allocate the work among the duplicates. –If one fails, the other can continue to offer service.

17. 4/6/01EMTM 55317 Two Clone Design Styles Shared Nothing is simpler to implement and scales IO bandwidth as the site grows. Shared Disc design is more economical for large or update-intensive databases.

18. 4/6/01EMTM 55318 RACS RACS (Reliable Array of Cloned Services) –a collection of clones for a particular service –shared-nothing RACS oeach clone duplicates the storage locally oupdates should be applied to all clone’ s storage –shared-disk RACS (cluster) oall the clones share a common storage manager ostorage server should be fault-tolerant osubtle algorithms need to manage updates (cache invalidation, lock managers, etc.)

19. 4/6/01EMTM 55319 Clones and RACS can be used for read-mostly applications with low consistency requirements. –i.e., Web servers, file servers, security servers… the requirements of cloned services: –automatic replication of software and data to new clones –automatic request routing to load balance the work –route around failures –recognize repaired and new nodes

20. 4/6/01EMTM 55320 Partitions and Packs Data Objects (mailboxes, database records, business objects,…) are partitioned among storage and server nodes. For availability, the storage elements may be served by a pack of servers.

21. 4/6/01EMTM 55321 Partition grows a service by –duplicating the hardware and software –dividing the data among the nodes (by object), e.g., mail servers by mailboxes should be transparent to the application –requests to a partitioned service are routed to the partition with the relevant data does not improve availability –the data is stored in only one place - partitions are implemented as a pack of two or more nodes that provide access to the storage

22. 4/6/01EMTM 55322 Taxonomy of Scaleability Designs

23. 4/6/01EMTM 55323 RAPS RAPS (Reliable Array of Partitioned Services) –nodes that support a packed-partitioned service –shared-nothing RAPS, shared-disk RAPS Update-intensive and large database applications are better served by routing requests to servers dedicated to serving a partition of the data (RAPS).

24. 4/6/01EMTM 55324 Performance and Summary What is the right building block for a site? –IBM mainframe (OS390): highly available,.99999 up (less than 5 mins outage per year) –Sun UE1000 –PC servers –Homogenous site is easier to manage (all NT, all FreeBSD, Solaris, OS390) Summary, –For scalability, replicate. –Shared-nothing clones (RACS) –For databases or update-intensive services, packed- partition (RAPS)

25. 4/6/01EMTM 55325 Load Sharing

26. 4/6/01EMTM 55326 Load Sharing What is the problem? –Too much load –Need to replicate servers Why do we need to balance load?

27. 4/6/01EMTM 55327 Load Sharing Strategies Flat architecture –DNS rotation, switch based Hierarchical architecture

28. 4/6/01EMTM 55328 DNS Rotation - Round Robin Cluster

29. 4/6/01EMTM 55329 Flat Architecture - DNS Rotation DNS rotates IP addresses of a Web site –treat all nodes equally Pros: –A simple clustering strategy Cons: –Client-side IP caching: load imbalance, connection to down node Hot-standby machine (failover) –expensive, inefficient Problem –Not sufficient for dynamic content

30. 4/6/01EMTM 55330 Switch-based Cluster

31. 4/6/01EMTM 55331 Flat Architecture - Switch Based Switching products –Cluster servers by one IP –Distribute workload (load balancing) otypically round-robin –Failure detection –Cisco, Foundry Networks, and F5Labs Problem –Not sufficient for dynamic content

32. 4/6/01EMTM 55332 Flat Architectures in General Problems –Not sufficient for dynamic content –Adding/Removing nodes is difficult oManual configuration required –limited load balancing in switch

33. 4/6/01EMTM 55333 Hierarchical Architecture Master/slave architecture Two levels –Level I oMaster: static and dynamic content –Level II oSlave: only dynamic

34. 4/6/01EMTM 55334 Hierarchical Architecture M/S Architecture

35. 4/6/01EMTM 55335 Hierarchical Architecture

36. 4/6/01EMTM 55336 Hierarchical Architecture Benefits –Better failover support oMaster restarts job if a slave fails –Separate dynamic and static content oresource intensive jobs (CGI scripts) runs by slave

37. 4/6/01EMTM 55337 Web Caching

38. 4/6/01EMTM 55338 World Wide Web A large distributed information system Inexpensive and faster accesses to information Rapid growth of WWW (15% per month) Web performance and scalability issues –network congestion –server overloading

39. 4/6/01EMTM 55339 Web Architecture Client (browser), Web server Web server Client (browser)

40. 4/6/01EMTM 55340 Web Proxy Intermediate between clients and Web servers –To implement firewall –To improve performance, caching can be placed Client (browser)Web server

41. 4/6/01EMTM 55341 Web Architecture Client (browser), Proxy, Web server Web server Proxy Client (browser) Firewall

42. 4/6/01EMTM 55342 Web Caching System Caching popular objects is one way to improve Web performance. Web caching at clients, proxies, and servers. Proxy Client (browser) Web server

43. 4/6/01EMTM 55343 Advantages of Web Caching Reduces bandwidth consumption (decrease network traffic) Reduces access latency in the case of cache hit Reduces the workload of the Web server Enhances the robustness of the Web service Usage history collected by Proxy cache can be used to determine the usage patterns and allow the use of different cache replacement and prefetching policies.

44. 4/6/01EMTM 55344 Disadvantages of Web Caching Stale data can be serviced due to the lack of proper updating Latency may increase in the case of a cache miss A single proxy cache is always a bottleneck. A single proxy is a single point of failure Client-side and proxy cache reduces the hits on the original server.

45. 4/6/01EMTM 55345 Web Caching Issues Cache replacement Prefetching Cache coherency Dynamic data caching

46. 4/6/01EMTM 55346 Cache Replacement Characteristics of Web objects –different size, accessing cost, access pattern. Traditional replacement policies do not work well –LRU (Least Recently Used), LFU (Least Frequently Used), FIFO (First In First Out), etc There are new replacement policies for Web objects: –key-based –cost-based

47. 4/6/01EMTM 55347 Two Replacement Schemes Key-based replacement policies: –Size: evicts the largest objects –LRU-MIN: evicts the least recently used object among ones with largest log(size) –Lowest Latency First: evicts the object with the lowest download latency Cost-based replacement policies –Cost function of factors such as last access time, cache entry time, transfer time cost, and so on –Least Normalized Cost Replacement: based on the access frequency, the transfer time cost and the size. –Server-assisted scheme: based on fetching cost, size, next request time, and cache prices during request intervals.

48. 4/6/01EMTM 55348 Prefetching The benefit from caching is limited. –Maximum cache hit rate - no more than 40-50% –to increase hit rate, anticipate future document requests and prefetch the documents in caches documents to prefetch –considered as popular at servers –predicted to be accessed by user soon, based on the access pattern It can reduce client latency at the expense of increasing the network traffic.

49. 4/6/01EMTM 55349 Cache Coherence Cache may provide users with stale documents. HTTP commands for cache coherence –GET: retrieves a document given its URL –Conditional GET: GET combined with the header IF- Modified-Since. –Pragma no-cache: this header indicate that the object be reloaded from the server. –Last-Modified: returned with every GET message and indicate the last modification time of the document. Two possible semantics –Strong cache consistency –Weak cache consistency

50. 4/6/01EMTM 55350 Strong cache consistency Client validation (polling-every-time) –sends an IF-Modified-Since header with each access of the resources –server responses with a Not Modified message if the resource does not change Server invalidation –whenever a resource changes, the server sends invalidation to all clients that potentially cached the resource. –Server should keep track of clients to use. –Server may send invalidation to clients who are no longer caching the resource.

51. 4/6/01EMTM 55351 Weak Cache Consistency –Adaptive TTL (time-to-live) oadjust a TTL based on a lifetime (age) - if a file has not been modified for a long time, it tends to stay unchanged. oThis approach can be shown to keep the probability of stale documents within reasonable bounds ( < 5%). oMost proxy servers use this mechanism. oNo strong guarantee as to document staleness –Piggyback Invalidation oPiggyback Cache Validation (PCV) - whenever a client communicates with a server, it piggybacks a list of cached, but potentially stale, resources from that server for validation. oPiggyback Server Invalidation (PSI) - a server piggybacks on a reply to a client, the list of resources that have changed since the last access by the client. oIf access intervals are small, then the PSI is good. But, if the gaps are long, then the PCV is good.

52. 4/6/01EMTM 55352 Dynamic Data Caching Non-cacheable data –authenticated data, server dynamically generated data, etc. –how to make more data cacheable –how to reduce the latency to access non-cacheable data Active Cache –allows servers to supply cache applets to be attached with documents. –the cache applets are invoked upon cache hits to finish necessary processing without contacting the server. –bandwidth savings at the expense of CPU costs –due to significant CPU overhead, user access latencies are much larger than without caching dynamic objects.

53. 4/6/01EMTM 55353 Dynamic Data Caching Web server accelerator –resides in front of one or more Web servers –provides an API which allows applications to explicitly add, delete, and update cached data. –The API allows static/dynamic data to be cached. –An example - the official Web site for the 1998 Olympic Winter Games owhenever new content became available, updated Web reflecting these changes were made available within seconds. oData Update Propagation (DUP, IBM Watson) is used for improving performance.

54. 4/6/01EMTM 55354 Dynamic Data Caching Data Update Propagation (DUP) –maintains data dependence information between cached objects and the underlying data which affect their values –upon any change to underlying data, determines which cached objects are affected by the change. –Such affected cached objects are then either invalidated or updated. –With DUP, about 100% cache hit rate at the 1998 Olympic Winter Games official Web site. –Without DUP, 80% cache hit rate at the 1996 Olympic Games official Web site.

55. 4/6/01EMTM 55355 Web Caching and Replication Products Design goals –Performance oMinimize document retrieval latency –Scalability oReduce the amount of data transferred oBalance bandwidth use –Manageability: easy to deploy and support oMaintain backward compatibility with existing client software –Availability oMaximize document availability (fault tolerant) –Flexibility: run on multiple hardware and OS platforms, support multiple protocols, HTTP, FTP, etc. oRetain transparency to user

56. 4/6/01EMTM 55356 Commercial products www.cacheflow.com, CA-600, CA-6000, SA- 700,SA-7000, hardware and softwarewww.cacheflow.com www.netapp.com, NetCache, files system softwarewww.netapp.com www.cisco.com, Cache Engine, hardwarewww.cisco.com www.akamai.com, FreeFlow server, softwarewww.akamai.com www.inktomi.com, Traffic server, softwarewww.inktomi.com

57. 4/6/01EMTM 55357 Q&AQ&A