1. Web Performance Optimization: scaling the witchcrafts Xiaoliang “David” Wei www.facebook.com/DavidWei

2. 1 Web sites are distributed systems 2 Web performance 3 Adaptive performance optimization 4 Lessons learnt Agenda

3. A web page request can involve many parties: Client browser Web server (and backend caches, DBs) Content distributor A web site is a distributed system

4. ▪ From a user clicking event to the presentation of the page at the browser: Life of a web page request Web Servers Browsers

5. ▪ From a user clicking event to the presentation of the page at the browser: ▪ time to send to and receive from web server through Internet Web Servers Browsers ▪ ½ NetTime Life of a web page request

6. ▪ From a user clicking event to the presentation of the page at the browser: ▪ time to send to and receive from web server through Internet ▪ time to generate HTML at web server Web Servers Browsers ▪ GenTime ▪ ½ NetTime Life of a page request

7. ▪ From a user clicking event to the presentation of the page at the browser: ▪ time to send to and receive from web server through Internet ▪ time to generate HTML at web server Web Servers Browsers ▪ GenTime ▪ NetTime Life of a page request

8. ▪ From a user clicking event to the presentation of the page at the browser: ▪ time to send to and receive from web server through Internet ▪ time to generate HTML at web server ▪ time to download multimedia contents and to render the page at browser Web Servers Content Distribution Network (CDN) Browsers ▪ GenTime ▪ NetTime Render Time Life of a page request

9. ▪ Questions: ▪ Scalability ▪ Consistency ▪ Performance Web Servers Content Distribution Network (CDN) Browsers ▪ GenTime ▪ NetTime Render Time Web service: a distributed system

10. Web Performance Optimization

11. Google: +400ms => -0.6% daily search per user Bing: +500ms => -1% click, +1.2 sec time to click (2x) Facebook: user session time stays the same, faster => more page views Performance impacts business

12. Status of web performance optimization Decade-old protocols: TCP, HTTP, HTML Fast-changing industry: browser war Optimization is mostly witchcraft Some rules emerging (e.g. Steve Souder’s books) Large scale optimization is difficult

13. Web performance: TCP / HTTP / HTML TCP overhead: 3-way handshake, slow start => persistent conn TCP performance: depends on loss rate and distance => multiple conn HTTP persistent connection: maximum 2 per server HTTP Pipeline: failed, sequential loading HTML: very limited description for priority/concurrency => Big resources, multiple connections are preferred

14. Web performance: Browsers most browsers blocked on JavaScript download / execution; some blocked on Stylesheet Stylesheet application might be slow JS execution might be slow Maximum connections per browser => The less the better

15. Web performance: “best practice rules” Minimize HTTP requests Gzip resources Preload Components Reduce DNS Lookup Split Requests Across Domain

16. Web performance: trade-offs Minimize HTTP requests  Increased unused bytes Gzip resources  Increased CPU utilization Preload Components  Increased bandwidth cost Reduce DNS Lookup  Split Requests Across Domain Split Requests Across Domain  Reduce DNS Lookup

17. Web performance optimization Improve speed Reduce cost (both latency penalty and $ cost) Operate on the equilibrium Easy for a single page, hard for large scale systems

18. Large scale Viral growth Agile development Performance optimization: challenges

19. Thousands of servers in multiple locations Millions of users per minute Billions of page views served per day Hundreds of thousands of developers Thousands of web resource components Challenges: Large scale

20. Concept of “Platform”: A single web page can have many features from many developers A feature could be adopted by millions of users in a day Features come and go Challenges: Viral growth

21. Fast release, allow mistake, fix quickly Frequent code base changes Continuous release Challenges: Agile development

22. Web performance: the gap “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Large Scale Operation Large scale Viral growth Agile development

23. Web performance: fill the gap “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Interface, Model, and Infrastructure Large Scale Operation Large scale Viral growth Agile development

24. Adaptive Performance Optimization

25. Adaptive Performance optimization Example: “Best Practices for Speeding Up You Web Site” – YUI Rule 1: Minimize HTTP Request Combined files: package Javscript and Stylesheets Sprite Images …

26. Performance optimization: Case Study Day 1: Some smart engineers start a project! … “Let’s write a new page with features A, B and C!”

27. Day 2: Some smart engineers read YUI blog and says… … “A & B & C are always used; let’s package them together!”

28. Day 2: Awesome! …

29. Day 3: feature C evolves… If (users_signup_for_C()) { } …

30. Day 3: If (users_signup_for_C()) { } … A&B are always used, while C is not...

31. Day 4: feature C is deprecated // no one uses C { } …

32. Day 4: we start to send unused bits // no one uses C { } … It is hard to remember we should remove C here.

33. One months later… if (F is used) if (F is not used) { } … Thousands of dead CSS bytes in the package.

34. One months later… if (F is used) if (F is not used) { } … Performance Optimization: the gap “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Large Scale Operation

35. Adaptive Static Resource Optimization Fill the gap: Build interface to separate requirement declaration and delivery of static resources Model the trade-off of optimization Infrastructure to analyze requirement logs and globally optimize the delivery performance “Reduce # of HTTP Requests” Package JS/CSS; Sprite Image Large Scale Operation Interface, Model, and Infrastructure

36. Adaptive Packager: Designs

37. Interfaces Back to Day 1: require_static(A_css); require_static(B_css); require_static(C_css); Separate Declaration from actual Delivery Global Optimization on Delivery Requirement Declaration lives with HTML

38. Offline analysis Packager: Global JS/CSS Optimization Online process require_static(A_css); require_static(B_css); require_static(C_css); Usage Pattern logs Packaging algorithm “Optimal” packages

39. Packager: Usage Pattern Logs Usage Pattern logs Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 11 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111

40. Packager: Cost/Benefit Model ▪ To package two files A & B: ▪ “Cost”: for page requests that only uses A, we waste the bytes of B, vice versa ▪ “Benefit”: for page requests that uses both A and B: we save one round trip ▪ Bytes / Bandwidth ~ Latency ▪ “Profit” to be maximized: Benefit – Cost Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 11 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111

41. Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 11 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111 Packager: Cost/Benefit Model ▪ Assume: latency = 40ms, and bandwidth = 1 Mbps ▪ A+B: 40ms * 11.131M No cost, pure gain. Definitely package

42. Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111 Packager: Cost/Benefit Model ▪ Assume: latency = 40ms, and bandwidth = 1 Mbps ▪ B+C: 40ms * 11.1M – 300B / 1Mbps * 0.031M Benefit larger than cost OK to package

43. Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111 Packager: Cost/Benefit Model ▪ Assume: latency = 40ms, and bandwidth = 1 Mbps ▪ B+D: 40ms * 1K – 2K / 1Mbps * 11.13M Cost larger than benefit Don’t package

44. Packager: Optimal packages Usage Pattern logs Packaging algorithm “Optimal” packages Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 111 F.css (400B) 11 G.css (50B) 11111 ▪ Pkg 1: A, B, C ▪ Pkg 2: E, F, G

45. Packager: Cost/Benefit Model Page Load12345…10000 0 Page Count 10 M 1M1M 100 K 20 K 10 K 1K A.css (1 KB) 111111 B.css (1 KB) 111111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 111 F.css (400B) 11 G.css (600B) 11111 The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix). P[i,j] = 1 if resource j is in package i.

46. Packager: Cost/Benefit Model The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix) Byte size of each package Packages for each page load

47. Packager: Cost/Benefit Model The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix) Byte Cost RT Gain

48. Packager: Cost/Benefit Model The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix)

49. Packager: Algorithms Packaging algorithm ▪ Greedy algorithm ▪ Simulated Annealing ▪ Clustering algorithms ▪ …

50. Adaptive Static Resource Optimization Adaptive Packaging / Spriting Cross-feature optimizations (e.g. search + ads) Adaptive to change of user behaviors and code developments Same methodology works for image spriting

51. Adaptive Packager: Operations

52. Update frequency Too slow: fail to adapt to usage changes Too fast: invalidate caches Usually weekly Can be smarter: calculate theoretic gain daily

53. Resources for new features No usage logs available Developers to estimate the initial usage pattern Update packages one day after release, if necessary

54. Deployment status since Nov 2008 Rebalance weekly based on previous week’s usage pattern (>100K unique usage patterns) Javascript and CSS packaging only (image soon)

55. Scalable with development JS/CSS Code base: Nov 2008 -> Sep 2009 Date# of JS files# of JS bytes # of pkg at a home.php # of bytes at a home.php Nov 20084614.4 MB29629 KB Sep 200910456.8 MB19487 KB Date# CSS files # of CSS bytes # of pkg at a home.php # of bytes at a home.php Nov 20084871.7 MB2469 KB Sep 20099381.8 MB1448 KB

56. Adaptive to new usages # of JS/CSS Packages served in the past month

57. Experiment: Adaptive Image Spriting The puzzle of image spriting: Thousands of virtual gifts with static images, which to sprite?

58. Experiment: Adaptive Image Spriting The puzzle of image spriting: The answer is…

59. Lessons learnt

60. Regression detection can be trickier “The page has 5KB more Javascripts!” Regression introduced by Packager? Or, by the code?

61. When data go wrong

62. “All models are wrong. Some models are useful.” Cache is not considered in the model Browse session is not considered Not all users have 1Mbps and 40ms RTT connections A/B Testing to find an optimal model Customized packaging for user groups Per browser groups: ~5% improvements Locale / IP Prefix might yield better performance

63. Packager: Cost/Benefit Model with Cache The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix) Probability that resource n is not cached in user’s browser (cache miss for a file) Probability that package i is not cached in user’s browser (cache miss for a package)

64. Packager: Cost/Benefit Model with Cache The usage matrix (0/1) on the left for m page loads and n resources The weight (count) of each page load The byte size of each resource The package definition (0/1 matrix) Cache-miss rate for file Byte Cost RT Gain Cache-miss rate for package

65. Human errors are possible Automatic analysis is preferable: require_static(A_css); //forgot to remove the require_static require_static(B_css); require_static(C_css);

66. Other application: Adaptive resource preloading User navigation path: Index -> home -> profile Predict the user’s next step and load the resources, or look at the resource usage history in the whole session log Cost: loading extra resources for misprediction (bandwidth $ cost) Gain: saving the loading time of a resource (roundtrip + bandwidth gain)

67. Adaptive Static Resource Optimization Adaptive progressive rendering flush the commonly used JS/CSS/Images to the browsers, before the page is generated at the server Browser downloads JS/CSS in parallel with page generation at server side

68. Adaptive Progressive Rendering Adaptive progressive rendering Progressive rendering: flush the commonly used JS/CSS/Images to the browsers, before the page is generated at the server

69. Adaptive progressive rendering What are the common JS / CSS should we flush? Usage Pattern logs Page Loadhomeprofilehom e profilephot o … Page Count 10 M1 M100 K20 K10 K A.css (1 KB) 11111 B.css (1 KB) 11111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 11 F.css (400B) 11 G.css (600B) 111

70. Adaptive progressive rendering Sum on the page name: Page Loadhomeprofilehom e profilephot o … Page Count 10 M1 M100 K20 K10 K A.css (1 KB) 11111 B.css (1 KB) 11111 C.css (300B) 111 D.css (2 KB) 1 E.css (700B) 11 F.css (400B) 11 G.css (600B) 111 Page Loadhomeprofilephot o … Page Count 10.1 M 1.02 M10 K A.css (1 KB) 10.1M1.02M10K B.css (1 KB) 10.1M1.02M10K C.css (300B) 10.1M1M D.css (2 KB) 10M E.css (700B) 0.02M10K F.css (400B) 0.02M10K G.css (600B) 10M0.02M10K

71. Adaptive progressive rendering Remove the rare usages Page Loadhomeprofilephot o … Page Count 10.1 M 1.02 M10 K A.css (1 KB) 10.1M1.02M10K B.css (1 KB) 10.1M1.02M10K C.css (300B) 10.1M1M D.css (2 KB) 10M E.css (700B) 0.02M10K F.css (400B) 0.02M10K G.css (600B) 10M0.02M10K Page Loadhomeprofilephot o … A.css (1 KB) YYY B.css (1 KB) YYY C.css (300B) YY D.css (2 KB) Y E.css (700B) Y F.css (400B) Y G.css (600B) YY

72. Adaptive Static Resource Optimization Adaptive progressive rendering More than 500 msec gain for home page end-to-end time Adaptive to the change of popular resources Need to solve issues with redirections, HTTP-only cookie and buffers

73. Summary ▪ Performance optimization must be adaptive ▪ Good interface helps optimization ▪ Trace-based analysis is powerful ▪ Always monitor the optimization effect

74. Web performance optimization Decade-old protocols: performance under new usage model Fast-changing industry: client side caching, asynchronous loading Modeling and scaling the best-practices Measurement: business impact of performance Technical basis (e.g. cache hit rate)

75. Thank you!