1. DotSlash: Handling Web Hotspots at Dynamic Content Web Sites Weibin Zhao Henning Schulzrinne {zwb,hgs}@cs.columbia.edu Department of Computer Science Columbia University Global Internet 2005 March 19, 2005

2. DotSlash2 Web Hotspots A well-identified problem Flash crowds, the Slashdot effect 15 minutes of fame Examples Slashdotting, featured Google search, special events, breaking news, … Web Server Internet

3. March 19, 2005 DotSlash3 The Challenge Short-term dramatic surge of request rate Large & quick increase Last for a short period Existing mechanisms are not sufficient Capacity planning, CDNs Good for long term, not cost-effective for hotspots Caching Not fully controlled by origin server Service degradation, admission control Last resort, not user friendly

4. March 19, 2005 DotSlash4 Dynamic Content Web Sites More vulnerable to hotspots CPU-bound, request rate supported is low Hard to cache dynamic content A much harder problem Different bottlenecks Database server: on-line bookstore (Amazon) Web server: auction (eBay), bulletin board (Slashdot) Caching & consistency control

5. March 19, 2005 DotSlash5 Our Approach DotSlash  counteract the Slashdot effect Rescue system Triggered automatically when load spikes Mutual-aid model: for different web sites Cost effective: for rare events Automated rescue process Self-configuring: build an adaptive distributed web server system on the fly Techniques: service discovery, dynamic virtual hosting, adaptive overload control, dynamic script replication

6. March 19, 2005 DotSlash6 Rescue Relationship  Can provide rescue to multiple servers: S3  Can get rescue from multiple servers: S1  Cannot provide/get rescue simultaneously Origin Server: S1, S2 Rescue Server: S3, S4, S5, S6 S1 S5 S4 S3 S6 S2 S8 S7 rescuing

7. March 19, 2005 DotSlash7 Service Discovery DotSlash directory services Enable web servers from different sites to learn about each other: register/query Built upon mSLP (Mesh-enhanced Service Location Protocol): replicated Directory Agents (DAs) Discover mSLP DAs dot-slash.net DNS domain DNS SRV for dot-slash.net query_name=_slpda._tcp.dot-slash.net, query_type=srv

8. March 19, 2005 DotSlash8 Workload Monitoring Bottlenecks & Metrics Network (static content): outbound HTTP traffic CPU (dynamic content): /proc/stat Moving average filter Load regions Desired Configurable: [40%, 60%] Trigger rescue actions Desired load region Heavily loaded region Lightly loaded region

9. March 19, 2005 DotSlash9 DotSlash Rescue Protocol Application level request response Requests: SOS, RATE, SHUTDOWN SOS: initiate a rescue origin  rescue RATE: adjust allowed redirect (data) rate rescue  origin SHUTDOWN: end a rescue origin  rescue

10. March 19, 2005 DotSlash10 Rescue Control Normal Rescue SOS Get rescue Release rescue Provide rescueShutdown last rescue Increase Pr Decrease Pr Request more rescue Increase Rr Decrease Rr Provide more rescue Shutdown some rescue

11. March 19, 2005 DotSlash11 Request Redirection Origin server Offload client requests to rescue servers Two-level redirection DNS-RR Add/remove rescue server IP addresses via dynamic DNS update HTTP redirect Use rescue server aliases Don’t redirect requests from rescue servers Redirect policies WRR based on rescue server capacity

12. March 19, 2005 DotSlash12 Dynamic Virtual Hosting Rescue server Serve new content (origin server) on the fly Alias Generate dynamically, and register via dynamic DNS update Mapping: request  itself / origin server Based on the Host header in the request Three cases Its configured name: www.rescue.com  itselfwww.rescue.com An alias: www-vh1.rescue.com (HTTP redirect)  originwww-vh1.rescue.com An origin server name: www.origin.com (DNS-RR)  originwww.origin.com Handle expired mapping

13. March 19, 2005 DotSlash13 DotSlash for Dynamic Content Remove the web server bottleneck Dynamic Script Replication LAMP configuration origin serverdatabase rescue server MySQLApache (5) PHP(6) (1) (2) (3) (4) Client (7) (8)

14. March 19, 2005 DotSlash14 Dynamic Script Replication Rescue server Map a redirected URI to a script file Trigger 404 handler if the script file not found Retrieve the script file Handle file inclusions Set query variables Run the script by invoking native include Origin server If a request is from a rescue server and for dynamic content, return the script file

15. March 19, 2005 DotSlash15 Handle File Inclusions The problem A replicated script may include files that are located at the origin server Assume: included files under DocumentRoot Approaches Renaming inclusion statements Need to parse scripts Customized error handler Catch inclusion errors

16. March 19, 2005 DotSlash16 Implementation Apache module, PHP extension Dynamic DNS: dot-slash.net Service discovery: enhanced SLP BINDmSLP HTTP SHM SLP DNS DSRP Other Dotsd Client Apache Dotsd Mod_dots

17. March 19, 2005 DotSlash17 Evaluation Experimental Setup Linux machines: Redhat 9.0 HC: 2 GHz CPU, 1 GB memory LC: 1 GHz CPU, 512 MB memory Apache: 2.0.48, DotSlash module, PHP: 4.3.6, DotSlash extension MySQL: 4.0.18 Benchmark RUBBoS (Rice U.) bulletin board 19 scripts: 1 KB to 7 KB 439 MB database

18. March 19, 2005 DotSlash18 Increasing Max Request Rate: R No rescue: R=118 With rescue: R=245 #rescue servers: 9 Origin (HC)DB (HC) Rescue (LC) Configuration: Rescue (LC) 245/118>2 CPU: Origin=100% DB=45% CPU: Origin=55% DB=100%

19. March 19, 2005 DotSlash19 Effectiveness No rescue: R=49 With rescue: R=245 245/49=5 origin serverHCLC improvement25 #rescue servers910 Conclusion: remove web server bottleneck Origin (LC) DB (HC) Rescue (LC) Another Configuration: Comparison: #rescue server: 10 Rescue (LC)

20. March 19, 2005 DotSlash20 CPU Utilization Control

21. March 19, 2005 DotSlash21 Workload Migration

22. March 19, 2005 DotSlash22 Conclusions DotSlash framework, prototype, evaluation Fully automated rescue system, transparent to clients Scalable Get 10-fold improvement (static content) Remove web server bottleneck (dynamic content) Future work Remove database server bottleneck For further information http://www.cs.columbia.edu/~zwb/project/dotslash WCW’04