1. Jeremy Bingham, DailyKos.com

2. Initial Problems Site tipped over way too often Required intensive administration Load times needed to be faster

3. MySQL Issues Table design didn't scale Redesigned tables with more sensible primary keys Lots of MySQL 3.23isms everywhere Re-engineered tables & code to take advantage of new features Had to stop assuming stuff was there for “a good reason”

4. Caching and proxying Started off with apache 1.3 as proxy Cached front page & recent posts to disk every 5 minutes for anonymous readers Caused problems during the Iowa caucuses when all servers updated their file caches simultaneously. Memcached Using since 2006 Used even more in 2008

5. Lighttpd as proxy Spent weekend with K&R Added memcached support to mod_magnet Hacked up mod_magnet had issues with flexibility, also did way more than needed Enter mod_mcpage Only caches pages, nothing else Gzips large pages Works entirely in lighttpd Greatly lowered stress on servers

6. Hardware, before Before: 10 dual Xeons w/ 2GB RAM, 2 dual Operons w/ 8GB RAM & 2 73 GB SCSI disks, image server, memcached server, combined search & SMTP server Each webhead independent; all updates had to be applied to each server. Inconvenient, nerve-wracking possibilities for error

7. Hardware, after After: 6 quad Xeons w/ 8GB RAM, SATA scratch disks, 2 octocore Xeons w/ 16 GB RAM, 2 73 GB SCSI for / and /tmp, 6x73GB RAID-10 w/ finely tuned XFS for DB files, consolidated image origin/search/SMTP, Xen VM for db backup, CDN for images Webheads netboot with NFS root, run as cluster Lower costs, better performance New webheads pop right in

9. DK Monthly Traffic, by Year

10. How It Worked Out Performance good overall Unexpected issues w/ the Flash electoral map - 100 mbit switch no one noticed Webhead load generally between 0.5 and 1.8 Some slowdowns from ads and the map