Capacity Planning for LAMP Architectures John Allspaw Manager, Operations Flickr.com Web Builder 2.0 Las Vegas

Capacity Planning for LAMP Architectures Capacity planning: the ability to make snap decisions to spend millions of dollars with not enough information - Kevin Murphy

Capacity Planning for LAMP Architectures It is NOT: Performance tuning Tips and tricks to be scalable

Capacity Planning for LAMP Architectures It IS: What comes after youve made it all scalable Making sure that you have enough equipment to handle gradual and bursty traffic

Capacity Planning for LAMP Architectures Questions to answer

Capacity Planning for LAMP Architectures How many of each server class should you add as you grow ? Hint: Dont add too much (too much $$! Ahh!) Hint: Dont add too little (too much traffic! Ahh!)

Capacity Planning for LAMP Architectures How to make it easier to predict the future* ? How to make it easier to justify those predictions ? How to make it easier to predict the future…in the future ?! *You cant predict the future, but you can try.

Capacity Planning for LAMP Architectures The OLD way of doing things was easier A.k.a. web1.0 Small number of content producers Control over the content Capacity was dictated by the demand for that static/small content Even bbs/communities/ecommerce had relatively predictable growth

Capacity Planning for LAMP Architectures Todays way of doing things is harder/fun No control over content (users have control) No control over usage (users have control)

Capacity Planning for LAMP Architectures Todays way of doing things is harder/fun Network effect, nonlinear growth (more users/content/contacts/activity mean >> usage) Event-related growth (press, news event, social trend, etc. can affect usage and content) Example: London bombing, tsunami, holidays, etc.

Capacity Planning for LAMP Architectures Considerations for social applications User behavior should guide you with defining capacity metrics (not just server stats) Usage can accelerate, not just grow

Capacity Planning for LAMP Architectures How we do it at Flickr

Capacity Planning for LAMP Architectures Gathering Usage Application-level information (users, photos, activity, etc.) Server-level information (cpu, disk I/O, memory, etc.) We tie the two together

Capacity Planning for LAMP Architectures BEFORE we start collecting server stats What resources are peak-driven ? (concurrent use) –Ex: photo processed/sec, pages/sec, images/sec, db qps What resources are permanently consumable ? –Ex: database space, storage (GB/day) etc.

Capacity Planning for LAMP Architectures BEFORE we start collecting server stats What is an average user:consumption ratio ? (example: user: photo) What is the high and low of ratios ? Is the average ratio changing over time ?

Capacity Planning for LAMP Architectures Non-linear growth

Capacity Planning for LAMP Architectures Non-linear growth

Capacity Planning for LAMP Architectures Linear relationships, though

Capacity Planning for LAMP Architectures Server and Network statistics Ganglia - (we love ganglia!) –Multicast-y goodness –SUPER simple to make a graph from any stat –Clustering Other custom rrdtool-based stuff MRTG

Capacity Planning for LAMP Architectures Photos uploaded/processed/min Avg processing time per minute Avg CPU per minute

Capacity Planning for LAMP Architectures Gather and record statistics Accept the observer effect (its worth it) Aggregate your stats across clusters –Stacked graphs –Totals and averages

Capacity Planning for LAMP Architectures Squid client requests (24 hours) (Y axis is req/sec)

Capacity Planning for LAMP Architectures Squid LRU reference age Over 24 hours -Y axis is days -So peak has 3.6hours

Capacity Planning for LAMP Architectures Find the ceiling of each class/function/server Maximum allowable somethings –MySQL : queries/sec before slave lag sets in –Apache/php : page requests/sec before total meltdown –Squid/memcached : cache churn rate, request rate –Storage : disk I/O utilization, storage limit(!) –Etc.

Capacity Planning for LAMP Architectures Forget benchmarking, use real load –Make sure you have a easy mechanism to take servers in and out of production –Pull machines from a balanced pool during medium-level traffic (very carefully) –Watch and record

Capacity Planning for LAMP Architectures Build the infrastructure to make it EASY to measure Obvious things to help this: Load balancing Network segmentation Carve up functions into clusters –Dont let a machine do more than one primary thing (if you can help it) this isnt for performance! If it makes it faster/better, then bonus!

Capacity Planning for LAMP Architectures For graphs you dont have raw data for GraphClick - graph digitizer package - $8 US - you pick points on a calibrated image, it spits out tabular data

Capacity Planning for LAMP Architectures

Once you have: 1. Time history of metrics 2. Ideal peak levels (ceiling) Then you can: 3.Predict the future!*

Capacity Planning for LAMP Architectures Example: Photo Processing

Capacity Planning for LAMP Architectures Photos uploaded/processed/min Avg processing time per minute Avg CPU per minute

Capacity Planning for LAMP Architectures

Dirty linear math 25% photos/min 40% photos/min So….take a ceiling: 75% photos/min = 6720 photos/hour (but double-check the process time)

Capacity Planning for LAMP Architectures Conclusion

Capacity Planning for LAMP Architectures Know your machines and their limits Measure how the site is being used with application- level stats Tie real-world observations to server stats

Capacity Planning for LAMP Architectures Some Flickr statistics 300M photos, 4 or 5 different sizes Keep ~25M images in cache at any time, ~1M from RAM 2B MySQL queries/day 21k req/sec to memcached 1.2 PT raw disk storage