1. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Web Builder 2.0 Las Vegas
John Allspaw
Manager, Operations
Flickr.com
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

2. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Capacity planning:
“the ability to make snap decisions to spend millions of dollars with not enough information”
- Kevin Murphy
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

3. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
It is NOT:
Performance tuning
Tips and tricks to be “scalable”
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

4. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
It IS:
What comes after you’ve made it all “scalable”
Making sure that you have enough equipment to handle gradual and bursty traffic
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

5. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Questions to answer
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

6. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
How many of each server “class” should you add as you grow ?
Hint: Don’t add too much (too much $$! Ahh!)
Hint: Don’t add too little (too much traffic! Ahh!)
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

7. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
How to make it easier to predict the future* ?
How to make it easier to justify those predictions ?
How to make it easier to predict the future…in the future ?!
*You can’t predict the future, but you can try.
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

8. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
The OLD way of doing things was easier
A.k.a. “web1.0”
Small number of content producers
Control over the content
Capacity was dictated by the demand for that static/small content
Even bbs/communities/ecommerce had relatively predictable growth
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

9. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Today’s way of doing things is harder/fun
No control over content (users have control)
No control over usage (users have control)
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

10. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Today’s way of doing things is harder/fun
Network effect, nonlinear growth (more users/content/contacts/activity mean >> usage)
Event-related growth (press, news event, social trend, etc. can affect usage and content)
Example: London bombing, tsunami, holidays, etc.
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

11. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Considerations for “social” applications
User behavior should guide you with defining capacity metrics (not just server stats)
Usage can accelerate, not just grow
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

12. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
How we do it at Flickr
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

13. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Gathering Usage
Application-level information (users, photos, activity, etc.)
Server-level information (cpu, disk I/O, memory, etc.)
We tie the two together
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

14. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
BEFORE we start collecting server stats
What resources are peak-driven ? (concurrent use)
Ex: photo processed/sec, pages/sec, images/sec, db qps
What resources are permanently consumable ?
Ex: database space, storage (GB/day) etc.
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

15. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
BEFORE we start collecting server stats
What is an average user:consumption ratio ? (example: user: photo)
What is the high and low of ratios ?
Is the average ratio changing over time ?
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

16. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Non-linear growth
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

17. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Non-linear growth
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

18. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Linear relationships, though
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

19. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Server and Network statistics
Ganglia - (we love ganglia!)
Multicast-y goodness
SUPER simple to make a graph from any stat
Clustering
Other custom rrdtool-based stuff
MRTG
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

20. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Photos uploaded/processed/min
Avg processing time per minute
Avg CPU per minute
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

21. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Gather and record statistics
Accept the ‘observer effect’ (it’s worth it)
Aggregate your stats across clusters
Stacked graphs
Totals and averages
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

22. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Squid client requests
(24 hours)
(Y axis is req/sec)
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

23. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Squid
LRU reference age
Over 24 hours
Y axis is “days”
So peak has 3.6hours
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

24. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Find the ceiling of each class/function/server
Maximum allowable somethings
MySQL : queries/sec before slave lag sets in
Apache/php : page requests/sec before total meltdown
Squid/memcached : cache churn rate, request rate
Storage : disk I/O utilization, storage limit(!)
Etc.
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

25. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Forget benchmarking, use real load
Make sure you have a easy mechanism to take servers in and out of production
Pull machines from a balanced pool during medium-level traffic (very carefully)
Watch and record
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

26. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Build the infrastructure to make it EASY to measure
Obvious things to help this:
Load balancing
Network segmentation
Carve up functions into clusters
Don’t let a machine do more than one primary thing (if you can help it)
this isn’t for performance! If it makes it faster/better, then bonus!
User:ratios - sometimes users have no photos, only lurk
Sometimes users have LOTs of photos, don’t comment/favorite, etc.
Ratio rate change: adjust ceiling on user-driven resources
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

27. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
For graphs you don’t have raw data for
GraphClick
- graph digitizer package
- $8 US
- you pick points on a calibrated image, it spits out tabular data
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

28. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

29. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Once you have:
Time history of metrics
“Ideal” peak levels (ceiling)
Then you can:
3. Predict the future!*
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

30. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Example:
Photo Processing
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

31. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Photos uploaded/processed/min
Avg processing time per minute
Avg CPU per minute
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

32. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

33. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Dirty linear math
25% photos/min
40% photos/min
So….take a “ceiling”:
75% photos/min = 6720 photos/hour
(but double-check the process time)
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

34. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Conclusion
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

35. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Know your machines and their limits
Measure how the site is being used with application-level stats
Tie real-world observations to server stats
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

36. Capacity Planning for LAMP Architectures
MGB 2003
Capacity Planning for LAMP Architectures
Some Flickr statistics
300M photos, 4 or 5 different sizes
Keep ~25M images in cache at any time, ~1M from RAM
2B MySQL queries/day
21k req/sec to memcached
1.2 PT raw disk storage
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

37. MGB 2003 © 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.