1. An Analysis of Facebook photo Caching
Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd,
Sanjeev Kumar,Harry C. Li
Cornell Univ, Princeton Univ, Facebook Inc
SOSP 2013

2. Content
Introduction Facebook‘s Photo-Serving stack Method Potential Improvement Conclusion Critics

3. Introduction Upload the 250 billions of Photo and accessed client
How much of the access traffic in server?
How request travel all photo-serving stack?
How different cache size and eviction algorithm?

4. Facebook‘s Photo-Serving stack
Local Photo-Serving stack
Client
Client
Browser
Cache
Local
Fetch

5. Facebook‘s Photo-Serving stack
Geo-distributed Edge Cache (FIFO)
Browser
Cache
Client
PoP
Edge
Cache
(Millions)
(Nines)

6. Facebook‘s Photo-Serving stack
Single Global Origin Cache (FIFO)
Client
PoP
Data Center
Browser
Cache
Edge
Cache
Origin
Cache
(Millions)
(Nines)
(Four)
Hash(url)

7. Facebook‘s Photo-Serving stack
Haystack
Client
PoP
Data Center
Browser
Cache
Edge
Cache
Origin
Cache
Backend (Haystack)
(Millions)
(Nines)
(Four)

8. Facebook‘s Photo-Serving stack
Organization of Photo-Serving stack
Browser cache
First cache layer and co-located with the client
Using in-memory hash table
Using LRU(Least recently used) eviction
Edge cache
Second cache layer and spreaded across US
Using in-memory hash table and hold metadata
Using FIFO(First In First Out) replacement policy
Main Goal : traffic sheltering and bandwidth reduction
Traffic sheltering : prioritization drive a number of decisions

9. Facebook‘s Photo-Serving stack
Organization of Photo-Serving stack
Origin cache
Thrid cache layer and co-located with backend
Using hash mapping based on the unique id of the photo
Using FIFO replacement policy
Haystack
Lowest level of the photo serving stack
Using a compact blob(binary large object) and stored images

10. Facebook‘s Photo-Serving stack
Photo transformation
Can serve many flatform
such as mobile, desktop, laptop
Resizer in backend and caching layer
Resized and cropped photos saved Backend haystack machines
Single cache has many size of same photo
If Akamai CDN(Contents Delivery Network)reqest, dosen’t store storage

11. Instrumentation Scope
Methodology
Trace Collection
Objective
Collecting a representative sample that could permits correlation of events related to the same request
Client
PoP
Data Center
Instrumentation Scope
Browser
Cache
Edge
Cache
Origin
Cache
Backend (Haystack)

12. Methodology Methodology Trivial problem Client Edge cache/Origin cache
Changed code in facebook using javascript
If user select photo in facebook browser load specific photo in sampling
Periodically javascript uploads which is record
Aggeregated result from multiple clients by webserver and report
Edge cache/Origin cache
Report all situation in cache
Trivial problem
This approach would be disruptive to the existing Facebook base code
Give unique request ID
This information travel along the stack

13. Methodology Sampling strategies Request-based Objected-based
Sampling requests randomly
Bias on popular content
Objected-based
Focused on some subset of photo by deterministic test on photoId
Fair coverage of unpopular photos
Cross stack analysis

14. Workload Analysis Objectives Traffic sheltering effects of caches
Photo popularity distribution
Geographic traffic distribution
Idea which can make the cache better
Impact of sizes & algorithm

15. Workload Traffic Sheltering 77.2M 26.6M 65.5% 11.2M 58.0% 7.6M 31.8%
Client
PoP
Data Center
Browser
Cache
Edge
Cache
Origin
Cache
Backend (Haystack) R
77.2M
26.6M
65.5%
11.2M
58.0%
7.6M
31.8%
Traffic Share
65.5%
20.0%
4.6%
9.9%

16. Workload Popularity Distribution Popularity Rank shift
Most popular object(3th-10th) dropped out
More popular object(10th-100th) drop to 1000th-10000th

17. Workload
Hit Ratio
Insight of distribution for distinct photo blobs

18. Geographic Traffic Distribution
Client to Edge cache traffic

19. Cross-Region Traffic at Backend
Misdirected resizing traffic
Didn’t copy adequate number of backup copy
Failed local fetch
Migration

20. Potential Improvement
Browser Cache
What-if question : “what the client browser hit ratios would have been with an infinite cache size?”

21. Potential Improvement
Edge Cache
Test on San Jose Edge Cache
Cache size
algorithms

22. Potential Improvement
Origin Cache
Cache size and algorithm

23. Social-Network Analysis
Content Age Effects
Content Age
Age of content that request photo upload in 24hours
Exclud profile photo

24. Social-Network Analysis
Social Effects
Effect of popularity
Hit rate dependent on followers and many friends

25. Conclusion Conclusion
Instrumented th entire Face book photo-serving stack
Traced representative of Facebook’s full workload
Workload include pattern, distribution, geographic system
Tested the option of cache that change algorithm

26. Critics
Using ‘intuitionally’ too much