1. Scalable Data Management@facebook
Srinivas Narayanan
11/13/09

2. Scale
So what scale are we really talking about?

3. >200 billion monthly page views
Over 300 million active users
> 3.9 trillion feed actions processed per day
>200 billion monthly page views
100 million search queries per day
Over 1 million developers in 180 countries
#2 site on the Internet
(time on site)
Exciting and also humbling to be able to serve 300M users, and so many operations.
If Facebook were a country, we would be the 3rd largest after China and India.
If we look at the amount of data users on Facebook upload daily, it’d exceed 350 feature length movies.
Our users upload an average of 500GB of structured data per day in addition to about 2TB of photos and videos.
As we will see during the talk,this kind of load is highly non-trivial.
The challenge is to build systems that can support this scale and keep the site running 24 hours a day, each and every day.
More than 232 photos…
2 billion pieces of
content per week
6 billion minutes per day

4. Growth Rate Active Users 300M 2009
People log in more than once a month :)
- It took over four years to reach 100 million users, a level we achieved in September of 2008.
And another 1 year to reach 300 million!
- Technology adoption is accelerating...Radio took 38 yrs to reach 50M, TV took 13 yrs, computers took 4 years... Facebook took only 3 years to reach 50M active users
Why is this important? The rate of growth matters because you have very little time to change things. You just re-built something yesterday and yet, it no longer works today! Designing for exponential growth is really hard. Hard to predict what will go wrong or where the next set of bottlenecks will be.

5. Social Networks
So what makes the work at Facebook challenging?

6. The social graph links everything
People are only one dimension of the social graph.
Social applications links people to many types of data..photos, videos, music, blog posts, groups, events, organizations, and even other applications

7. Scaling Social Networks
Much harder than typical websites where...
Typically 1-2% online: easy to cache the data
Partitioning & scaling relatively easy
What do you do when everything is interconnected?
Facebook and social networks in general have a unique problem. The data is so connected that the only reasonable way to store it is essentially over a uniformly distributed set of data providers. It’s difficult to segment our data in any meaningful way to reside on the same disks without duplicating it everywhere. It’s also so frequently accessed that we simply can’t hit the database for each access.

8. name, status, privacy, profile photo
name, status, privacy, video thumbnail
name, status, privacy, video thumbnail
name, status, privacy, video thumbnail
name, status, privacy, video thumbnail
name, status, privacy, video thumbnail
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, video thumbnail
name, status, privacy, video thumbnail
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, video thumbnail
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
[Consequence of a Click -- II] When I click on a friend’s photo, for example, a lot of things happen. The data is retrieved from the database and checked in real time for privacy, visibility and other rules. Each time you click on a photo, status, name, friends, friends of friends, this process takes place.
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, video thumbnail
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo
name, status, privacy, profile photo

9. System Architecture
So what makes the work at Facebook challenging?

10. Architecture Load Balancer (assigns a web server)
Web Server (PHP assembles data)
Kinda inaccurate - memcache and db are on the side. Lots of services...
Memcache (fast, simple)
Database (slow, persistent)

11. Memcache Simple in-memory hash table
Supports get/set,delete,multiget, multiset
Not a write-through cache
Pros and Cons
The Database Shield!
Low latency, very high request rates
Can be easy to corrupt, inefficient for very small items
In many ways, this is the heart or core of the site! 120 million queries/second!!!
equivalent of typing out over 50 volumes of the Encyclopedia Britannica in 1/10th second.

12. Memcache Optimization
Multithreading and efficient protocol code - 50k req/s
Polling network drivers - 150k req/s
Breaking up stats lock - 200k req/s
Batching packet handling - 250k req/s
Breaking up cache lock - future

13. Network Incast Memcache Memcache Memcache Memcache Switch
Many Small Get Requests
PHP Client

14. Network Incast Memcache Memcache Memcache Memcache Switch
Many big data packets
PHP Client

15. Network Incast
Memcache
Memcache
Memcache
Memcache
Switch
PHP Client

16. Network Incast Memcache Memcache Memcache Memcache Switch
- Implement flow control on client over multiple udp connections- aggressive timeout - blowing up memcache past a threshold
PHP Client

17. Memcache Clustering Many small objects per server
Many servers per large object
If objects are small, round trips dominate so you want objects clusteredIf objects are large, transfer time dominates so you want objects distributedIn a web application you will almost always be dealing with small objects, and you can get in a situation where adding machines doesn’t help scaling
Many small objects per server
Many servers per large object

18. Memcache Clustering
Memcache
10 Objects
PHP Client

19. Memcache Clustering Memcache Memcache PHP Client 5 Objects 5 Objects
2 round trips total1 round trip per server
PHP Client

20. Memcache Clustering Memcache Memcache Memcache PHP Client 4 Objects
3 round trips total1 round trip per server
PHP Client

21. Memcache Pool Optimization
Currently a manual process
Replication for obvious hot data sets
Interesting problem: Optimize the allocation based on access patterns

22. Vertical Partitioning of Object Types
Specialized Replica 1
Specialized Replica 2
Shard 1
Shard 2
Shard 1
Shard 2
General pool with wide fanout
Shard 1
Shard 2
Shard 3
Shard n
...

23. MySQL has played a role from the beginning
Thousands of MySQL servers in two datacenters
Scribe
Today, our user database cluster is a large pool of independent MySQL servers. We have chosen to use a shared-nothing architecture both to achieve scalability and fault isolation.
Battleship vs. army of foot soldiers

24. MySQL Usage Pretty solid transactional persistent store
Logical migration of data is difficult
Logical-Physical db mapping
Rarely use advanced query features
Performance
Database resources are precious
Web tier CPU is relatively cheap
Distributed data - no joins!
Sound administrative model

25. MySQL is better because it is Open Source
We can enhance or extend the database
...as we see fit
...when we see fit
Facebook extended MySQL to support distributed cache invalidation for memcache
INSERT table_foo (a,b,c) VALUES (1,2,3) MEMCACHE_DIRTY key1,key2,...

26. Scaling across datacenters
West Coast
East Coast
VA MySQL
VA Web
VA Memcache
Memcache Proxy
SC Memcache
SC Web
SC MySQL
SF Web
SF Memcache
Memcache Proxy
MySql replication

27. Other Interesting Issues
Application level batching and parallelization
Super hot data items
Cachekey versioning with continuous availability

28. Photos
So what makes the work at Facebook challenging?

29. Photos + Social Graph = Awesome!

30. Photos: Scale 20 billion photos x4 = 80 billion
Would wrap around the world more than 10 times!
Over 40M new photos per day
600K photos / second

31. Photos Scaling - The easy wins
Upload tier - handles uploads, scales images, stores on NFS
Serving tier: Images served from NFS via HTTP
However...
File systems are not good at supporting large number of files
Metadata too large to fit in memory causing too many IOs for each file read
Limited by I/O not storage density
Easy wins
CDN
Cachr (http server + caching)
NFS file handle cache
10 i/os - 3 ios with easy wins

32. Photos: Haystack
Overlay file system
Index in memory
One IO per read

33. Data Warehousing
So what makes the work at Facebook challenging?

34. Data: How much? 200GB per day in March 2008
2+TB(compressed) raw data per day in April 2009
4+TB(compressed) raw data per day today

35. The Data Age Free or low cost of user services
Consumer behavior hard to predict
Data and analysis are critical
More data beats better algorithms

36. Deficiencies of existing technologies
Analysis/storage on proprietary systems too expensive
Closed systems are hard to extend

37. Hadoop & Hive
So what makes the work at Facebook challenging?

38. Hadoop
Superior availability/scalability/manageability despite lower single node performance
Open system
Scalable costs
Cons: Programmability and Metadata
Map-reduce hard to program (users know sql/bash/python/perl)
Need to publish data in well known schemas

39. Hive
A system for managing and querying structured data built on top of Hadoop
Components
Map-Reduce for execution
HDFS for storage
Metadata in an RDBMS

40. Hive: New Technology, Familiar Interface
hive> select key, count(1) from kv1 where key > 100 group by key;
vs.
$ cat > /tmp/reducer.sh
uniq -c | awk '{print $2"\t"$1}‘
$ cat > /tmp/map.sh
awk -F '\001' '{if($1 > 100) print $1}‘
$ bin/hadoop jar contrib/hadoop dev-streaming.jar - input /user/hive/warehouse/kv1 -mapper map.sh -file /tmp/reducer.sh -file /tmp/map.sh -reducer reducer.sh - output /tmp/largekey -numReduceTasks 1
$ bin/hadoop dfs –cat /tmp/largekey/part*

41. Hive: Sample Applications
Reporting
E.g.,: Daily/Weekly aggregations of impression/click counts
Measures of user engagement
Ad hoc Analysis
E.g.,: how many group admins broken down by state/country
Machine Learning (Assembling training data)
Ad Optimization
E.g.,: User Engagement as a function of user attributes
Lots More

42. Hive: Server Infrastructure
4800 cores, Storage capacity of 5.5 PetaBytes, 12 TB per node
Two level network topology
1 Gbit/sec from node to rack switch
4 Gbit/sec to top level rack switch

43. Hive & Hadoop: Usage Stats
4 TB of compressed new data added per day
135TB of compressed data scanned per day
7500+ Hive jobs on per day
80K compute hours per day
200 people run jobs on Hadoop/Hive
Analysts (non-engineers) use Hadoop through Hive
95% of jobs are Hive Jobs

44. Hive: Technical Overview
So what makes the work at Facebook challenging?

45. Hive: Open and Extensible
Query your own formats and types with your own Serializer/Deserializers
Extend the SQL functionality through User Defined Functions
Do any non-SQL transformations through TRANSFORM operator that sends data from Hive to any user program/script

46. Hive: Smarter Execution Plans
Map-side Joins
Predicate Pushdown
Partition Pruning
Hash based Aggregations
Parallel execution of operator trees
Intelligent Scheduling

47. Hive: Possible Future Optimizations
Pipelining?
Finer operator control (controlling sorts)
Cost based optimizations?
HBase

48. Spikes: The Username Launch
The numbers presented earlier should give you a sense of the scale Facebook operates at and the challenges they might throw. Facebook requires a giant infrastructure and at also a very diverse array of components. The cool thing is that for every class you’re into, we have the opportunity for you to dive into that field, and be right at the forefront. And luckily, Facebook is also a place that is really a continuation of our study, that is broad and challenging.

49. System Design Database tier cannot handle the load
Dedicated memcache tier for assigned usernames
Miss => Available
Avoid database hits altogether
Blacklists: bucketize, local tier cache
timeout
- Performance of avail checker was one of the most critical parts of the system
- This wasn’t a big issue when were going down the auction path, but became critical once we decided to do FCFS
- Generating suggestions adds more load (could do maybe 6-10 or more checks per page load)
- Usual caching doesn’t help a lot since there will be a lot of misses. DB tier cannot handle the load
Constant refreshing was a concern - so we added a countdown to incentivize users to just watch. Upon countdown completion, don’t auto-refresh - show a continue button.
Disabled Chat bar.
Lite include stack.
Don’t pull in your set of pages unnecessarily.
Tradeoffs between UX (extra clicks) and performance.

50. Username Memcache Tier
Parallel pool in each data center
Writes replicated to all nodes
8 nodes per pool
Reads can go to any node (hashed by uid)
PHP Client
...
- Replicated memcache tier
- Key gets hashed to a number 0-7 (based on uid?)
- But these are not db backed keys & so sets need to be replicated
UN0
UN1
UN7
Username Memcache

51. Write Optimization Hashout store
Distributed key-value store (MySQL backed)
Lockless (optimistic) concurrency control
- Hashout store stores the mapping between username -> <alias fbid> (which has some more data)
- - optimistic concurrency => no locks are obtained and since conflict rates are low, it is a win

52. Fault Tolerance Memcache nodes can go down
Always check another node on miss
Replay from a log file (scribe)
Memcache sets are not guaranteed to succeed
Self-correcting code: write again to mc if we detect it during db writes
- One of the issues we were worried about with non-db backed keys & assuming MC as the system of record was that mc nodes can go down.
- MC replicated pools allows some redundancy (on miss, the replicated pool mc infrastructure allows us to check another node on miss)
- replay scribe logs for recovery from more than 1 failure (this actually happened!)
- MC sets not guaranteed to succeed => bad user experience. writes will not succeed since the database will complain on writes, but still this is not ideal. So, if we find that a username is taken during the db-write stage, we call set again to write those usernames to the UN memcache tier. Future users are prevented from seeing the same problem

53. Nuclear Options Newsfeed Reduce number of stories
Turn off scrolling, highlights
Profile
Make info tab the default
Chat
Reduce buddy list refresh rate
Turn if off!
Even though we designed the system to be highly performant, and tested it under a lot of load, there were chances that if a really really large number of people showed up at launch time, our system may not be able to handle it. So, we thought about ways in which we can reduce the load caused by other features on the site to provide extra capacity for the username launch. We affectionately called this the “nuclear options” since this was a last resort for handling the enormous load.

54. How much load? 200k in 3 min 1M in 1 hour 50M in first month
Prepared for over 10x!
The actual load was high, but not anywhere near what we were prepared for. Shows you an example of careful design, planning and testing can lead to a successful launch.

55. Some interesting problems
The spike!

56. Some interesting problems
Graph models and languages
Low latency fast access
Slightly more expressive queries
Consistency, Staleness can be a bit loose
Analysis over large data sets
Privacy as part of the model
Fat data pipes
Push enormous volumes of data to several third party applications (E.g., entire newsfeed to search partners).
Controllable QoS

57. Some interesting problems (contd.)
Search relevance
Storage systems
Middle tier (cache) optimization
Application data access language

58. Questions?
Sum it up.