1. Real-Time Performance at Massive Scale
Francois Orsini
Machine Zone, Inc.

2. Massive Scalability What does Massive Scalability mean?
What is the Upper bound?
How to reach Scalability Nirvana?
Brewer CAP Theorem
Constant growth + real-time = Challenge
What kind of growth? Exponential, linear, cubic, logarithmic, logistics (upper bound)?
Exponential Decay
Have a performance goal
Always have Plan B
Measure everything
Consistency, Availability and Partition Tolerance

3. Real-Time @ Massive Scale

4. Real-Time Translation @ Massive Scale

5. Global Global Challenges Worldwide Reach Single World / N Regions
No Language Barrier
5M-40M+ Daily Active Users
k Concurrent Users
N client platforms
iOS
Android
Windows
24 x 7 non-stop
Global

6. Real-Time Real-Time Real-Time Chats Real-Time Events High Availability
Including real-time translation
Real-Time Events
Massive Parallel broadcasting
Auto-subscribe model
Real-Time Push Notifications
High Availability
Hot stand-by
99.99% (“four nines”) ~< 53mns down / year
Real-Time

7. Massive Scaling Massive Scaling Big Scale Architecture
Scaling Up + Out
TCO what?
Fully Distributed
Fully Redundant
Cluster Native
Automatic Failover
Cloud Aware
Massive Scaling

8. Social 2.0 Social 2.0 Defining Social 2.0 Social World Changer
No language barrier
Real-time
Messaging
Translation
Global scale and reach
Social World Changer
Not just social gaming
Social 2.0

9. What’s the Secret Sauce?
Know your subject
Understand what you are building
New is good but bench / stress test it!
Use the right tools for the Job!
Do not hesitate to vary
Pick proven software (not just the latest cool one)
Open Source
Fork as needed
Rewrite early as necessary

10. What’s the Secret Sauce? (Cont.)
Profile everything
Start doing it early
Care about performance
No single point of Failure
Horizontally and vertically
Yeah right == Wrong
Too many companies launching very popular games and hitting a wall

11. Real-Time Ingredient Erlang Functional programming WTH? Hidden jewel
Proven, created in 1986, open sourced in 1998
Supports distributed, fault-tolerant, soft real-time non-stop applications
Great level of concurrency (no locks)
Hot-swapping of Code
Scales up & out
Functional programming WTH?
Paradigm shift but makes a lot of sense
Hire the right competences
Get trained (don’t learn on your own)

12. Erlang Components Real-time Chat Real-time Events Real-time Timers
Large chat rooms
Private 1-1 chat
Full chats history support
Languages transformation & translation
Real-time Events
Real-time Timers
Real-time Push Notifications
Real-time Message Queuing & Delivery

13. Data Storage Ingredient
MySQL (Percona)
Very reliable and predictable
Proven (used by the biggest applications out there)
Not the perfect storage solution but goes a long way
Scale out solution (“logical shards” / partitions)
Cluster (auto-failover)
InnoDB is reliable
Support for direct data access (e.g. HandlerSocket)
SQL == Most expressive declarative language
SQL is NOT dead == WITHSQL

14. Data Storage (Cont.) Master-Master topology Scale Up and Out
Use 5.5+
How to reduce replication lag
Partition tables (5.1+)
HandlerSocket plug-in (750K QPS), Memcached plug-in
Facebook Faker (
Percona Toolkit, ”user_stat”
What!? No NOSQL!?
Fear not … The next slide is near
Know what you will gain and lose
WITHSQL + NOSQL

15. Data Shards Storage Topology
Primary
Secondary
Shards Map

16. NOSQL Ingredient Redis Works great coupled with HaProxy / Heartbeat
Proven, fast, powerful, yet simple
Scale out with Sharding (Cluster is Beta)
Great for object caching
100k+ requests per second
Extremely reliable
Fast, Fast, Fast!
Works great coupled with HaProxy / Heartbeat

17. Shards REDIS HA TOPOLOGY Master Slave Middle-Tier / WAS HA HA HA
(Zookeeper)
Slave
Middle-Tier / WAS

18. Some Numbers
3.5M soft real-time messages to a single channel per second.
Benchmarked with actual connected endpoints
500k concurrent and active users
Benchmarked with actual connected player bots
> 100k writes per second
> 250K reads per second
Hundreds of real-time battles in 1 single Map chunk
> 1k phrase translations per second minimum

19. Alternatives & Misc Products
NOSQL
Cassandra
RIAK (Erlang)
Couchbase
MariaDB (MySQL Fork + Percona XtraDB)
PostgreSQL
Other Commercial RDBMS
Continuent Tungsten

20. Summary Know what you’re up to! Use proven approaches
Global scale, volumes
Research!
Use proven approaches
Don’t be afraid to innovate
Bench, bench and bench!
Don’t wait a few weeks before WW launch
Profile the code
Have a plan B