1. Engineering v v Adam Cataldo Tuesday, January 24, 2012 Quick Deploy A distributed systems approach to developer productivity

2. Engineering Outline  LinkedIn architecture  Quick Deploy  Cool technical details 2

3. Engineering LinkedIn 3  Largest professional networking site  135M+ members  14 languages  200+ countries and territories  Estimated 4B+ people searches per year

4. Engineering LinkedIn stack at 20,000 feet  Online system for serving up data to clients –Profile pages, company pages, LinkedIn groups, etc. –Key metric: Latency  Offline system for creating insight from data –People you may know, contact degree of separation, search indexing, etc. –Key metric: Throughput 4

5. Engineering Offline data processing 5 Hadoop Voldemort Map Reduce

6. Engineering Online web application 6 ClientPresentationBusiness Logic Data Service Datastore

7. Engineering Growing Pains for the Online System As time has gone by: –We’ve added new services to support new features –We’ve added new services to make our site faster and more available –Our engineering productivity dropped 7

8. Engineering Quick Deploy  Developers only locally deploy the services they actually modify  Every service your service depends on is available in some data center  This makes development move much faster! 8 Angelika Clayton

9. Engineering 9 Early Integration DatacenterDev Box QD Proxy Browser Local Mid Tier Shared Frontend Shared Data Service DB

10. Engineering In Detail 10 Browser Proxy EI Frontend Dev-local Mid Tier EI Data Service QD Cookie QD IC

11. Engineering Proxy Behavior 11 Proxy receives request Has QD cookie? Route to developer’s local service Route to service in EI Read identifier from cookie Read identifier from IC Is service local? no yes no

12. Engineering QD Cookie and Invocation Context  A developer creates an identifier key, and registers his local services with the proxy  He identifies himself by adding his key to a cookie from his browser  When a front end gets a request with a QD cookie, it puts the key in the invocation context  Services pass the invocation context with every outbound request 12

13. Engineering Shared data 13 Oracle MySQL Voldemort Memcached Oracle MySQL Voldemort Memcached Change Alfred’s language to Japanese きんぎょ Huh?

14. Engineering ActiveMQ 14 Early Integration DatacenterDev Box Producer Consumer Produces message Consumes message Queue Never sees message

15. Engineering Timers 15 Early Integration DatacenterDev Box Service RPC Target Schedules time-triggered RPC call Gets RPC call after timer triggers Never gets RPC call

16. Engineering Proxy in detail 16 Admin Console Python/Flask Web App Admin Console Python/Flask Web App Proxy Apache Traffic Server Proxy Apache Traffic Server > 10,000 QPS! Routing Profiles Apache Zookeeper Routing Profiles Apache Zookeeper Proxy Apache Traffic Server Proxy Apache Traffic Server Proxy Apache Traffic Server Proxy Apache Traffic Server

17. Engineering Why Apache Zookeeper  Distributed file system for small files  Basically, a key-value data store, where the keys are file names  Clients can listen for file change and directory change events  Great for storing data  Also, great publish/subscribe system 17

18. Engineering Dynamic Discovery 18 Early Integration DatacenterDev Box QD Proxy Browser Local Mid Tier Shared Frontend Shared Data Service DB

19. Engineering Conclusion  The LinkedIn architecture scales well in production  Quick deploy makes the architecture scale for developers  20% productivity gain across the engineering organization 19

20. Engineering Thanks 20 Rob Loh Swee Lim Steve Ihde Jim Dumont