Download presentation
Presentation is loading. Please wait.
1
Project Voldemort Bhupesh Bansal & Jay Kreps
Thanks all, excited to talk to you 4/15/2017
2
The Plan Motivation Core Concepts Implementation In Practice Results
3
LinkedIn’s Data & Analytics Team Project Voldemort Hadoop system
Who Are We? LinkedIn’s Data & Analytics Team Project Voldemort Hadoop system Recommendation Engine Relevance and Analysis Data intensive features People you may know Who viewed my profile User history service
4
Motivation Why write a database?
How did this come about--personal project with uses
5
The Idea of the Relational Database
One-size-fits-all Stonebraker
6
The Reality of a Modern Web Site
7
Specialized systems are efficient (10-100x) Search: Inverted index
Why did this happen? Specialized systems are efficient (10-100x) Search: Inverted index Offline: Hadoop, Terradata, Oracle DWH Memcached In memory systems (social graph) Specialized system are scalable New data and problems Graphs, sequences, and text 10-100x faster Scaling Once you have 4 mandatory specialists what is left for the RDMS?
8
Services and Scale Break Relational DBs
No joins Lots of denormalization ORM is pointless No constraints, triggers, etc disappear Natural operations not captured easily Caching => key/value model Latency is key No Joins Across data domains due to APIs Within data domains due to performance Natural operation: getAll(id…) Latency: if you want to call 30 services on your main pages, they better be quick (30 * 20ms = 600ms)
9
Two Cheers For Relational Databases
The relational model is a triumph of computer science: General Concise Well understood But then again: SQL is a pain Hard to build re-usable data structures Don’t hide the memory hierarchy! Good: Filesystem API Bad: SQL, some RPCs Exposes seek/hides seek
10
Who is responsible for performance (engineers? DBA? site operations?)
Other Considerations Who is responsible for performance (engineers? DBA? site operations?) Can you do capacity planning? Can you simulate the problem early in the design phase? How do you do upgrades? Can you mock your database?
11
Some domain constraints
This is a real-time system Data set is large and persistent Cannot be all in memory Partitioning is key 80% of caching tiers are fixing problems that shouldn’t exist Need control over system availability and data durability Must replicate data on multiple machines Cost of scalability can’t be too high Must support diverse usages Explain about ratio of memory to disk
12
Inspired By Amazon Dynamo & Memcached
Amazon’s Dynamo storage system Works across data centers Eventual consistency Commodity hardware Not too hard to build Memcached Actually works Really fast Really simple Decisions: Multiple reads/writes Consistent hashing for data distribution Key-Value model Data versioning
13
Priorities Performance and scalability Actually works Community
Data consistency Flexible & Extensible Everything else -If you haven’t solved a performance problem you haven’t done anything -- mysql is really quite good for < 2,000 requests / sec and smaller data sizes Storage is Achilles heal of services, can bring down the site Don’t want to have the wrong stuff Very diverse use cases
14
Failures in a distributed system are much more complicated
Why Is This Hard? Failures in a distributed system are much more complicated A can talk to B does not imply B can talk to A A can talk to B does not imply C can talk to B Getting a consistent view of the cluster is as hard as getting a consistent view of the data Nodes will fail and come back to life with stale data I/O has high request latency variance I/O on commodity disks is even worse Intermittent failures are common User must be isolated from these problems There are fundamental trade-offs between availability and consistency
15
Core Concepts Switch to Bhup
16
ACID CAP Theorem Consistency Models Core Concepts - I
Great for single centralized server. CAP Theorem Consistency (Strict), Availability , Partition Tolerance Impossible to achieve all three at same time in distributed platform Can choose 2 out of 3 Dynamo chooses High Availability and Partition Tolerance by sacrificing Strict Consistency to Eventual consistency Consistency Models Strict consistency 2 Phase Commits PAXOS : distributed algorithm to ensure quorum for consistency Eventual consistency Different nodes can have different views of value In a steady state system will return last written value. BUT Can have much strong guarantees. - Strong Consistency: all clients see the same view, even in presence of updates - High Availability: all clients can find some replica of the data, even in the presence of failures Partition-tolerance: the system properties hold even when the system is partitioned high availability : Mantra for websites Better to deal with inconsistencies, because their primary need is to scale well to allow for a smooth user experience. Proprietary & Confidential 4/15/2017 16 16
17
Key space is Partitioned Partitions never change Replication
Hashing .. Why do we need it ?? Basic problem : Clients need to know which data is where ?? Many ways of solving it Central configuration Hashing Linear hashing works : issue is when cluster is dynamic ?? KeyHash –node IDmapping change for a lot of entries When you add new slots Consistent hashing : preserves key –Node mapping for most of the keys and only change the minimal amount needed How to do it ?? Number of partitions Arbitrary , each node is allocated many partitions (better load balancing and fault tolerance) Few hundreds to few thousands .. Key partition mapping is fixed and only ownership of partitions can change Core Concept - II Consistent Hashing Key space is Partitioned Many small partitions Partitions never change Partitions ownership can change Replication Each partition is stored by ‘N’ nodes Node Failures Transient (short term) Long term Needs faster bootstrapping Proprietary & Confidential 4/15/2017 17
18
N - The replication factor R - The number of blocking reads
Core Concept - III N - The replication factor R - The number of blocking reads W - The number of blocking writes If R+W > N then we have a quorum-like algorithm Guarantees that we will read latest writes OR fail R, W, N can be tuned for different use cases W = 1, Highly available writes R = 1, Read intensive workloads Knobs to tune performance, durability and availability Proprietary & Confidential 4/15/2017 18
19
A vector clock is a tuple {t1 , t2 , ..., tn } of counters.
Core Concepts - IV Vector Clock [Lamport] provides way to order events in a distributed system. A vector clock is a tuple {t1 , t2 , ..., tn } of counters. Each value update has a master node When data is written with master node i, it increments ti. All the replicas will receive the same version Helps resolving consistency between writes on multiple replicas If you get network partitions You can have a case where two vector clocks are not comparable. In this case Voldemort returns both values to clients for conflict resolution Fancy way of doing Optimistic locking Proprietary & Confidential 4/15/2017 19
20
Implementation
21
One interface for all layers: put/get/delete
Voldemort Design Will discuss each layer in more detail Layered design One interface for all layers: put/get/delete Each layer decorates the next Very flexible Easy to test Client API : very basic API just provides the raw interface to user Conflct reslution layer : handles all the versioning issues and provides hooks to supply custom conflict resolution strategy Serialization : Object <=> Data Network Layer : Depending on configuration can fit either here or below .. Main job is to handle the network interface, socket pooling other performance related optimizatons Routing layer : Handles and hide many details from the client/user hashing schema failed nodes replication required reads/required writes Storage engine Handle disk persistenct
22
Data is organized into “stores”, i.e. tables Key-value only
Client API Data is organized into “stores”, i.e. tables Key-value only But values can be arbitrarily rich or complex Maps, lists, nested combinations … Four operations PUT (Key K, Value V) GET (Key K) MULTI-GET (Iterator<Key> K), DELETE (Key K) / (Key K , Version ver) No Range Scans Very simple APIS NO Range Scans .. . No iterator on KeySet / Entry SET : Very hard to fix performance Have plans to provide such an iterator
23
Versioning & Conflict Resolution
Eventual Consistency allows multiple versions of value Need a way to understand which value is latest Need a way to say values are not comparable Solutions Timestamp Vector clocks Provides global ordering. No locking or blocking necessary Give example of read and writes with vector clocks Pros and cons vs paxos and 2pc User can supply strategy for handling cases where v1 and v2 are not comparable.
24
Serialization Really important Few Considerations Schema free?
Backward/Forward compatible Real life data structures Bytes <=> objects <=> strings? Size (No XML) Many ways to do it -- we allow anything Compressed JSON, Protocol Buffers, Thrift, Voldemort custom serializtion Avro is good, but new and unreleased Storing data is really different from an RPC IDL Data is forever (some data) Inverted index Threaded comment tree Don’t hardcode serialization Don’t just do byte[] -- checks are good, many features depend on knowing what the data is Xml profile
25
Routing Routing layer hides lot of complexity Hashing schema Replication (N, R , W) Failures Read-Repair (online repair mechanism) Hinted Handoff (Long term recovery mechanism) Easy to add domain specific strategies E.g. only do synchronous operations on nodes in the local data center Client Side / Server Side / Hybrid
26
Voldemort Physical Deployment
Explain about partitions Make things fast by removing slow things, not by tuning HTTP client not performant Separate caching layer
27
Need to be very very fast View of server state may be inconsistent
Routing With Failures Failure Detection Requirements Need to be very very fast View of server state may be inconsistent A can talk to B but C cannot A can talk to C , B can talk to A but not to C Currently done by routing layer (request timeouts) Periodically retries failed nodes. All requests must have hard SLAs Other possible solutions Central server Gossip protocol Need to look more into this. Client v. server - client is better - but harder
28
Bootstrapping mechanism (We don’t have it yet)
Repair Mechanism Read Repair Online repair mechanism Routing client receives values from multiple node Notify a node if you see an old value Only works for keys which are read after failures Hinted Handoff If a write fails write it to any random node Just mark the write as a special write Each node periodically tries to get rid of all special entries Bootstrapping mechanism (We don’t have it yet) If a node was down for long time Hinted handoff can generate ton of traffic Need a better way to bootstrap and clear hinted handoff tables Proprietary & Confidential 4/15/2017 28
29
Network Layer Network is the major bottleneck in many uses Client performance turns out to be harder than server (client must wait!) Lots of issue with socket buffer size/socket pool Server is also a Client Two implementations HTTP + servlet container Simple socket protocol + custom server HTTP server is great, but http client is 5-10X slower Socket protocol is what we use in production Recently added a non-blocking version of the server
30
Single machine key-value storage is a commodity
Persistence Single machine key-value storage is a commodity Plugins are better than tying yourself to a single strategy Different use cases optimize reads optimize writes Large vs Small values SSDs may completely change this layer Couple of different options BDB, MySQL and mmap’d file implementations Berkeley DBs most popular In memory plugin for testing Btrees are still the best all-purpose structure No flush on write is a huge, huge win You can write an implementation very easily We support plugins so you can run this backend without being in the main code base
31
In Practice
32
LinkedIn problems we wanted to solve
Application Examples People You May Know Item-Item Recommendations Typeahead selection Member and Company Derived Data User’s Network statistics Who Viewed My Profile? Abuse detection User’s History Service Relevance data Crawler detection Many others have come up since Some data is batch computed and served as read only Some data is very high write load Latency is key Not discusssing these because they are too linkedin-specific
33
User Data lookup service
Example Use Cases - I User Data lookup service Returns user data for a user. Create a single Voldemort Store Key : User ID Value : User Settings Treat Voldemort as a simple Hash Table with extra benefits Scalability Fault tolerance Persistence Data Size > RAM Can control Data / RAM ratio for better performance There are “surprisingly” a lot of use cases which can be served by this simple model “provide best seller lists, shopping carts, customer preferences, session management, sales rank, and product catalog” [Dynamo paper] Proprietary & Confidential 4/15/2017 33
34
User Comments Service Insert time: Query time: Example Use Cases - II
Return Latest ‘N’ comments for a topic Create two Voldemort Stores Store 1: Key: Topic Id Value : CommentID tree reverse sorted by timestamp Store 2: Key : commentID Value: Comment data Insert time: Get the commentID tree using store 1. Update the tree and put it back in store1 Insert new key/value pair in store2 Query time: Do a query on Store1 and then multi-get Query for last ‘N’ comments. Proprietary & Confidential 4/15/2017 34 34
35
Inbox Search Service Insert time Query Time Example Use Cases - III
Search for a ‘keyword’ phrase in member Inbox data Create a Voldemort Store. Key: User ID Value : Search Index for user s Insert time Get the search index for user given userID Update search index with new update and put back to store. Query Time get the right index using userID use search index to query for ‘keyword’ and return results Try to colocate voldemort/Inbox Search service to avoid network hops Proprietary & Confidential 4/15/2017 35
36
Example which do not work
Financial Transactions Please donot use *any* alpha system to save credit card details. Transaction requirement between 2 stores Write key1 to store A iff key2 to store B succeeded. Range scans Switch to Jay! Proprietary & Confidential 4/15/2017 36
37
Example V: Hadoop and Voldemort sitting in a tree…
Hadoop can generate a lot of data Bottleneck 1: Getting the data out of hadoop Bottleneck 2: Transfer to DB Bottleneck 3: Index building We had a critical process where this process took a DBA a week to run! Index building is a batch operation Fixing a bottleneck in one system creates bottlenecks elsewhere 4/15/2017
39
Read-only storage engine
Throughput vs. Latency Index building done in Hadoop Fully parallel transfer Very efficient on-disk structure Heavy reliance on OS pagecache Rollback! Transfer time: 30 minutes Can max out a gb network, so be careful
40
Wide variety of data sizes, clients, needs My team: 12 machines
Voldemort At LinkedIn 4 Clusters, 4 teams Wide variety of data sizes, clients, needs My team: 12 machines Nice servers 500M operations/day ~4 billion events in 10 stores (one per event type) Peak load > 10k operations / second Other teams: news article data, related data, UI settings Bigger than core db!
41
Gilt Group Gilt Groupe Inc ( Thanks to Geir Magnusson for sharing numbers 3 Clusters (Another 3 coming in 3 weeks) 4 Machines each Replication 2 required Bdb storage engine Values avg size about 5K QPS : upto 2700/sec Latency 96 % < 20ms 99.7 % < 40 ms Production issues seen: load balancer started caching causing writes to fail
42
Results State of the project Performance What is left to do Etc.
43
Some performance numbers
Production stats Median: 0.1 ms 99.9 percentile GET: 3 ms Single node max throughput (1 client node, 1 server node): 19,384 reads/sec 16,559 writes/sec These numbers are for mostly in-memory problems Client bound
44
Glaring Weaknesses Not nearly enough documentation
Need a rigorous performance and multi-machine failure tests running NIGHTLY No online cluster expansion (without reduced guarantees) Need more clients in other languages (Java, Python, and C++ currently) Better tools for cluster-wide control and monitoring
45
State of the Project Active mailing list
4-5 regular committers outside LinkedIn Lots of contributors Equal contribution from in and out of LinkedIn Project basics IRC Some documentation Lots more to do > 300 unit tests that run on every checkin (and pass) Pretty clean code Moved to GitHub (by popular demand) Production usage at a half dozen companies Not just a LinkedIn project anymore But LinkedIn is really committed to it (and we are hiring to work on it) - Open source n00b - Very impressed by all the friendly smart people out there who love to hack on things
46
Index build runs 100% in Hadoop
MapReduce job outputs Voldemort Stores to HDFS Nodes all download their pre-built stores in parallel Atomic swap to make the data live Heavily optimized storage engine for read-only data I/O Throttling on the transfer to protect the live servers
47
Some new & upcoming things
Python client Non-blocking socket server Alpha round on online cluster expansion Read-only store and Hadoop integration Improved monitoring stats Compression Future Publish/Subscribe model to track changes Great performance and integration tests Improved failure detection
48
Shameless promotion Check it out: project-voldemort.com We love getting patches. We kind of love getting bug reports. LinkedIn is hiring, so you can work on this full time. me if interested
49
The End Questions, comments, etc
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.