Amazon’s Dynamo Simple Cloud Storage

Foundations 1970 – E.F. Codd “A Relational Model of Data for Large Shared Data Banks”E.F. Codd –Idea of tabular data –SQL Foundations Codd’s 12 rules –How database structured and what is available to user –Application not dependent on physical or logical levels of database –Insert, Update, Delete operators

Foundations, continued First Relational database management systems (RDBMS) –Oracle in 1979, first SQL based system –Microsoft SQL server, etc –Open source software would follow later (mySQL) Follows Codd’s ideas –Complexity on the server side, let the query do all the work –Very stringent requirements

Drawbacks Licensing fees on a per processor rate – High end Oracle system is mind-numbingly expensivemind-numbingly expensive Load Distribution requires specific nodes to handle – Some servers have specific roles, failure point in network Complexity on servers creates difficultly with maintenance, upgrades – Upgrades often all at once as result, not incremental

A New Direction Simplify services that the database provides – Easier scaling and error handling Take a more pragmatic approach – Tailor system to sacrifice some aspects of the traditional RDBMS to gain performance in others – Systems less general, specific end requirements in mind when creating

Examples Amazon Dynamo – Simple primary key – Highly available, end user based model – Low cost virtualized nodes Facebook Cassandra – Similar goals to Amazon’s Dynamo – Highly avaiable, incremental scalablilty Google File System – Master node – Data distributed across low cost nodes

Dynamo Goals Scale – adding systems to network causes minimal impact Symmetry – No special roles, all features in all nodes Decentralization – No Master node(s) Highly Available – Focus on end user experience SPEED – A system can only be as fast as the lowest level Service Level Agreements – System can be adapted to an application’s specific needs, allows flexibility

Dynamo Assumptions Query Model – Simple interface exposed to application level – Get(), Put() – No Delete() Atomicity, Consistency, Isolation, Durability – Operations either succeed or fail, no middle ground – System will be eventually consistent, no sacrifice of availability to assure consistency – Conflicts can occur while updates propagate through system – System can still function while entire sections of network are down Efficiency – Measure system by the 99.9 th percentile – Important with millions of users, 0.1% can be in the 10,000s Non Hostile Environment - No need to authenticate query, speed boost

Wanted Results Deliver requests in a bounded time Always writable – Highly available to users No dedicated roles Work split between nodes fairly

Techniques

Partitioning Consistent Hashing – Changing the number of slots in hash table results in only a small number of keys to remap – More info More info A ring of virtual nodes – Node responsible for region between it and its predecessor

Virtual Node Physical Machine has # of virtual nodes based on performance Can adapt load more easily if a machine goes down Likewise, assign nodes to a new machine in network

Replication Application provided parameter N Replication on different physical nodes – Data still available if nodes go down – Makes part of preference list for query

Versioning and Vector Clocks Updates propagate asynchronously, need a way of distinguishing conflicts – Possible reason for absence of Delete() Vector Clock – List of (node, counter) – Limited size, limit overhead for data – If all fields are less than or equal, first can be updated by second

Sloppy Quorum and Hinted Handoff W and R parameter set min # of nodes in a read or write Read and write on the first N healthy nodes, no strict membership, can vary over time Hint in metadata for intended node, will update once that node is again available Allows for temporary failure in nodes or entire networks

Synchronization and Gossip Merkle Trees - InfoInfo Use common key values between two nodes – Traverse tree and check vector clocks to see if updates are needed – Exchange information on most current version of the data if inconsistencies are found Gossip – Nodes select neighbors at random and reconcile membership change histories Use seed nodes to initialize Detect failures

Routing get() and put() Two Techniques – Route request through a load balancer Slower Simpler application level code – Partition aware client, route directly to appropriate nodes Faster More complicated application level First node routed to is “coordinator” node – Generates vector clock for put and gives data to N highest healthy nodes – Queries N highest nodes for all versions, returned all versions found

Implementation Java based – Hardware independent, JVM Allows different back-end systems to be used, based on size of data needed to be stored – Berkeley Database Transactional Data Store – BDB Java Edition – MySQL, can handle large objects Coordinator node is a state machine for read/writes for client – Coordinator for a write determined by fastest read

Flexibility Changing W, R, N – Business logic specific reconciliation Data replicated over nodes Application level reconciliation fro conflicting objects – Timestamp Reconciliation Similar to above, last write wins – High performance read engine By setting R = 1, W = N Reads fast and numerous, few updates

Observed Results - Speed

Observed Results – Load Balancing Higher traffic causes load to be balanced more evenly – Requests of popular keys let system to balance more easily In lower traffic, less important to balance load

Observed Results - Coordination Client coordination can provide a speed boost Read and write latency nearly identical Results as expected

Observed Results - Versions Measured over 24 hour period for shopping cart – 99.94% of users saw 1 version – % saw 2 versions – % saw 3 versions – % saw 4 versions Increase caused by increase in number of concurrent writers, most likely

Conclusions Dynamo allows Amazon’s customers to have a consistent experience even in face of server and network errors Gives a scalable solution with millions of data points to be queried quickly and efficiently Offloads complexity to the application to provide a simple, flexible, and fast server-side implementation

Thanks for listening!