PNUTS: Yahoo!’s Hosted Data Serving Platform Brian F. Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Philip Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver and Ramana Yerneni Yahoo! Research

How do I build a cool new web app? Option 1: Code it up! Make it live! Scale it later It gets posted to slashdot Scale it now! Flickr, Twitter, MySpace, Facebook, …

How do I build a cool new web app? Option 2: Make it industrial strength! Evaluate scalable database backends Evaluate scalable indexing systems Evaluate scalable caching systems Architect data partitioning schemes Architect data replication schemes Architect monitoring and reporting infrastructure Write application Go live Realize it doesn’t scale as well as you hoped Rearchitect around bottlenecks 1 year later – ready to go!

Example: social network updates Brian Sonja Jimi Brandon Kurt What are my friends up to? Sonja: Brandon:

Example: social network updates 6 Jimi <ph.. 8 Mary <re.. 12 Sonja <ph.. 15 Brandon <po.. <photo> <title>Flower</title> <url>www.flickr.com</url> </photo> 16 Mike <ph.. 17 Bob <re..

What do we need from our DBMS? Web applications need: Scalability And the ability to scale linearly Geographic scope High availability Web applications typically have: Simplified query needs No joins, aggregations Relaxed consistency needs Applications can tolerate stale or reordered data

What is PNUTS?

What is PNUTS? Indexes and views CREATE TABLE Parts ( ID VARCHAR, E 75656 C A 42342 E B 42521 W C 66354 W D 12352 E F 15677 E A 42342 E B 42521 W C 66354 W D 12352 E E 75656 C F 15677 E Indexes and views CREATE TABLE Parts ( ID VARCHAR, StockNumber INT, Status VARCHAR … ) E 75656 C A 42342 E B 42521 W C 66354 W D 12352 E F 15677 E Geographic replication Parallel database Structured, flexible schema Hosted, managed infrastructure

Query model Per-record operations Multi-record operations Get Set Delete Multi-record operations Multiget Scan Getrange Web service (RESTful) API

Detailed architecture Clients Data-path components REST API Routers Message Broker Tablet controller Storage units

Detailed architecture Local region Remote regions Clients REST API Routers YMB Tablet controller Storage units

Tablet splitting and balancing Each storage unit has many tablets (horizontal partitions of the table) Storage unit may become a hotspot Storage unit Tablet Overfull tablets split Tablets may grow over time Shed load by moving tablets to other servers

Query processing

Range queries Grapefruit…Pear? Storage unit 1 Storage unit 2 Apple Avocado Banana Blueberry SU1 Strawberry-MAX SU2 Lime-Strawberry SU3 Canteloupe-Lime MIN-Canteloupe MIN-Canteloupe SU1 Canteloupe-Lime SU3 Lime-Strawberry SU2 Strawberry-MAX Grapefruit…Pear? Grapefruit…Lime? Lime…Pear? Canteloupe Grape Kiwi Lemon Router Lime Mango Orange Storage unit 1 Storage unit 2 Storage unit 3 Strawberry Tomato Watermelon

Updates SU SU SU Routers Message brokers 1 8 Sequence # for key k Write key k Routers Message brokers 2 Write key k 3 Write key k 7 4 Sequence # for key k 5 SUCCESS SU SU SU 6 Write key k

Asynchronous replication and consistency

Asynchronous replication

Consistency model Goal: make it easier for applications to reason about updates and cope with asynchrony What happens to a record with primary key “Brian”? Record inserted Update Update Update Update Update Update Update Delete v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Time Time Generation 1

Consistency model Read Stale version Stale version Current version Time Generation 1

Consistency model Read up-to-date Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Time Generation 1

Consistency model Read ≥ v.6 Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Time Generation 1

Consistency model Write Stale version Stale version Current version Time Generation 1

Consistency model Write if = v.7 Stale version Stale version ERROR Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Time Generation 1

Consistency model Mechanism: per record mastership Write if = v.7 ERROR Stale version Stale version Current version v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 v. 7 v. 8 Time Generation 1

Experiments

Experimental setup Production PNUTS code Three PNUTS regions Workload Enhanced with ordered table type Three PNUTS regions 2 west coast, 1 east coast 5 storage units, 2 message brokers, 1 router West: Dual 2.8 GHz Xeon, 4GB RAM, 6 disk RAID 5 array East: Quad 2.13 GHz Xeon, 4GB RAM, 1 SATA disk Workload 1200-3600 requests/second 0-50% writes 80% locality

Scalability

Request skew

Size of range scans

Related work Distributed and parallel databases Especially query processing and transactions BigTable, Dynamo, S3, SimpleDB, SQL Server Data Services, Cassandra Distributed filesystems Ceph, Boxwood, Sinfonia Distributed (P2P) hash tables Chord, Pastry, … Database replication Master-slave, epidemic/gossip, synchronous…

Conclusions and ongoing work PNUTS is an interesting research product Research: consistency, performance, fault tolerance, rich functionality Product: make it work, keep it (relatively) simple, learn from experience and real applications Ongoing work Indexes and materialized views Bundled updates Batch query processing

Thanks! cooperb@yahoo-inc.com research.yahoo.com