Benchmarking Cloud Serving Systems with YCSB Presenter: Zhujun Xiao Oct. 25 Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, Russell Sears (Yahoo! Research), SoCC’2010

Motivation (1) New systems for data storage and management “in the cloud” Open source systems: Cassandra, HBase… Cloud services: Microsoft Azure, Amazon web service (S3, SimpleDB) … Private systems: Yahoo!’s PNUTS … Serving systems: provide online read/write access to data. as oppose to batch or analytical systems.

Motivation (2) Hard to choose the appropriate system! However, it is hard to compare their performances Different data models BigTable model in Cassandra and HBase Simple hashtable model in Voldemort Document model in CouchDB Different design decisions (trade-offs) Read or write optimized Synchronous or asynchronous replication Latency or durability: sync writes to disk at write time or not Data partitioning: row-based or column-based storage Evaluted on different workloads These systems only report performance for the “sweet spot” workloads for their system. Hard to choose the appropriate system! 1. Appending all updates to a sequential disk-based log can achieve high write throughput, but is not efficient for reads. 2. Synchronous replication can ensure all the copies are up-to-date, but bring long latency on updates. Asynchronous replication can avoid long latency but risk losing the updates due to failure before it can be replicated. 3. row-based storage can quickly read an entire record, and column-based storage could read a subset of the columns. 4. Sync writes to disk latter can reduce latency, but risk losing the unsynched data if the server crashes. Result: developers have to manually evaluate multiple systems for their target application, which is expensive and time-consuming.

Goal Build a standard benchmark for cloud serving systems Evaluate different systems on different workloads A package of standard workloads A workload generator Evaluate both performance and scalability Future direction: availability and replication

Benchmark Tiers Tier 1: performance Tier 2: scaling Better system = achieve target latency and throughput with fewer server Sizeup: latency versus throughput as throughput increases Tier 2: scaling Scaleup: latency should remain constant when number of servers and amount of data scale proportionally. Elastic speedup: how latency changes as more servers are added. Scaleup: performance (e.g. latency) should remain constant as number of servers, amount of data scale proportionally.

YCSB client Define the dataset and load it into the database Data records A set of operations Execute the operations while measuring performance

Workload (1) A table of data records Basic operations Each with 𝐹 fields. Each record is identified by a primary key. The values of each record is a random string of length 𝐿. E.g. 1000 byte records, 𝐹 =10, 𝐿=100. Basic operations Insert, update, read, scan Update a record by replacing the value of one field Scan records in order, starting at a randomly chosen record key. The number of records to scan is randomly chosen.

Workload (2) One workload is a mix of basic operations with choices of Which operations to perform Which record to read or write How many records to scan Decisions are governed by random distributions Uniform Zipfian Latest Multinomial Zipfian: some records are more popular. More likely to be chosen. Following a Zipfian distribution. Latest: latest records are more likely to be chosen. Multinomial: Give each item a specific probability. The Latest distribution is meant to model applications where recency matters; for example, only recent blog posts or news stories are popular, and the popularity decays quickly.

YCSB client architecture Java program Client threads Load the database Execute the workload Measure the latency and achieved throughput Extensibility Extensible: define new workloads. Extensible: plug in new DB interface. Figure 1. YCSB client architecture

Experimental setup (1) Servers Tested systems Database Six storage servers YCSB client runs on a separate server. Tested systems Cassandra, HBase, PNUTS, and Shared MySQL. Database 120 million 1KB records = 20 GB per sever dual 64-bit quad core 2.5 GHz Intel Xeon CPUs, 8 GB of RAM, 6 disk RAID-10 array and gigabit ethernet

Experimental setup (2) YCSB provides a package of standard workloads.

Workload A: update heavy 50% reads and 50% updates. for all systems, operation la- tency increased as offered throughput increased. Cassandra is optimized for write-heavy workload, achieved best throughput and the lowest latency for reads. HBase had very low latency for updates, since updates are buffered in memory. HBase does not sync log updates to disk, which provides low latency updates and high throughput.

Workload B: Read heavy 95% reads and 5% updates PNUTS and sharded MySQL are now able to provide lower latencies on reads than either Cassandra or HBase. overwrites records when they are updated. reads are fast because a single I/O can retrieve the entire, up-to-date record

Workload E: short ranges Scans of 1-100 records of size 1KB Our sharded MySQL implementation is a hash table and does not sup- port range scans. Cas- sandra performs much worse. Cassandra’s range scan sup- port is relatively new in version 0.5.0, and is not yet heavily optimized Hbase and PNUTS have similar scan performance.

Scalability (1) Scaleup: Vary the number of storage severs from 2 to 12 while varying the data size and request rate proportionally. Cassandra and PNUTS, indicating good elastic scaling properties HBase is that its behavior is erratic for very small clusters

Scalability (2) Elastic speedup: run a read-heavy workload on 2 servers; add a 3rd server, then 4th, then 5th, then 6th. 5 servers to 6 servers this results in a sharp increase in latency, as well as a wide variance in the latency. repartitioning is a known issue with Cassandra This performance degradation results from moving data to the 6th server; regular serving requests compete for disk and network resources with the data repartitioning process, resulting in high and highly variable latency. HBase is able to shift read and write load to the new server, resulting in lower latency also moves data to the new server, resulting in higher latency after the sixth server is added, as well as latency variability. However, PNUTS is able to stabilize more quickly, as its repartitioning scheme is more optimized.

Discussion What is the challenging part of building a benchmark, compared with simply implementing several systems and reporting their performance metrics? All the design decisions would contribute to the final performance. Could YCSB be used for contribution breakdown? Is it reasonable to use the distribution listed in the paper? Rather, does the real request pattern follow these distribution? Maybe in some cases, there is not regular pattern. For example, Hbase has low write latency. Because it performs sequential I/O for updates, and HBase does not sync log updates to disk. Which decision contributes more