Databases Architectures & Hypertable Doug Judd CEO, Hypertable, Inc.

Database Terminology

Structured, Semi-Structured, and Unstructured Data Structured is what RDBMS store Data is broken into discrete components Types associated with each component: integer, floating point, date, string Unstructured is free-form text Semi-structured is combination of sturctured and semi-structured www.hypertable.org

Document-Oriented Semi-structured documents Accepts documents in a format such as JSON, XML, YAML Often Schema-less Auto-index fields Examples: CouchDB, MongoDB Best Fit: XML or Web documents www.hypertable.org

Graph Databases Database designed to represent graphs APIs for performing graph operations Traversal (depth-first, breadth-first) Shortest/Cheapest path Partitioning Some allow Hypergraphs Examples: Neo4j, HyperGraphDB, InfoGrid, AllegroGraph, Sones, DEX, FlockDB, OrientDB, VertexDB, InfiniteGraph, Filament More info: sones graphdb landscape www.hypertable.org

Column-Oriented Data physically stored by column RDBMS typically row-oriented Improved performance for column operations Better data compression Examples: Hypertable, HBase, Cassandra, Vertica www.hypertable.org

In-Memory Data set stored in RAM Extremely fast access Limited capacity Examples: Memcached, Redis, MonetDB, VoltDB www.hypertable.org

Horizontal Scalability Scale out Increase capacity by adding machines Opposite of vertical scalability (scale up) Commodity Hardware www.hypertable.org

Distributed Hash Table (DHT) Horizontally Scalable Decentralized Fast access Restricted API: GET,SET,DELETE Peer-to-peer file sharing systems: BitTorrent, Napster, Gnutella, Freenet Examples: Dynamo, Cassandra, Riak, Project Voldemort, SimpleDB, S3, Redis, Scalaris, Membase www.hypertable.org

Scalable Database Architectures

Auto-Sharding Splits table data into horizontal “shards” Shards managed by traditional RDBMS (e.g. MySQL, Postgres) Automated “glue” code to handle sharding and request routing Examples: MongoDB, AsterData, Greenplum www.hypertable.org

MongoDB www.hypertable.org

Dynamo Developed by Amazon.com for their Shopping Cart Designed for high write availability Eventually Consistent DHT Implementations: Cassandra Project Voldemort Riak Dynomite www.hypertable.org

Eventual Consistency Database update semantics in a distributed system with data replication Strong Consistency - after an update completes all processes see the updated value Eventual Consistency - eventually all processes will see the updated value Most well-known eventual consistency system is DNS www.hypertable.org

Eventual Consistency www.hypertable.org

Consistent Hashing www.hypertable.org

Amazon AWS S3 SimpleDB RDS Online storage web service Designed for larger amounts of data Cost $0.15/GB per month SimpleDB Designed for smaller amounts of data Provides indexing and richer query capability Cost $027/GB per month + machine utilization fee RDS Managed MySQL instances www.hypertable.org

Order Preserving Partitioner (Cassandra) www.recipezaar.com 1091721999…629750272 + www.ribbonprinters.com 1091721999…965293103 / 2 = www.rgb????i?pQdp?.??? 1091721999…297521687 www.hypertable.org

Order Preserving Partitioner Balance Problem www.hypertable.org

Bigtable: the infrastructure that Google is built on Bigtable underpins 100+ Google services, including: YouTube, Blogger, Google Earth, Google Maps, Orkut, Gmail, Google Analytics, Google Book Search, Google Code, Crawl Database… Implementations Hypertable HBase Describe the 360 degree panoramic view feature of Google Maps www.hypertable.org

Google Stack GFS - Replicates data inter-machine MapReduce - Efficiently process data in GFS Bigtable - Indexed table structure www.hypertable.org

Google File System www.hypertable.org

Google File System www.hypertable.org

System Overview www.hypertable.org

Data Model Sparse, two-dimensional table with cell versions Cells are identified by a 4-part key Row (string) Column Family (byte) Column Qualifier (string) Timestamp (long integer) Spend some time www.hypertable.org

Table: Visual Representation Spend some time. www.hypertable.org

Table: Actual Representation www.hypertable.org

Scaling (part I) www.hypertable.org

Scaling (part II) www.hypertable.org

Scaling (part III) www.hypertable.org

Request Routing www.hypertable.org

Hypertable

Hypertable Overview Massively Scalable Database Modeled after Google’s Bigtable High Performance Implementation (C++) Thrift Interface for all popular High Level Languages: Java, Ruby, Python, PHP, etc Open Source (GPL license) Project started March 2007 @ Zvents www.hypertable.org

Hypertable In Use Today www.hypertable.org

Hypertable vs. HBase www.hypertable.org

Hypertable vs. HBase Test Hypertable Advantage Relative to HBase (%) Random Read Zipfian 80 GB 925 Random Read Zipfian 20 GB 777 Random Read Zipfian 2.5 GB 100 Random Write 10KB values 51 Random Write 1KB values 102 Random Write 100 byte values 427 Random Write 10 byte values 931 Sequential Read 10KB values 1060 Sequential Read 1KB values 68 Sequential Read 100 byte values 129 Scan 10KB values 2 Scan 1KB values 58 Scan 100 byte values 75 Scan 10 byte values 220 www.hypertable.org

Annual EC2 Cost Savings Assuming 200% improvement Extra large reserved instances www.hypertable.org

Resources Project Site Twitter Commercial Support www.hypertable.org Twitter hypertable Commercial Support www.hypertable.com Performance Evaluation Write-up blog.hypertable.com/?p=14 www.hypertable.org

Q&A