1. Lecture 11: Other NoSql Instructor: Weidong Shi (Larry), PhD
COSC6376 Cloud Computing
Lecture 11: Other NoSql
Instructor: Weidong Shi (Larry), PhD
Computer Science Department
University of Houston

2. Outline
Cassandra
Memcache

3. Cassandra

4. What Cassandra is… Cassandra is a massively
scalable, decentralized, structured data store (aka database).
A prophetess in Troy during the Trojan War. Her predictions were
always true, but never believed.

5. Cassandra: Big Table/Dynamo Hybrid
Originally from Facebook
Written by an original Dynamo developer
Now an Apache project
Facebook now uses Hbase
NetFlix is based on Cassandra
Written in Java
Follows the BigTable data model: column-oriented
Uses the Dynamo Eventual Consistency model
Uses Apache Thrift as it’s API
An interface definition language
Define and create services for numerous languages

6. Thrift Created at Facebook along with Cassandra
Is a cross-language, service-generation framework
Binary Protocol (like Google Protocol Buffers)
Compiles to: C++, Java, PHP, Ruby, Erlang, Perl, ...

7. Thrift SOAP XML, XML, and more XML CORBA Over designed and Heavyweight
Sending requests, getting results
Waiting for requests
(known location,
known port)
Communication protocol, Data format
SOAP
XML, XML, and more XML
CORBA
Over designed and Heavyweight
COM
Embraced mainly in Windows Client Software
Pillar
Slick! But no versioning/abstraction.
Protocol Buffers etc
Closed source Google deliciousness

8. Principle Of Operation
Define Data types and Service interfaces
Create a thrift file
eg demo.thrift
Thrift Code Generator Tool
(written in C++)
Build Thrift platform files
Demo.php
Demo.cpp
Demo.py
Demo.java
Server implements Services and Client calls them
Create Server/Client App
Run the Server

9. Projects Using Thrift Cassandra ThriftDB Scribe Hadoop / HBase
Facebook

10. Approaches of influence
BigTable
Sparse map data model
GFS, Chubby, et al
Dynamo
O(1) distributed hash table (DHT)
BASE (aka eventual consistency)
Client tunable consistency/availability
cassandra ~= bigtable + dynamo

11. Design Goals High availability Eventual consistency
trade-off strong consistency in favor of high availability
Incremental scalability
Optimistic Replication
“Knobs” to tune tradeoffs between consistency, durability and latency
Low total cost of ownership
Minimal administration

12. web 2.0 Proven The Facebook stores 150TB of data on 150 nodes
used at Twitter, Rackspace, Mahalo, Reddit, Cloudkick, Cisco, Digg, SimpleGeo, Ooyala, OpenX, others

13. Cassandra Data Model

14. Typical NoSQL API Basic API access:
get(key) -- Extract the value given a key
put(key, value) -- Create or update the value given its key
delete(key) -- Remove the key and its associated value
execute(key, operation, parameters) -- Invoke an operation to the value (given its key) which is a special data structure (e.g. List, Set, Map .... etc).

15. keyspace column family column Data Model name value clock
settings (eg, partitioner)
column family
settings (eg, type [Std])
column
name
value
clock

16. Data Model Keyspace ColumnFamily Column Uppermost namespace
Typically one per application
ColumnFamily
Associates records of a similar kind
Record-level Atomicity
Indexed
Column
Basic unit of storage

17. Keyspace ~= database typically one per application
some settings are configurable only per keyspace

18. Column Column: smallest data element, a tuple with a name and a value
Each column has 3 parts
name
determines sort order
used in queries
Value
timestamp
long (clock)
Here’s a column represented in JSON-ish notation:
{ // this is a column
name: " Address",
value:
timestamp: }

19. Column Family Group records of similar kind
Not same kind, because CFs are sparse tables
Example:
UserProfile = { // this is a ColumnFamily
phatduckk: { // this is the key to this Row inside the CF
// now we have an infinite # of columns in this row
username: "phatduckk",
phone: "(900) " },
// end row
ieure: { // this is the key to another row in the CF
// now we have another infinite # of columns in this row
username: "ieure",
phone: "(888) "
age: "66",
gender: "undecided" }, }

20. nickname=The Situation
Column Family
key123
user=eben
nickname=The Situation
key456
user=alison
icon=
n= 42
Think of it as hashmap or associative array
each row is uniquely identifiable by key

21. Super Column super columns group columns under a common name
A SuperColumn is a tuple with a name & a value which is a map containing an unbounded number of Columns (a map of columns)

22. Super Column { // this is a SuperColumn name: "homeAddress",
// with an infinite list of Columns
value: {
// note the keys is the name of the Column street: {name: "street", value: "1234 x street", timestamp: },
city: {name: "city", value: "san francisco", timestamp: },
zip: {name: "zip", value: "94107", timestamp: }, } }

23. Super Column Family A column family can be of type standard or super
Standard column family:
all the Rows contains a map of normal columns
Super column family:
each Row contains a map of super columns

24. super column family column key
AddressBook = { // this is a ColumnFamily of type Super phatduckk: { // this is the key to this row inside the Super CF // the key here is the name of the owner of the address book // now we have an infinite # of super columns in this row // the keys inside the row are the names for the SuperColumns // each of these SuperColumns is an address book entry friend1: {street: "8th street", zip: "90210", city: "Beverley Hills", state: "CA"}, // this is the address book entry for John in phatduckk's address book John: {street: "Howard street", zip: "94404", city: "FC", state: "CA"}, Kim: {street: "X street", zip: "87876", city: "Balls", state: "VA"}, Tod: {street: "Jerry street", zip: "54556", city: "Cartoon", state: "CO"}, Bob: {street: "Q Blvd", zip: "24252", city: "Nowhere", state: "MN"}, ... // we can have an infinite # of ScuperColumns (aka address book entries) }, // end row ieure: { // this is the key to another row in the Super CF // all the address book entries for ieure joey: {street: "A ave", zip: "55485", city: "Hell", state: "NV"}, William: {street: "Armpit Dr", zip: "93301", city: "Bakersfield", state: "CA"}, }, }
column
key

25. Datamodel explained by example (Twitter)

26. Example - Twitter

27. Example - Twitter
Supercolumn family

28. Write Operations
A client issues a write request to a random node in the Cassandra cluster.
The “Partitioner” determines the nodes responsible for the data.
Locally, write operations are logged and then applied to an in-memory version.
Commit log is stored on a dedicated disk local to the machine.

29. Write Operations No locks in the critical path No reads No seeks
Memtable
No locks in the critical path
No reads
No seeks
Append support
Fast
Sequential disk access
Atomic within a column family
≈ 0.2 ms
Commit
log
Threshold
Write
SSTable
SSTable

30. D E L E T E D Compaction MERGE SORT K2 < Serialized data >--
K4 < Serialized data >
K5 < Serialized data >
K10 < Serialized data > --
K1 < Serialized data >
K2 < Serialized data >
K3 < Serialized data > --
Sorted
Sorted
Sorted
MERGE SORT
Index File
K1 < Serialized data >
K2 < Serialized data >
K3 < Serialized data >
K4 < Serialized data >
K5 < Serialized data >
K10 < Serialized data >
K30 < Serialized data >
Loaded in memory
K1 Offset
K5 Offset
K30 Offset
Bloom Filter
Sorted
Data File

31. Reads
Memtable
Bloomfilter field to determine whether a provided key is in the SSTable
Index field for quick read
Any node
Read repair
≈ 15 ms
Read Bf
Idx Bf
Idx
SSTable
SSTable

32. Read repair if digests differ
Read Operations
Client
Query
Result
Cassandra Cluster
Read repair if digests differ
Closest replica
Result
Replica A
Digest Query
Digest Response
Digest Response
Replica B
Replica C

33. Cassandra and Consistency
Talked previous about eventual consistency
Cassandra has programmable read/writable consistency
One: Return from the first node that responds
Quorom: Query from all nodes and respond with the one that has latest timestamp once a majority of nodes responded
All: Query from all nodes and respond with the one that has latest timestamp once all nodes responded. An unresponsive node will fail the read

34. Cassandra and Consistency
Zero: Ensure nothing. Asynchronous write done in background
Any: Ensure that the write is written to at least 1 node
One: Ensure that the write is written to at least 1 node’s commit log and memory table before receipt to client
Quorom: Ensure that the write goes to node/2 + 1
All: Ensure that writes go to all nodes. An unresponsive node would fail the write

35. Architecture

36. Tombstones
“soft delete.” Instead of actually executing a delete SQL statement, the application will issue an update statement that changes a value in a column called something like “deleted”.
In Cassandra, it is called a tombstone.
When you execute a delete operation, the data is not immediately deleted. Instead, it’s treated as an update operation that places a tombstone on the value.
A tombstone is a deletion marker that is required to suppress older data in SSTables until compaction can run.

37. Hinted Handoff An optimization technique for data write on replicas
When a write is made and a replica node for the key is down
Cassandra writes a hint to a live replica node
That replica node will remind the downed node of changes once it is back on line
Hinted Handoff reduce write latency when a replica is temporarily down
Hinted Handoff provides high write availability at the cost of consistency
A hinted write does NOT count towards Consistency Level requirements for ONE, QUORUM, or ALL

38. MySQL Comparison
MySQL > 50 GB Data Writes Average : ~300 ms Reads Average : ~350 ms
Cassandra > 50 GB Data Writes Average : 0.12 ms Reads Average : 15 ms

39. Lessons Learnt Add fancy features only when absolutely required.
Many types of failures are possible.
Big systems need proper systems-level monitoring.
Value simple designs

40. Memcache

41. Memcache Memcache is not a database.
Memcache is a distributed cache system.
Memcache is not meant for providing any backup support. Its all about simple read and write.
Memcache is very fast.

42. Memcache users LiveJournal Wikipedia Flickr Twitter Youtube Dig
Wordpress
Craigslist
Facebook (around 200 dedicated memcache servers)‏

43. Memcache
Memcache is an in-memory key-value store for small chunks of arbitrary data (strings, objects)
> in-memory (volatile) key-value store
$memcache->set('unique_key', $value, $flag, $expiration_time);
$flag = 0 / MEMCACHE_COMPRESSED to store the item
compressed.
$expiration_time = 0 (never expire) / 30 (30 seconds) etc.
$memcache->get('unique_key');
NOTE: Missing key makes fetch time doubles.
> distributed memory caching system (you can use more than one server to cache your data)‏
$memcache->addServer('host1', 11211);
$memcache->addServer('host2', 11211);
$memcache->addServer('host3', 11211);

44. What can you store in Memcache?
Results of database calls, API calls (xml as string), page rendering (html as string) etc.
NOTE: Objects are serialized before being stored to memcache

45. Caching

46. Caching Use memcache What can be cached? What are the benefits?
In-memory key-value store
Distributed
What can be cached?
Common queries, results of database calls
Page rendering (html as string)
Sessions
What are the benefits?
Decreases load on DB
Faster response than DB

47. Caching

48. Additional NoSQL DBs

49. What else? MongoDB Voldemort Riak / Basho CouchDB Hibari Virtuoso
Many many others!

50. MongoDB Document–oriented All writes and reads are through the master
Documents stored as JSON objects
All writes and reads are through the master
Written in C++
Native Python bindings
Simple configuration

51. MemcacheDB MemcacheD with persistence Uses Memcache API
Uses Berkelely DB
Master/Slave
Read from any slave
Write only to the master

52. HyperTable open source Inspired Google's BigTable
runs on top of a distributed file system such as the Apache Hadoop DFS, GlusterFS, or the Kosmos File System (KFS)
written almost entirely in C++
Developed in-house at Zvents Inc

53. Voldemort Dynamo clone by LinkedIn Eventually consistent
Multiple versions may be returned on a Get
Uses Berkeley DB for persistence
Thrift interface
Written in java

54. Datastores
Replication
Consistency
CAP
Data Model
Range Queries
Cassandra
Yes
Eventual AP
Column oriented
Hbase
Strong CP
Hypertable
MemcacheDB
Key/Value
MongoDB
Document
MySQL
Relational
Voldemort No
Eventual Consistency: Apps can see inconsistent data if they are not careful about choice of R and W
Might not see its own writes or successive reads might see a row’s state jump back and forth in time

55. List of NoSQL databases [122+]
Wide Column Store / Column Families
HBase, Cassandra, Hypertable, Cloudata, Cloudera, Amazon SimpleDB
Document Stores
CouchDB, MongoDB, Terrastore, ThruDB, OrientDB, RavenDB, Citrusleaf, SisoDB
Key Value / Tuple Store
Azure Table Storage, MEMBASE, Riak, Redis, Chordless, GenieDB, Scalaris, Tokyo Cabinet / Tyrant, Keyspace
Berkeley DB, MemcacheDB, Faircom C-Tree, Mnesia, LightCloud, Hibari, HamsterDB, STSdb, Pincaster, RaptorDB
Eventually Consistent Key Value Stores
Amazon Dynamo, Voldemort, Dynomite, KAI
Graph Databases
Neo4J, Infinite Graph, Sones, InfoGrid, HyperGraphDB, Trinity, AllegroGraph, Bigdata, DEX, OpenLink, Virtuoso, VertexDB, FlockDB
Object Databases
db4o, Versant, Objectivity, Gemstone, Progress, Starcounter, Perst, Caching, ZODB, NEO, PicoLisp, Sterling
More and more databases