Jeremy Shafer Temple University NoSQL & Hadoop Jeremy Shafer Temple University

What is NoSQL? Before we can answer that question…. We need to know something about A.C.I.D. transactions The ideas behind A.C.I.D. transacitons were fomalized in the 1970s, and the acronym A.C.I.D. was coined in 1983.

Atomic - requires that each transaction be "all or nothing" A.C.I.D. transactions Atomic - requires that each transaction be "all or nothing" Consistent - ensures that any transaction will bring the database from one valid state to another Isolation - concurrent execution of transactions result in a system state that would be obtained if transactions were executed serially. Durable - once a transaction has been committed, it will remain so, even in the event of power loss, crashes or errors. Imagine a system designed to send money from one bank to another … how important would each of these be?

CAP Theorem

NoSQL challenges these assumptions In the 70s, that seemed like a bad idea. In the 80s, that seemed like a bad idea. In the 90s, that seemed like a bad idea. But in the late 2000s – NoSQL started to gain notable traction in the IT community. Why?

Thanks, Gartner research!! Big Data Velocity Variety Volume Thanks, Gartner research!!

So… what is NoSQL? NoSQL - an umbrella term for a loosely defined class of non-relational data stores Started to emerge in 2009 No predefined schema and/or tolerant of schema changes IT / Social Media companies needed techniques for dealing with large volumes of distributed data.

What is NoSQL? (Continued) The data structures used by NoSQL databases (meant to hold multiple, complex values) differ from those used in relational databases. This makes some operations faster in NoSQL and some faster in relational databases. Consider NoSQL technologies if you are: willing to accept the possibility of data loss for the sake of speed/performance dealing with tons of user generated content in varied formats willing to accept the possibility of some inconsitency in exchange for a system that can scale up to hundreds of thousands of users NoSQL is important, but also a bit over-hyped Watch what is NoSQL and Exploring possibilities video For a laugh, see: https://www.youtube.com/watch?v=b2F-DItXtZs (Warning: Not family-friendly language!)

Contenders MongoDB (Open Source, MongoDB Inc.) CouchDB (Open Source, Apache Software Foundation) Cassandra (Open Source, Facebook) Hadoop HIVE (Open Source, Apache Software Foundation) None of these technologies have “easy” user interfaces. They are, however, advanced… cutting edge. Being willing to sacrifice some ease-of-use for the sake of competitive advantage is an important business trade off! Watch CAP theorem video clip

Why I like HIVE Redundancy built on HDFS and the general approach makes sense (to me) Leverages knowledge of the SQL syntax This is a “Not Only” SQL solution

A little experiment

Hey… I thought this was supposed to be fast?

Where to from here? In the future, organizations will use many data technologies. Data professionals will need to be familiar with these different approaches and know how to match them to different problems. Learning the concepts is an important first step, but to really understand you’ll need to get experiment with these new technologies, even if only on a small scale.

If I were you… Learn some Linux: http://www.ee.surrey.ac.uk/Teaching/Unix/ Watch and work through the “Up and Running with NoSQL databases” tutorial at http://lynda.temple.edu Explore Hortonworks.com tutorials: http://hortonworks.com/tutorials If you are interested in HIVE see: http://hortonworks.com/blog/hive-cheat-sheet-for-sql-users/ If you are going to explore MongoDb or CouchDB, learn some Javascript. See Javascript Essential Training at http://lynda.temple.edu and take MIS3502! Let me know if you think there should be a “big data” course by emailing me at jeremy@temple.edu