1. Transaction Processing on Top of Hadoop Spring 2012 Aviram Rehana Lior Zeno Supervisor : Edward Bortnikov

2. Agenda The problem and motivation Background on databases’ access What is done so far Project goals Effort assessment and roadmap

3. The problem and motivation Combining transactional and non-transactional operations on top of a distributed NoSQL database. A little bit of motivation: A new approach in the distributed databases field. Better performance in comparison to related solutions. Ease of use from a programmer’s perspective.

4. Background on databases Historically, data processing was done with SQL databases, which supported transactions. The computer and internet evolution caused information growth and specifically in databases area there was a need for bigger databases support. Due to the above scalability issues, SQL databases were replaced by Web-scale NoSQL stores in the early 2000s. These technologies dropped txn support though.

5. Background on databases Lately, some NoSQL databases like HBase started supporting txn's. Application designers noticed the potential of transactions in the distributed field and solutions have risen over the years. Our goal is to extend the mechanism to support both types of traffic.

6. What is done so far Existing implementation open-source stack: HDFS – a distributed file system primarily designed for read only access (In use by Facebook and others). HBase – a distributed R/W database on top of HDFS without support for Txns. OMID – transaction processing service on top of HBase designed by Yahoo! Research team. OMID does not guarantee consistency between transactional and non-transactional access to the same data at the same time. We wish to extend OMID to support transactional as well as traditional non-transactional clients.

7. The OMID model

8. An example of mixed access What happens if there are no rules:

9. What is a transaction? A transaction is a collection of operations in an application that fulfill the ACID Properties: Atomicity - Either all or none of the transaction's operations are performed. Consistency - T(x) is a correct transformation of the state. Isolation - Even though txs execute concurrently, it appears to each tx T that others executed either before T or after (serializable). Durability - Results of a committed transaction are permanent, in spite of possible failures.

10. Levels of Isolation Nevertheless, achieving the highest level of isolation (Serializability) is too expensive. It requires locking – less scalability. How do we overcome this issue? Snapshot Isolation - A guarantee that all reads made in a transaction will see a consistent snapshot of the database (in practice it reads the last committed values that existed at the time it started), and the transaction itself will successfully commit only if no updates it has made conflict with any concurrent updates made since that snapshot. transaction SI is less consistent, but it is much more scalable!

11. Project goals Extend OMID work to support both transactional and non-transactional accesses against the distributed database. Implementation of base algorithms in use for the project: Lazy Snapshot Algorithm (LSA) RingSTM Achieve better performance in comparison to OMID and other solutions in regular transactional clients.

12. Roadmap of the project Phase 1: Design RingSTM + LSA components – 2 weeks Integrate the design into OMID – 5 weeks Simulations and benchmarking – 2 weeks Phase 2: Design write conflict against non-transactional clients module – 2 weeks Integrate above design into OMID – 3 weeks Simulations and benchmarking of whole project – 2 weeks Final conclusions – 1 week Future work: Implementing serializability and dynamic separation.

13. Questions?

14. Special thanks: Eshcar Hillel.