1. Jason Baker, Chris Bond, James C. Corbett, JJ Furman, Andrey Khorlin, James Larson,Jean Michel L´eon, Yawei Li, Alexander Lloyd, Vadim Yushprakh Megastore Scalable Highly Available Storage for Interactive Systems Presented By: Hamid Seyedmoradi Ayoub Hamidi Ehsan Mohamad Nezamian Advanced Database Systems SRBIAU, Kurdistan Campus 10May2012

2. Megastore  Motivation  Introduction  NoSQL & RDBMS  Megastore  Paxos SRBIAU, Kurdistan Campus Advanced Database Systems2

3. Megastore wow! more than 3 billion write and 20 billion read daily  key Contribution  Data Model and Storage System  Paxos Replication  Report on Experience SRBIAU, Kurdistan Campus Advanced Database Systems3

4. AVAILABILITY & SCALE  Replication For Availability, we implemented a synchronous, fault tolerant log replicator optimized for long distance links  Partitioning and Locality for scale, we partitioned data into a vast space of small databases SRBIAU, Kurdistan Campus Advanced Database Systems4

5. AVAILABILITY & SCALE  Replication  Strategies  Asynchronous Master/Slave  Synchronous Master/Slave  Optimistic Replication We decided to use Paxos SRBIAU, Kurdistan Campus Advanced Database Systems5

6. Technology Options SRBIAU, Kurdistan Campus Advanced Database Systems 6

7. 7 Technology Options

8. AVAILABILITY & SCALE  Partitioning and Locality  Replication SRBIAU, Kurdistan Campus Advanced Database Systems8 Datacenters Entity Groups Partition the datastore Each entity group is synchronously replicated across datacenters ACID semantics within an entity group Looser consistency across entity groups Entity group data and replication metadata stored in scalable NoSQL datastores

9. AVAILABILITY & SCALE  Partitioning and Locality  Operations: SRBIAU, Kurdistan Campus Advanced Database Systems9 Entities (Units of data) Entity Group 1 Most transactions are within a single entity group Cross Entity group transactions supported via Two – Phase Commit Asynch communication between entity groups supported by Queues Global Indexes span entity groups but have weaker consistency Entity Group 2 Local Index Send queue receive

10. AVAILABILITY & SCALE  Partitioning and Locality  Entity Groups  Selecting Entity Group Boundaries  Example Email Blogs  Physical Layout SRBIAU, Kurdistan Campus Advanced Database Systems10

11. Megastore  API Design Philosophy  Data Model  Pre-Joining with Keys  SCATTER  Indexes  Storing Clause  Repeated Indexes.  Inline Indexes  Mapping to Bigtable SRBIAU, Kurdistan Campus Advanced Database Systems11

12. Megastore SRBIAU, Kurdistan Campus Advanced Database Systems12

13. Megastore  Transactions and Concurrency Control  Read  current  snapshot  inconsistent  Transaction Lifecycle 1-Read 3-Commit 5-Clean up 2-Application logic 4-Apply SRBIAU, Kurdistan Campus Advanced Database Systems13

14. Megastore  Queues  Two Phase Commit SRBIAU, Kurdistan Campus Advanced Database Systems14

15. REPLICATION  Brief Summary of Paxos  Megastore’s Approach  Fast Reads  Fast Writes  Replica Types  Witness Replica  Architecture SRBIAU, Kurdistan Campus Advanced Database Systems15

16. Architecture SRBIAU, Kurdistan Campus Advanced Database Systems16

17. Data Structures and Algorithms  Replicated Logs SRBIAU, Kurdistan Campus Advanced Database Systems17

18. Data Structures and Algorithms  Reads  Query Local  Find Position  Local read  Majority read  Catchup  Validate  Query Data SRBIAU, Kurdistan Campus Advanced Database Systems18

19. Data Structures and Algorithms SRBIAU, Kurdistan Campus Advanced Database Systems19

20. Data Structures and Algorithms  Writes  Accept Leader  Prepare  Accept  Invalidate  Apply SRBIAU, Kurdistan Campus Advanced Database Systems20

21. Feedback SRBIAU, Kurdistan Campus Advanced Database Systems21

22. END With Thanks Question ? SRBIAU, Kurdistan Campus Advanced Database Systems22