1. Hot With an Increasing Chance of Clouds... Data Management Research Forecast: Hot With an Increasing Chance of Clouds... Michael Carey Information Systems Group CS Department UC Irvine

2. Cloud-Related DB Bandwagons MapReduce and Hadoop – Parallel programming for dummies – But now Hive, Pig, Scope, Jaql, …  MapReduce is the new runtime DFSs and HDFS (and CSS) – Scalable, self-managed, Really Big Files – But now BigTable, HBase, …  HDFS (or CSS) is the new file storage Key-value stores – Charter members of the “NoSQL movement” – Includes S3, Dynamo, HBase, Cassandra, …  Key-value stores are the new record managers 1

3. Perhaps Our Brains Are Clouded! Some of the possible symptoms include… – A number of us are focused on tweaking Hadoop – Most of us are compiling our declarative data analysis languages into Hadoop jobs (map/reduce/combine) – Some of us are building “cloud database systems” as a layer on top of “givens” such as HDFS – Some other possible signs of “cloud dementia” include… We keep forgetting where we put our schemas We don’t remember working on parallel DBs in the 80’s We’re about to learn some hard lessons all over again… Thou Shalt Worship DFS & BigTable Thou Shalt MapReduce! 2

4. DB-ers: Let’s Do This Stuff “Right”! In my opinion: – Yes, the OS/DS folks out-scaled us (oops!) – But, we’d be “nuts” to simply build on their current (i.e., first-generation) foundation Let’s bring our DB experience to the party…! – Recall important past architectural lessons I’ll give some quick examples of this… – Plus, it can be “windy” in the stratosphere I’ll give one quick example of this too… 3

5. 4 Current Data Layers HDFS HBase Cloud DBMS? Ordered byte streams Append only Randomly partitioned Sets of records/columns Key-based retrieval Range partitioned Some data model? Some query language?

6. 5……

7. 6

8. 7

9. Let’s Do This Stuff “Right”! (cont.) Identify the lessons, then do it “right” – Open-source S/W on commodity H/W – Non-monolithic software components – Equal opportunity data access (external sources) – Fault-tolerant query execution (for data analysis) – Little pre-planning or DBA-type work required – Tolerant of flexible / nested / absent schemas – Types and declarative languages (duh…! ) 8

10. What If We’d Meant To Do This? What is the “right” basis for analyzing and managing the data of the future? – Storage and data distribution layers? – Query runtime layer (and division of labor)? – …. Let’s explore how to build new information management systems to live in the cloud that… – Scale to thousands of nodes (and beyond) – Make effective use of shared (and elastic) resources – Execute queries in the face of partial failures – Seamlessly support external data access – Don’t require five-star wizard administrators – …. 9

11. In Short… Ask not what cloud software can do for you, but what you can do for cloud software…! – We are, after all, the data experts! – We just have to remember our roots and bring our experiences and past lessons to bear on cloud issues We’re asking this question at UCI (and UCSD/UCR): – ASTERIX: – ASTERIX: A new data platform to ingest, store, manage, index, query, analyze, and publish vast quantities of semi-structured information – Hyracks: – Hyracks: A new foundation for scalable data analysis – both a runtime system for ASTERIX and an extensible parallel data analysis platform in its own right – Open-source sharing planned for both (eventually) 10

12. Semistructured data management – Core work exists – XML & XQuery, JSON, … – Time to parallelize and scale out Parallel database systems – Research quiesced in mid-1990’s – Renewed industrial interest – Time to scale up and de-schema-tize Data-intensive computing – MapReduce and Hadoop quite popular – Language efforts even more popular (Pig, Hive, Jaql, …) – Ripe for parallel DB ideas (e.g., for query processing) and support for stored, indexed data sets The ASTERIX Project Semistructured Data Management Parallel Database Systems Data-Intensive Computing 11

13. ASTERIX Project Overview Disk Main Memory Disk CPU(s) ADM Data Main Memory Disk CPU(s) ADM Data ADM Data Hi-Speed Interconnect Data loads & feeds from external sources (XML, JSON, …) AQL queries & scripting requests and programs Data publishing to external sources and apps ASTERIX Goal: To ingest, digest, persist, index, manage, query, analyze, and publish massive quantities of semistructured information… (ADM = ASTERIX Data Model; AQL = ASTERIX Query Language) Main Memory CPU(s) 12

14. ASTERIX User Model Data model (ADM) with open and closed types Query language (AQL) for nested and semi- structured querying Support for both stored and external datasets for $user in dataset(‘User’) where some $i in $user.interests satisfies $i =“movies” return {“name”: $user.name }; 13

15. Hyracks Runtime Partitioned-parallel platform for data-intensive computing Job = dataflow DAG of operators and connectors – Operators consume/produce partitions of data – Connectors repartition/route data between operators Hyracks vs. the “competition” – Based on time-tested parallel database principles – vs. Hadoop: More flexible model, less “pessimistic”implementation – vs. Dryad: Supports data as a first-class citizen 14

16. ASTERIX Research Issue Sampler Semistructured data modeling – Open/closed types, type evolution, relationships, …. – Efficient physical storage scheme(s) Scalable storage and indexing – Self-managing scalable partitioned datasets – Ditto for indexes (hash, range, spatial, fuzzy; combos) Large scale parallel query processing – Division of labor between compiler and runtime – Query plan decision-making timing and basis AQUA – Model-independent complex object algebra (AQUA) – Fuzzy – Fuzzy matching as well as exact-match queries Multiuser workload management (scheduling) – Uniformly cited: Facebook, Yahoo!, eBay, Teradata, …. 15

17. Closing Thoughts IMO, this is the most exciting time to be a data management researcher in the last 20-30 years! – Data coming out our ears (and other openings) thanks to WWW, Twitter, FaceBook,... (so-called “data exhaust”) – Data models and query languages totally up for grabs (witness the “NoSQL movement”) – System layers and architectures also up for grabs ( ) Many interesting challenges to Cloud our thinking – Opportunities for DB, OS, and PL research to converge – Important to learn from the past as we plan for the future – Resource management at scale is one of the biggies – Consistency models / programming models are another 16