Big Data Analytics and Data Warehousing with Data Cubes Carlos Ordonez University of Houston ATT Research Labs NY

Goals of talk 1.Big Data 2.Cubes 3.Highlight some of my “cubic” research 2/79

Big Data: what is different from large databases? VVV+V Variety: –Loosely specified or no schema –Storage: Record -> files; data types -> any digital content Volume: –Higher volume, including streaming –Multiple levels of granularity Velocity: speed of arrival/processing Veracity: Internet, multiple versions of data 3/79

Big Data: specific technical details Integration needed! Finer granularity than transactions; give up ACID? cannot be directly analyzed: pre-processing Diverse data sources, beyond relational/alphanumeric Volume requires parallelism Skip ETL: load files directly Web logs, user interactions, social nets, streams Still only HDD provides capacity and good $; SSD $$; future non-volatile RAM? 4/79

Current technologies for Big Data DBMS –Row –Column –Other: array, XML, Datalog is back Hadoop stack –Apache: + important than GNU in corp. world –MapReduce: forgotten?; HDFS: dominant –Hive; SPARQL; Cassandra, Impala, Cask –Many more: open-source 5/79

Data Warehouse versus Data Lake (Data Swamp?) 6/79 FeatureData WarehouseData Lake Database ModelER modelNone ETLInvolved, data transformation Copy file QueryingSQLSparql, Java program, SQL? Grow # nodesDifficult, $$$Easy, $$

Big Data Analytics Processing Data Profiling Data Exploration; univariate stats Data Preparation Multivariate Statistics Machine Learning Algorithms Analytic Modeling Scoring Lifecycle Maintenance Model Deployment Highly Iterative Process

Processing: DBMS versus Hadoop TaskSQLHadoop /noSQL Available Sequential open-sourceyy Parallel open sourceny Fault tolerant on long jobsnY LibrarieslimitedMany Arrays and matriceslimitedgood Massive parallelism (# servers, 1000s of CPUs)ny 8/60

Some cons about big data not using DBMS technology No SQL No model, no DDL no consistency, although transaction too stringent Web-scale data tough, but not universal Database integration and cleaning much harder Parallel processing with too much hardware Fact: SQL remains main query mechanism 9/79

Why analysis inside a DBMS? llllllll Teradata Your PC with Warehouse Miner ODBC Huge data volumes: potentially better results with larger amounts of data; less process. time Minimizes data redundancy; Eliminate proprietary data structures; simplifies data management; security Caveats: SQL, limited statistical functionality, complex DBMS architecture

DBMS Sequential vs Parallel Physical Operators Serial DBMS (one CPU, RAID): –table Scan –join: hash join, sort merge join, nested loop –external merge sort Parallel DBMS (shared-nothing): –even row distribution, hashing –parallel table scan –parallel joins: large/large (sort-merge, hash); large/short (replicate short) –distributed sort 11/60

Big Data Analytics Overview Simple: –Ad-hoc Queries –Cubes: OLAP, MOLAP, includes descriptive statistics: histograms, means, plots, statistical tests Complex: –Statistical and Machine Learning Models –Patterns: Graphs subsuming other problems 12/60

Cube Processing Input Data set F : n records, d dimensions, e measures Dimensions: discrete, measures: numeric Focus of the talk, d dimensions I/O bottleneck: Cube: lattice of d dimensions High d harder than n 13/60

Cube computations Explore lattice of dimensions Large n: F cannot fit in RAM, minimize I/O Multidimensional –d: tens, maybe hundreds of dimensions Internally computed with data structures 14/60

Cube algorithms: elevator story Behavior with respect to data set X: –Level-wise: k passes Time complexity bottleneck d: O(n2 d ) Cubes research today: –Parallel processing –Data structures incompatible with relational DB –different time complexity in SQL/Hadoop/MapReduce –Incremental and online 15/60

Cubes inside DBMS: more involved Assumption: –data records are in the DBMS; exporting slow –row-based or column-based storage Programming alternatives: –SQL and UDFs: SQL code generation (JDBC), precompiled UDFs. Extra: SP, embedded SQL, cursors –Internal C Code (direct access to file system and mem) DBMS advantages: –Columns 10X faster: compression + efficient projection –mportant: storage, queries, security –maybe: recovery, concurrency control, integrity, transactions (i.e. some ACID ok) 16/60

Cubes outside DBMS: alternatives Hadoop: dump data to Data Lake; SQL-like later MOLAP tools: –Push hard aggregations with SQL –Memory-based lattice traversal –Interaction with spreadsheets Imperative programming languages instead of SQL: C++, Java –Arrays, functions, modules, classes –flexibility of control statements 17/60

Cube Processing Optimizations Algorithmic & Systems Algorithmic (90% research, but not in a DBMS) –accelerate/reduce cube computations –database systems focus: reduce I/O passes –approximate solutions: good for count(*), sum() looked at with suspicion –parallel Systems (SQL, Hadoop, MapReduce, Libraries) –Platform: parallel DBMS server vs cluster of computers vs multicore CPUs –Programming: SQL/C++ versus Java 18/60

Research Highlights research with my students Comprehensive –Modeling –Query processing –Visualization Biased –Motivated by DOLAP! –Influenced by Stonebraker –Mostly with my students –Hadoop ignored 19/79

A glimpse Preparing and cleaning data takes time: ETL Lots of SQL and scripts written to prepare data sets for statistical analysis Data quality was hot; worth revisiting w/big data Graph analytics Cube computation is the most researched topic; cube result analysis/interpretation 2 nd priority Is “Big data” different? 20/79

SQL and ER: can they get closer? Goal: creating a data set X with d dimensions D(K,A), K commonly a single id Lots of SQL queries, many temporary tables Users do not like to look at someone else’s SQL code Decoupled from ER model, not reused Many transformations: cubes, variable creation, even math transformation for statistical analysis 21/79

Representing Data Transformations done with SQL queries 22/79

SQL transformations in ER 23/79

Extended ER zoom in 24/79

Referential Integrity QMs 25/79

SQL Optimizations: Queries vs UDFs SQL query optimization –mathematical equations as queries –Turing-complete: SQL code generation and programming language UDFs as optimization –substitute difficult/slow math computations –push processing into RAM memory 26/60

SQL Query Processing Columns will take over rows [Stonebraker] –Vertica and MonetDB “pure” column –Hybrid: Oracle Exadata, Teradata, SQL Server indexes But a lot of work to do –OLTP: rows, not columns (slow conversion) –still a lot of data warehouses working in row form –Many external tools store by row 27/79

Horizontal aggregations Create cross-tabular tables from cube PIVOT requires knowing values Aggregations in horizontal layout 28/79

Prepare Data Set Horizontal aggregations

Horizontal Meta-optimizer 30/79

Graph Analytics Recursive queries in SQL Patterns: paths, cycles, cliques Examples: –Twitter: who follows who?, how many #? –Facebook: family, close friends, social circles, friends in common –Airline: list all flights from A to B; balance cost/distance Surprisingly: SQL is good!, but with a column DBMS 31/79

A Benchmark to compute # of paths in a graph of length k 32/79

Cube computation with UDF (table function) Data structure in RAM; maybe one pass It requires maximal cuboid or choosing k dimensions 33/79

Cube in UDF Lattice manipulated with hash table 34/79

Cube visualization: harder than 2D or 3D data! Lattice exploration Projection into 2D Comparing cuboids 35/79

Cube interpretation & visualization statistical tests on cubes

Can we do “search engines”? Keyword search, ranking

Acknowledgments Il-Yeol Song, since we met in 2010, but I started sending papers to DOLAP in 2003 Mike Stonebraker: one size does not fit all My students 38/79