Better Performance for Big Data Shuya Zhang; Shyam Sundar Somasundaram [10/03/13] 1 [1] Bhasker Allene, Marco Righini, “Better Performance for Big Data” Intel White Paper, 2013 Intel Corporation. Reference

What is Hadoop  Apache Hadoop an open-source software framework  Supports data-intensive distributed applications  Enables the running of applications on large clusters of commodity Hardware  Derived from Google's MapReduce and Google File System (GFS) papers 2 Hadoop is one of the poster children of Big Data, especially the most beloved one!

The Intel Distribution for Hadoop framework ● Oozie is a workflow scheduler ● Hive enables SQL queries on Hadoop ● Hbase is a non-relational, distributed database

Oozie Workflow  Step1 Convert EBCDIC to ASCII  Step 2 Scan for New Columns  Step3 Move Columns List to Metadata  Step4 Optimize Data for Hive  Step5 Move Data to Hive Warehouse  Step6 Drop and Create Hive Table Structure

Data Flow & Data Optimization ● Benefits: - Data is stored in a normalized way - Hive queries quite similar to RDBMS queries - The learning curve is minimized, with fewer computations and less disk space - Data is easily consumable for data analysis tools

Comparing SQL and Hive Queries Query 1 RDBMS: select distinct tp_ndg as N010 from scontrpf50m Hive: SELECT DISTINCT N010 FROM O0A011 LIMIT10; Query 2 RDBMS: SELECT DISTINCT B. COD_UO, b. Tp_conto, a. Tp_ndg as N010 FROM A JOIN SCONTRPF50M CUBOM0100M B ON A. NDG = B. NDG WHERE POSIZ_SOFF_INCAGLT = '1 'and a. TP_NDG in ('DIN', 'IOC', 'SPF') and b. dt_accs_rapprt> = ' 'and b. dt_accs_rapprt <= ' ' ORDER BY B. COD_UO, b. Tp_conto, a. TP_NDG HIVE: SELECT DISTINCT B.DOOR, B.TP_INCOME, A.N010 FROM O0A011 A JOIN CUBOM0100M B ON A.NDG = B.NDG WHERE N011 ='1' AND A.N010 in ('DIN', 'IOC', 'SPF') AND B.R021 > '130100' AND B.R021 < '130316'; ORDER BY DOOR, TP_INCOME, N010; Query 3 RDBMS: select b. cod_uo, b. forma_tec, TP_NDG as N010, substring (a. sae_rae, 1, 3) as N003, a. u_segmgest_2004 as N088, a. u_modserv_gest as N089, sum (b. qc_rata_scd) as D505 from scon- trpf50m to join cubom0100m b on a. ndg = b. ndg where a. TP_NDG in ('DIN', 'IOC', 'SPF') and b. forma_tec in ('MW500', 'MW100', 'MW200') group by b. cod_uo, b. forma_tec, a. TP_NDG, substring (a. sae_rae, 1, 3), a. u_segmgest_2004, a. u_modserv_gest order by cod_uo, forma_tec, TP_NDG, sub- string (a. sae_rae, 1, 3), a. u_seg- mgest_2004, a. u_modserv_gest

What is related with our course  Hadoop VS SQL  New Trend: Big Data, Cloud Computing RDBMS (Relational DBMS) OLAP NoSQL Database Management Systems TablesCubesCollections Structured Data Structured Data Structured/ Unstructured

Questions?