1. Hadoop/Hive/PIG Open-Source Solutions for Huge Data Sets

2. Data Scalability Problems Search Engine 10KB / doc * 20B docs = 200TB Reindex every 30 days: 200TB/30days = 6 TB/day Log Processing / Data Warehousing 0.5KB/events * 3B pageview events/day = 1.5TB/day 100M users * 5 events * 100 feed/event * 0.1KB/feed = 5TB/day Multipliers: 3 copies of data, 3-10 passes of raw data Processing Speed (Single Machine) 2-20MB/second * 100K seconds/day = 0.2-2 TB/day

3. Google’s Solution Google File System – SOSP’2003 Map-Reduce – OSDI’2004 Sawzall – Scientific Programming Journal’2005 Big Table – OSDI’2006 Chubby – OSDI’2006

4. Open Source World’s Solution Google File System – Hadoop Distributed FS Map-Reduce – Hadoop Map-Reduce Sawzall – Pig, Hive, JAQL Big Table – Hadoop HBase, Cassandra Chubby – Zookeeper

5. Simplified Search Engine Architecture Spider Runtime Batch Processing System on top of Hadoop SE Web Server Search Log Storage Internet

6. Simplified Data Warehouse Architecture Database Batch Processing System on top fo Hadoop Web Server View/Click/Events Log Storage Business Intelligence Domain Knowledge

7. Hadoop History Jan 2006 – Doug Cutting joins Yahoo Feb 2006 – Hadoop splits out of Nutch and Yahoo starts using it. Dec 2006 – Yahoo creating 100-node Webmap with Hadoop Apr 2007 – Yahoo on 1000-node cluster Jan 2008 – Hadoop made a top-level Apache project Dec 2007 – Yahoo creating 1000-node Webmap with Hadoop Sep 2008 – Hive added to Hadoop as a contrib project

8. Hadoop Introduction Open Source Apache Project http://hadoop.apache.org/ Book: http://oreilly.com/catalog/9780596521998/index.html Written in Java Does work with other languages Runs on Linux, Windows and more Commodity hardware with high failure rate

9. Current Status of Hadoop Facebook Cluster > 2000 nodes (8 cores each), 30 PB disk Used by companies / universities over the world Yahoo, Facebook, etc Cloud Computing Donation from Google and IBM Companies focusing solely on providing services for hadoop Cloudera

10. Example Applications and Organizations using Hadoop A9.com – Amazon: To build Amazon's product search indices; process millions of sessions daily for analytics, using both the Java and streaming APIs; clusters vary from 1 to 100 nodes. A9.com Yahoo! : More than 100,000 CPUs in ~20,000 computers running Hadoop; biggest cluster: 2000 nodes (2*4cpu boxes with 4TB disk each); used to support research for Ad Systems and Web Search Yahoo! AOL : Used for a variety of things ranging from statistics generation to running advanced algorithms for doing behavioral analysis and targeting; cluster size is 50 machines, Intel Xeon, dual processors, dual core, each with 16GB Ram and 800 GB hard-disk giving us a total of 37 TB HDFS capacity. AOL Facebook: To store copies of internal log and dimension data sources and use it as a source for reporting/analytics and machine learning; > 2000 machines with 8 cores each and over 30TB HDFS Facebook FOX Interactive Media : 3 X 20 machine cluster (8 cores/machine, 2TB/machine storage) ; 10 machine cluster (8 cores/machine, 1TB/machine storage); Used for log analysis, data mining and machine learning FOX Interactive Media University of Nebraska Lincoln: one medium-sized Hadoop cluster (200TB) to store and serve physics data; 10 Sathya Sai University, Prashanti Nilayam

11. Hadoop Components Hadoop Distributed File System (HDFS) Hadoop Map-Reduce Contributes Hadoop Streaming Pig / JAQL / Hive HBase

12. Hadoop Distributed File System

13. Goals of HDFS Very Large Distributed File System 10K nodes, 100 million files, 10 PB Convenient Cluster Management Load balancing Node failures Cluster expansion Optimized for Batch Processing Allow move computation to data Maximize throughput

14. HDFS Details Data Coherency Write-once-read-many access model Client can only append to existing files Files are broken up into blocks Typically 128 MB block size Each block replicated on multiple DataNodes Intelligent Client Client can find location of blocks Client accesses data directly from DataNode

16. HDFS User Interface Java API Command Line hadoop fs -mkdir /foodir hadoop fs -cat /foodir/myfile.txt hadoop fs -rm /foodir myfile.txt hadoop fs –copyFromLocal hadoop dfsadmin -report hadoop dfsadmin -decommission datanodename Web Interface http://host:port/dfshealth.jsp

17. Hadoop Map-Reduce and Hadoop Streaming

18. Hadoop Map-Reduce Introduction Map/Reduce works like a parallel Unix pipeline: cat input | grep | sort | uniq -c | cat > output Input | Map | Shuffle & Sort | Reduce | Output Framework does inter-node communication Failure recovery, consistency etc Load balancing, scalability etc Fits a lot of batch processing applications Log processing Web index building

20. Machine 2 Machine 1 (Simplified) Map Reduce Review Local Map Global Shuffle Local Sort Local Reduce

21. Physical Flow

22. Example Code

23. Hadoop Streaming Allow to write Map and Reduce functions in any languages Hadoop Map/Reduce only accepts Java Example: Word Count hadoop streaming -input /user/zshao/articles -mapper ‘tr “ ” “\n”’ -reducer ‘uniq -c‘ -output /user/zshao/ -numReduceTasks 32

24. Hive - SQL on top of Hadoop

25. Map-Reduce and SQL Map-Reduce is scalable SQL has a huge user base SQL is easy to code Solution: Combine SQL and Map-Reduce Hive on top of Hadoop (open source) Aster Data (proprietary) Green Plum (proprietary)

26. Hive A database/data warehouse on top of Hadoop Rich data types (structs, lists and maps) Efficient implementations of SQL filters, joins and group- by’s on top of map reduce Allow users to access Hive data without using Hive Link: http://svn.apache.org/repos/asf/hadoop/hive/trunk/

27. Dealing with Structured Data Type system Primitive types Recursively build up using Composition/Maps/Lists Generic (De)Serialization Interface (SerDe) To recursively list schema To recursively access fields within a row object Serialization families implement interface Thrift DDL based SerDe Delimited text based SerDe You can write your own SerDe Schema Evolution

28. MetaStore Stores Table/Partition properties: Table schema and SerDe library Table Location on HDFS Logical Partitioning keys and types Other information Thrift API Current clients in Php (Web Interface), Python (old CLI), Java (Query Engine and CLI), Perl (Tests) Metadata can be stored as text files or even in a SQL backend

29. Hive CLI DDL: create table/drop table/rename table alter table add column Browsing: show tables describe table cat table Loading Data Queries

30. Web UI for Hive MetaStore UI: Browse and navigate all tables in the system Comment on each table and each column Also captures data dependencies HiPal: Interactively construct SQL queries by mouse clicks Support projection, filtering, group by and joining Also support

31. Hive Query Language Philosophy SQL Map-Reduce with custom scripts (hadoop streaming) Query Operators Projections Equi-joins Group by Sampling Order By

32. Hive QL – Custom Map/Reduce Scripts Extended SQL: FROM ( FROM pv_users MAP pv_users.userid, pv_users.date USING 'map_script' AS (dt, uid) CLUSTER BY dt) map INSERT INTO TABLE pv_users_reduced REDUCE map.dt, map.uid USING 'reduce_script' AS (date, count); Map-Reduce: similar to hadoop streaming

33. Hive Architecture HDFS Hive CLI DDL Queries Browsing Map Reduce SerDe ThriftJuteJSON Thrift API MetaStore Web UI Mgmt, etc Hive QL PlannerExecutionParser Planner

34. Hive QL – Join SQL: INSERT INTO TABLE pv_users SELECT pv.pageid, u.age FROM page_view pv JOIN user u ON (pv.userid = u.userid); page id user id time 1111 9:08:01 2111 9:08:13 1222 9:08:14 user id agegender 11125female 22232male page id age 125 2 132 X = page_view user pv_users

35. Hive QL – Join in Map Reduce keyvalue 111 111 222 page id user id time 1111 9:08:01 2111 9:08:13 1222 9:08:14 user id agegender 11125female 22232male page_view user pv_users keyvalue 111 222 Map keyvalue 111 111 111 keyvalue 222 222 Shuffle Sort pagei d age 125 2 pagei d age 132 Reduce

36. Hive QL – Group By SQL: ▪ INSERT INTO TABLE pageid_age_sum ▪ SELECT pageid, age, count(1) ▪ FROM pv_users  GROUP BY pageid, age; page id age 125 2 132 225 pv_users pageidageCount 1251 2 2 1321 pageid_age_sum

37. Hive QL – Group By in Map Reduce page id age 125 2 pv_users page id ageCou nt 1251 1321 pageid_age_sum page id age 132 225 Map keyvalue 1 1 keyvalue 1 1 keyvalue 1 1 keyvalue 1 1 Shuffle Sort page id ageCou nt 2252 Reduce

38. Hive QL – Group By with Distinct SQL SELECT pageid, COUNT(DISTINCT userid) FROM page_view GROUP BY pageid page id useri d time 1111 9:08:01 2111 9:08:13 1222 9:08:14 2111 9:08:20 page_view pagei d count_distinct_u serid 12 21 result

39. Hive QL – Group By with Distinct in Map Reduce page_view pagei d coun t 12 Shuffle and Sort pagei d coun t 21 Reduce pagei d useri d time 1111 9:08:01 2111 9:08:13 pagei d useri d time 1222 9:08:14 2111 9:08:20 keyv keyv Shuffle key is a prefix of the sort key.

40. Hive QL: Order By page_view Shuffle and Sort Reduce pagei d useri d time 2111 9:08:13 1111 9:08:01 pagei d useri d time 2111 9:08:20 1222 9:08:14 keyv 9:08:0 1 9:08:1 3 keyv 9:08:1 4 9:08:2 0 pagei d useri d time 1111 9:08:01 2111 9:08:13 pagei d useri d time 1222 9:08:14 2111 9:08:20 Shuffle randomly.

41. Hive Optimizations Efficient Execution of SQL on top of Map-Reduce

42. Machine 2 Machine 1 (Simplified) Map Reduce Revisit Local Map Global Shuffle Local Sort Local Reduce

43. Merge Sequential Map Reduce Jobs SQL: FROM (a join b on a.key = b.key) join c on a.key = c.key SELECT … ke y avbv 1111222 keyav 1111 A Map Reduce keybv 1222 B keycv 1333 C AB Map Reduce ke y avbvcv 1111222333 ABC

44. Share Common Read Operations Extended SQL ▪ FROM pv_users ▪ INSERT INTO TABLE pv_pageid_sum ▪ SELECT pageid, count(1) ▪ GROUP BY pageid ▪ INSERT INTO TABLE pv_age_sum ▪ SELECT age, count(1) ▪ GROUP BY age; page id age 125 232 Map Reduce page id cou nt 11 21 page id age 125 232 Map Reduce agecou nt 251 321

45. Load Balance Problem page id ag e 125 1 1 232 125 pv_users page id ag e cou nt 1254 2321 pageid_age_sum Map-Reduce page id ag e cou nt 1252 2321 1252 pageid_age_partial_sum Map-Reduce

46. Machine 2 Machine 1 Map-side Aggregation / Combiner Local Map Global Shuffle Local Sort Local Reduce

47. Query Rewrite Predicate Push-down select * from (select * from t) where col1 = ‘2008’; Column Pruning select col1, col3 from (select * from t);

48. Apache Pig Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Pig's infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs, Pig's language layer currently consists of a textual language called Pig Latin, which has the following key properties: Ease of programming. It is trivial to achieve parallel execution of simple, "embarrassingly parallel" data analysis tasks. Complex tasks comprised of multiple interrelated data transformations are explicitly encoded as data flow sequences, making them easy to write, understand, and maintain. Optimization opportunities. The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency. Extensibility. Users can create their own functions to do special-purpose processing. 10/1/2016

49. Running Pig You can execute Pig Latin statements: – Using grunt shell or command line $ pig... - Connecting to... grunt> A = load 'data'; grunt> B =... ; – In local mode or hadoop mapreduce mode $ pig myscript.pig Command Line - batch, local mode mode $ pig -x local myscript.pig – Either interactively or in batch 10/1/2016

50. Program/flow organization A LOAD statement reads data from the file system. A series of "transformation" statements process the data. A STORE statement writes output to the file system; or, a DUMP statement displays output to the screen. 10/1/2016

51. Interpretation In general, Pig processes Pig Latin statements as follows: – First, Pig validates the syntax and semantics of all statements. – Next, if Pig encounters a DUMP or STORE, Pig will execute the statements. A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float); B = FOREACH A GENERATE name; DUMP B; (John) (Mary) (Bill) (Joe) Store operator will store it in a file 10/1/2016

52. Simple Examples A = LOAD 'input' AS (x, y, z); B = FILTER A BY x > 5; DUMP B; C = FOREACH B GENERATE y, z; STORE C INTO 'output'; ----------------------------------------------------------------------------- A = LOAD 'input' AS (x, y, z); B = FILTER A BY x > 5; STORE B INTO 'output1'; C = FOREACH B GENERATE y, z; STORE C INTO 'output2' 10/1/2016