Hive : A Petabyte Scale Data Warehouse Using Hadoop Lecturer : Prof. Kyungbaek Kim Presenter : Alvin Prayuda Juniarta Dwiyantoro

Contents Background Description System Architecture Data Types Operations Data Models SerDe Installation Guide Practical Example

Background Hadoop = Hive is the solution Pro Cons Superior in scalability/availability/manageability Effeciency scaled with more hardware Cons Map-reduce hard to program (user know sql) Need to publish data in well known structure Hive is the solution

Description Hive is a data warehouse software to facilitate querying and manage larga datasets in distributed storages Provides access to file stored in HDFS and query execution via MapReduce Hive use a simple SQL-like querie language to enable users familiars with SQL to query the data Hive not designed for OLTP (online transactional processing) and doesn’t offer real-time queries. Hive values are in scalability, extensibility, fault-tolerance, loose- coupling with its input format

System Architecture UI – The user interface for users to submit queries and other operations to the system. As of 2011 the system had a command line interface and a web based GUI was being developed. Driver – The component which receives the queries. This component implements the notion of session handles and provides execute and fetch APIs modeled on JDBC/ODBC interfaces. Compiler – The component that parses the query, does semantic analysis on the different query blocks and query expressions and eventually generates an execution plan with the help of the table and partition metadata looked up from the metastore. Metastore – The component that stores all the structure information of the various tables and partitions in the warehouse including column and column type information, the serializers and deserializers necessary to read and write data and the corresponding HDFS files where the data is stored. Execution Engine – The component which executes the execution plan created by the compiler. The plan is a DAG of stages. The execution engine manages the dependencies between these different stages of the plan and executes these stages on the appropriate system components

Data Types Data type that supported by current Hive (v 0.13.1) Numeric Types Tinyint, smallint, int, bigint, float, double, decimal < (user defined precision and scale) Date/time Timestamp, date String String, varchar, char Misc Boolean, binary Complex Arrays, maps, structs, union

Operations 3 kinds of operation in Hive : Data Definition Language (DDL) Operation Data Manipulation Language (DML) Operation Structured Query Language (SQL) Operation

Operations DDL Operations Create/drop/alter database Use database Create/drop/truncate table Alter table/partition/column Create/drop/alter view Create/drop/alter index Create/drop function Create/drop/grant/revoke roles and privileges Show Describe Export/Import

Operations Example DDL Create table Alter table Drop table

Operations DML Operations Loading files into tables Inserting data into Hive Tables from Queries Writing data into filesystem from Queries Inserting values into tables from SQL (v.0.14) Updating values in tables from SQL (v.0.14) Deleting values in tables from SQL (v.0.14)

Operations Example DML Load data into table Write data into files

Operations SQL Operations Extensibility Select and Filters Group By Insert Overwrite and Insert into Join Multitable insert Extensibility Pluggable Map-reduce script using transform

Operations Example SQL Show data Insert data with select statement Group by

Operations Join Multitable Insert

Data Models Data in Hive is organized into : Table : A table model like in relational database, stored in HDFS Partition :  A partition of table which is stored in a sub-directory within a table’s directory, allow the system to prune data to be inspected based on query predicates Example : a query that is interested in rows from T that satisfy the predicate T.ds = '2008-09-01' would only have to look at files in <table location>/ds=2008-09-01/ directory in HDFS Bucket :  Data in each partition may in turn be divided into Buckets based on the hash of a column in the table. Each bucket is stored as a file in the partition directory, allows the system to efficiently evaluate queries that depend on a sample of data

SerDe SerDe is short for Serializer/Deserializer. Hive uses the SerDe interface for IO. The interface handles both serialization and deserialization and also interpreting the results of serialization as individual fields for processing. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. Anyone can write their own SerDe for their own data formats.

Projects Related to Hive Shark Fork of Apache Hive that using Spark instead of map-reduce Hivemall Machine-learning library for Hive Apache Sentry Role-based authorization system for Hive