2. Hive Index Yongqiang He Software Engineer Facebook Data Infrastructure Team

3. 1 Create Index 2 Update Index / Rebuild Index 3 Use Index 4 Metastore upgrade script Agenda

4. Create Index CREATE INDEX index_name ON TABLE table_name (col_name…) AS ‘index handler class name’ [WITH DEFERRED REBUILD] [INDEXPROPERTIES (prop_key=prop_value, …)] [IN TABLE index_table_name] [[ROW FORMAT …] STORED AS …] EXAMPLE 1: CREATE TABLE src (key int, value string); CREATE INDEX src_index ON TABLE src(key) as 'COMPACT' WITH DEFERRED REBUILD STORED AS RCFILE; EXAMPLE 2: CREATE TABLE srcpart_rc (key int, value string) PARTITIONED BY (ds string, hr int) STORED AS RCFILE; CREATE INDEX src_part_index ON TABLE srcpart_rc (key) as 'COMPACT' WITH DEFERRED REBUILD;

5. Update Index / Rebuild Index ALTER INDEX index_name ON table_name [partitionSpec] REBUILD; EXAMPLE : ALTER INDEX src_index ON src REBUILD; ALTER INDEX src_part_index ON srcpart_rc REBUILD; ALTER INDEX src_part_index ON srcpart_rc partition(ds=‘2010-08-09’) REBUILD;

6. Use Index No Optimizer Available right now. Need to use index manually in query. (Working on an index optimizer to direct a query against index.) EXAMPLE : Original query: SELECT key, value FROM srcpart_rc WHERE key=100; Query with index: INSERT OVERWRITE DIRECTORY "/tmp/index_result" SELECT `_bucketname`, `_offsets` FROM default__srcpart_rc_srcpart_rc_index__ WHERE key=100; SET hive.index.compact.file=/tmp/index_result; SET hive.input.format=org.apache.hadoop.hive.ql.index.compact.HiveCompactIndexInputFormat; SELECT key, value FROM srcpart_rc WHERE key=100 ORDER BY key; designed to help queries with filter clause (point query, range query).

7. Metastore upgrade script MySQL: http://wiki.apache.org/hadoop/Hive/IndexDev#Metastore_Upgrades