1. Monitoring Hive: Metrics and WebUI
Szehon Ho, Software Engineer

2. Motivation
Lagging Behind: Hive is “data warehouse” of Hadoop stack, but has least monitoring capability
Hadoop Components with WebUI/metrics: (HDFS, MR1, Spark, YARN, Impala, Oozie)
Hive moving to enterprise use-cases
More users, more queries, more resource-intensive queries, HS2/HMS crashes
Production environment, need answers when something goes wrong, indication when about to go wrong
Users forced to different places to find about Hive jobs:
Execution Engine monitoring (Yarn/MR, Spark, Tez)
Canary health checks
HMS/HS2 process monitoring (JMX, OS, etc)

3. Introduction Metrics: HIVE-10761 Admin WebUI: HIVE-12338
Statistics from Hive, long-term aggregates over Hive’s lifecycle
Consumed by third-party tools
Admin WebUI: HIVE-12338
Current jobs/status of Hive
User-facing

4. Codahale-Based Metrics: HIVE-10761
Java Metrics (ManagementFactory)
Gauges, HIstograms
Timers
Meters
GC, ClassLoader,
Buffer, Threads, Memory
Libraries enhancing metrics with statistical breakdowns,
like min,max, stddev, percentiles
Hive: Codahale Metrics System
JMX
HTTP
Stdout, log
File
Ganglia

5. HiveServer2 and Hive Metastore
Metrics Added So Far
HiveServer2 and Hive Metastore
Open Connections - Number of incoming connections to HS2/HMS
JVMPauseMonitor (from Hadoop, Oozie) - Number of extra seconds spent in sleep (for GC detection)
Java Process (from Codahale) – GC count, Threads count, memory breakdown, etc.
HiveServer2
API – Perf Logger timers
Operations –
Open Operations
Duration in each state (INIT, PENDING, RUNNING)
Final state counts (FINISHED, ERROR, CANCELLED -> CLOSED)
Thread Pool, Queue
Hive Metastore
API – Method timers
Metadata –
Databases
Tables
Partitions

6. Web UI for HS2– HIVE-12338 Pages
Status Page (Running queries and states, Build+Process information)
HiveServer2 Logs
Current Configuration
Stack-Trace
Security
SSL
SPNEGO (Kerberos)
Hadoop Admin Servlet (ACL’s)
Front-end
CSS, graphic files from Twitter Bootstrap

7. Going Forward
More Metrics (ie, track failures, more operation breakdown)
Aggregate metrics from distributed execution engines (MR, Tez, Spark)
WebUI historic query information
WebUI enhance query state information (more states: compiling, acquiring lock, running with Job URL, etc)
Hive Metastore WebUI

8. Thank you.