1. Why is my Hadoop* job slow?
Bikas Saha
@bikassaha
Hitesh Shah
*Apache Hadoop, Falcon, Atlas, Tez, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie, Zeppelin and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
Hortonworks: Powering the Future of Data

2. Agenda
Metrics and Monitoring Logging and Correlation Tracing and Analysis
Hortonworks: Powering the Future of Data

3. Metrics and Monitoring
Metrics as high level pointers
Ambari Metrics System
Ambari Grafana Integration
HBase, HDFS, YARN Dashboards
Metrics based alerting
Hortonworks: Powering the Future of Data

4. Metrics as high level pointers
Machine level metrics like CPU load
Application level metrics like HDFS counters
Metrics at point of time
Metrics anomalies along a time series
Correlated anomalies
Problem is to need to know what to look for

5. Ambari Metrics Service - Motivation
First version released with Ambari 2.0.0
Limited Ganglia capabilities
OpenTSDB – GPL license and needs a Hadoop cluster
Need service level aggregation as well as time based
Alerts based on metrics system
Ability to scale past a 1000 nodes
Ability to perform analytics based on a use case
Allow fine grained control over aspects like: retention, collection intervals, aggregation
Pluggable and Extensible

6. Ambari Grafana Integration
Open source dashboard builder integrated with AMS.
Available from Ambari-2.2.2
Pre-defined host level and service level (HDFS, HBase, Yarn etc) dashboards.
Added to Ambari through API after upgrade

7. HBase Dashboard

8. HDFS Dashboard

9. YARN Dashboard

10. Metrics based Alerting
Top N support to quickly identify potential offenders
Alerting based on time series

11. Agenda
Metrics and Monitoring Logging and Correlation Tracing and Analysis
Hortonworks: Powering the Future of Data

12. Logging and Correlation
HDFS, YARN Audit logs
Caller Context
YARN Application Timeline Service
Lineage tracking of operations across workloads
Ambari Log Search
Hortonworks: Powering the Future of Data

13. HDFS Audit Logs and Caller Context
FSNamesystem.audit: allowed=true ugi=userA (auth:SIMPLE) ip=/ cmd=create src=/tmp/in/_temporary/1/_temporary/attempt_ _0009_m_009995_0/part-m dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=tez_ta:attempt_ _0009_1_00_009995_0 FSNamesystem.audit: allowed=true ugi=userA (auth:SIMPLE) ip=/ cmd=create src=/tmp/in2/_temporary/1/_temporary/attempt_ _0011_m_000097_0/part-m dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=mr_attempt_ _0011_m_000097_0 FSNamesystem.audit: allowed=true ugi=userB (auth:SIMPLE) ip=/ cmd=create src=/tmp/in2/_temporary/1/_temporary/attempt_ _0011_m_000095_0/part-m dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=mr_attempt_ _0011_m_000095_0
It is now possible to infer which application/job did what in HDFS
Files created can be tracked down to the MR or Tez job and the specific task attempt that created them.
Using simple string manipulation and aggregations, you can file jobs inducing high loads against the Namenode.

14. ResourceManager Audit Logs and Caller Context
resourcemanager.RMAuditLogger: USER=userA IP= OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_ _0001 CALLERCONTEXT=PIG-pigSmoke.sh-8a e39-83b1-ebad699d8e2e resourcemanager.RMAuditLogger: USER=userA IP= OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_ _0009 CALLERCONTEXT=CLI resourcemanager.RMAuditLogger: USER=userB IP= OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_ _0008 CALLERCONTEXT=mr_attempt_ _0007_m_000000_0 resourcemanager.RMAuditLogger: USER=userB IP= OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_ _0012 CALLERCONTEXT=HIVE_SSN_ID:f3aadf99-9e36-494b-84a1-99b685ac344b
Tracking what YARN maps to what application type and instance is now much easier.
It could made more easier if “mr_attempt_ _0007_m_000000_0” pointed to an oozie worklow instead of the MR job 
Who killed my application and how (command-line, webservice)?

15. YARN Application Timeline Service
YARN service for fine grained application level tracing
Enables complex metadata to be recorded as the YARN app makes progress
Allows retrieval of this timeline data based on filters
Can be used to drive limited online analytics and extensive post-hoc analysis

16. Lineage Tracking using YARN Timeline
Timeline:8188/ws/v1/timeline/TEZ_DAG_ID/dag_ _0013_1
dagContext: { callerId: "root_ _006f c c1b506f68db2",
callerType: "HIVE_QUERY_ID",
context: "HIVE",
description: "select user, count(visit_id) as visits from users group by user order by visits” }
Timeline:8188/ws/v1/timeline/HIVE_QUERY_ID/root_ _006f c c1b506f68db2
hiveContext: { callerId: “workflow_abcd",
callerType: “OOZIE_ID",
context: “OOZIE",
description: “Daily ETL Summary Job” }

17. Ambari Log Search

18. Ambari Log Search

19. Agenda
Metrics and Monitoring Logging and Correlation Tracing and Analysis
Hortonworks: Powering the Future of Data

20. Tracing and Analysis Use Big Data methods to solve Big Data problems
Apache Zeppelin as analytical tool
Hive/Tez/YARN notebook for analysis
Hortonworks: Powering the Future of Data

21. Zeppelin for Ad-hoc Analytics

22. YARN Analyzer

23. Tez Analyzer

24. Tez Analyzer

25. Tez Analyzer

26. Thank You
Hortonworks: Powering the Future of Data