1. Resource Management with YARN: YARN Past, Present and Future
Anubhav Dhoot
Software Engineer
Cloudera

2. YARN (DYNAMIC RESOURCE MANAGEMENT)
Map Reduce
Impala
Spark
YARN (DYNAMIC RESOURCE MANAGEMENT)

3. YARN (Yet Another Resource Negotiator)
Traditional Operating System
Storage:
File System
Execution/
Scheduling:
Processes/
Kernel Scheduler
Hadoop
Storage:
Hadoop Distributed File System (HDFS)
Execution/
Scheduling:
Yet Another Resource Negotiator (YARN)
Think Hadoop as a distributed operating system, YARN is the kernel scheduler

4. Overview of Talk History of YARN Recent features On going features
Future

5. Why YARN

6. Traditional Distributed Execution Engines
Master
Worker
Task
Client
Typical distributed system architecture

7. JobTracker tracks every task in the cluster!
MapReduce v1 (MR1)
Job Tracker
Task Tracker
Map
Map
Task Tracker
Reduce
Client
Map
Client
Task Tracker
Map
Reduce
JobTracker tracks every task in the cluster!

8. Fixed-size slot model forces slots large enough for the biggest task!
MR1 Utilization
4 GB
Map
1024 MB
Map
1024 MB
Reduce
1024 MB
Reduce
1024 MB
Fixed-size slot model forces slots large enough for the biggest task!

9. Running multiple frameworks…
Master
Worker
Task
Client
Master
Worker
Task
Client
Master
Worker
Task
Client
Multiple framework manage resource on their own, they can stomp over each other

10. YARN to the rescue!
Scalability: Track only applications, not all tasks.
Utilization: Allocate only as many resources as needed.
Multi-tenancy: Share resources between frameworks and users
Physical resources – memory, CPU, disk, network

11. YARN Architecture Node Manager Resource Manager App Container Master
Client
App
Master
Container
Applications
State
Cluster State
Node heartbeat, AM manages each applications tasks->containers, RM only track overall application state

12. MR1 to YARN/MR2 functionality mapping
JobTracker is split into
ResourceManager – cluster-management, scheduling and application state handling
ApplicationMaster – Handle tasks (containers) per application (e.g. MR job)
JobHistoryServer – Serve MR history
TaskTracker maps to NodeManager

13. Early Features

14. Handing faults on Workers
Resource Manager
App
Master
Node Manager
Client
Node Manager
App
Master
Container
Client
Applications
State
Container
Node Manager
Cluster State
RM starts AM, AM will recover containers lost

15. Master Fault-tolerance - RM Recovery
Resource Manager
Node Manager
App
Master
Container
Client
Client
Applications
State
Node Manager
App
Master
Container
Cluster State
RM Store
In original HA, running containers were killed

16. Master Node Fault tolerance High Availability (Active / Standby)
Resource Manager
Node Manager
App
Master
Client
Elector
RM Store ZK
Standby
Resource Manager
Elector
Client
Node Manager
Builds on top of RM Recovery
ZK used for elector and for storing state

17. Master Node Fault tolerance High Availability (Active / Standby)
Resource Manager
Node Manager
Client
Elector
RM Store ZK
Active
Resource Manager
Elector
Client
App
Master
Node Manager
Same as recovery, AM and containers need to be restarted

18. Scheduler Inside ResourceManager
Decides who gets to run when and where
Uses “Queues” to describe organization needs
Applications are submitted to a queue
Two schedulers out of the box
Fair Scheduler
Capacity Scheduler

19. Fair Scheduler Hierarchical Queues
Root
Mem Capacity: 12 GB
CPU Capacity: 24 cores
Marketing
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
R&D
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
Sales
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
Jim’s Team
Fair Share Mem: 2 GB
Fair Share CPU: 4 cores
Bob’s Team
Fair Share Mem: 2 GB
Fair Share CPU: 4 cores

20. Fair Scheduler Queue Placement Policies
<queuePlacementPolicy> <rule name="specified" /> <rule name="primaryGroup" create="false" /> <rule name="default" /> </queuePlacementPolicy>

21. Multi-Resource Scheduling
Node capacities expressed in both memory and CPU
Memory in MB and CPU in terms of vcores
Scheduler uses dominant resource for making decisions

22. Multi-Resource Scheduling
6 cores
50% cap.
12 GB
33% cap.
10 GB
28% cap.
3 cores
25% cap.
Queue 1 Usage
Queue 2 Usage

23. Multi-Resource Enforcement
YARN kills containers that use too much memory
CGroups for limiting CPU

24. Recently added features

25. RM recovery without losing work
Preserving running containers on RM restart
NM no longer kills containers on resync
AM made to register on resync with RM

26. RM recovery without losing work
Resource Manager
Node Manager
Client
Client
Applications
State
Node Manager
App
Master
Container
Cluster State
RM Store
In original HA, running containers were killed

27. NM Recovery without losing work
NM stores container and its associated state in a local store
On restart reconstruct state from store
Default implementation using LevelDB
Supports rolling restarts with no user impact

28. NM Recovery without losing work
Resource Manager
Client
Node Manager
App
Master
Container
Client
Applications
State
State
Store
Cluster State

29. Fair Scheduler Dynamic User Queues
Root
Mem Capacity: 12 GB
CPU Capacity: 24 cores
Marketing
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
R&D
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
Sales
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
Moe
Fair Share Mem: 2 GB
Fair Share CPU: 4 cores
Moe
Fair Share Mem: 4 GB
Fair Share CPU: 8 cores
Larry
Fair Share Mem: 2 GB
Fair Share CPU: 4 cores

30. On going features

31. Long Running Apps on Secure Clusters (YARN-896)
Update tokens of running applications
Reset AM failure count to allow mulitple failures over a long time
Need to access logs while application is running
Need a way to show progress

32. Application Timeline Server (YARN-321, YARN-1530)
Currently we have a JobHistoryServer for MapReduce history
Generic history server
Gives information even while job is running
YARN supports generic applications and thus it needs a generic history server

33. Application Timeline Server
Store and serve generic data like when containers ran, container logs
Apps post app-specific events
e.g. MapReduce Attempt Succeeded/Failed
Pluggable framework-specific UIs
Pluggable storage backend
Default LevelDB

34. Disk scheduling (YARN-2139 )
Disk as a resource in addition to CPU and Memory
Expressed as virtual disk similar to vcore for cpu
Dominant resource fairness can handle this on the scheduling side
Use CGroups blkio controller for enforcement

35. Reservation-based Scheduling (YARN-1051)

36. Reservation-based Scheduling

37. FUTURE FEATUREs

38. Container Resizing (YARN-1197)
Change container’s resource allocation
Very useful for frameworks like Spark that schedule multiple tasks within a container
Follow same paths as for acquiring and releasing containers

39. Admin labels (YARN-796) Admin tags nodes with labels (e.g. GPU)
Applications can include labels in container requests
I want a GPU
Application Master
NodeManager
[Windows]
NodeManager
[GPU, beefy]

40. Container Delegation (YARN-1488)
Problem: single process wants to run work on behalf of multiple users.
Want to count resources used against users that use them.
E.g. Impala or HDFS caching

41. Container Delegation (YARN-1488)
Solution: let apps “delegate” their containers to other containers on the same node.
Delegated container never runs
Framework container gets its resources
Framework container responsible for fairness within itself

42. Questions?

43. Thank You! Anubhav Dhoot, Software Engineer, Cloudera