Resource Management with YARN: YARN Past, Present and Future Anubhav Dhoot Software Engineer Cloudera

YARN (DYNAMIC RESOURCE MANAGEMENT) Map Reduce Impala Spark YARN (DYNAMIC RESOURCE MANAGEMENT)

YARN (Yet Another Resource Negotiator) Traditional Operating System Storage: File System Execution/ Scheduling: Processes/ Kernel Scheduler Hadoop Storage: Hadoop Distributed File System (HDFS) Execution/ Scheduling: Yet Another Resource Negotiator (YARN) Think Hadoop as a distributed operating system, YARN is the kernel scheduler

Overview of Talk History of YARN Recent features On going features Future

Why YARN

Traditional Distributed Execution Engines Master Worker Task Client Typical distributed system architecture

JobTracker tracks every task in the cluster! MapReduce v1 (MR1) Job Tracker Task Tracker Map Map Task Tracker Reduce Client Map Client Task Tracker Map Reduce JobTracker tracks every task in the cluster!

Fixed-size slot model forces slots large enough for the biggest task! MR1 Utilization 4 GB Map 1024 MB Map 1024 MB Reduce 1024 MB Reduce 1024 MB Fixed-size slot model forces slots large enough for the biggest task!

Running multiple frameworks… Master Worker Task Client Master Worker Task Client Master Worker Task Client Multiple framework manage resource on their own, they can stomp over each other

YARN to the rescue! Scalability: Track only applications, not all tasks. Utilization: Allocate only as many resources as needed. Multi-tenancy: Share resources between frameworks and users Physical resources – memory, CPU, disk, network

YARN Architecture Node Manager Resource Manager App Container Master Client App Master Container Applications State Cluster State Node heartbeat, AM manages each applications tasks->containers, RM only track overall application state

MR1 to YARN/MR2 functionality mapping JobTracker is split into ResourceManager – cluster-management, scheduling and application state handling ApplicationMaster – Handle tasks (containers) per application (e.g. MR job) JobHistoryServer – Serve MR history TaskTracker maps to NodeManager

Early Features

Handing faults on Workers Resource Manager App Master Node Manager Client Node Manager App Master Container Client Applications State Container Node Manager Cluster State RM starts AM, AM will recover containers lost

Master Fault-tolerance - RM Recovery Resource Manager Node Manager App Master Container Client Client Applications State Node Manager App Master Container Cluster State RM Store In original HA, running containers were killed

Master Node Fault tolerance High Availability (Active / Standby) Resource Manager Node Manager App Master Client Elector RM Store ZK Standby Resource Manager Elector Client Node Manager Builds on top of RM Recovery ZK used for elector and for storing state

Master Node Fault tolerance High Availability (Active / Standby) Resource Manager Node Manager Client Elector RM Store ZK Active Resource Manager Elector Client App Master Node Manager Same as recovery, AM and containers need to be restarted

Scheduler Inside ResourceManager Decides who gets to run when and where Uses “Queues” to describe organization needs Applications are submitted to a queue Two schedulers out of the box Fair Scheduler Capacity Scheduler

Fair Scheduler Hierarchical Queues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Jim’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores Bob’s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores

Fair Scheduler Queue Placement Policies <queuePlacementPolicy> <rule name="specified" /> <rule name="primaryGroup" create="false" /> <rule name="default" /> </queuePlacementPolicy>

Multi-Resource Scheduling Node capacities expressed in both memory and CPU Memory in MB and CPU in terms of vcores Scheduler uses dominant resource for making decisions

Multi-Resource Scheduling 6 cores 50% cap. 12 GB 33% cap. 10 GB 28% cap. 3 cores 25% cap. Queue 1 Usage Queue 2 Usage

Multi-Resource Enforcement YARN kills containers that use too much memory CGroups for limiting CPU

Recently added features

RM recovery without losing work Preserving running containers on RM restart NM no longer kills containers on resync AM made to register on resync with RM

RM recovery without losing work Resource Manager Node Manager Client Client Applications State Node Manager App Master Container Cluster State RM Store In original HA, running containers were killed

NM Recovery without losing work NM stores container and its associated state in a local store On restart reconstruct state from store Default implementation using LevelDB Supports rolling restarts with no user impact

NM Recovery without losing work Resource Manager Client Node Manager App Master Container Client Applications State State Store Cluster State

Fair Scheduler Dynamic User Queues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Moe Fair Share Mem: 2 GB Fair Share CPU: 4 cores Moe Fair Share Mem: 4 GB Fair Share CPU: 8 cores Larry Fair Share Mem: 2 GB Fair Share CPU: 4 cores

On going features

Long Running Apps on Secure Clusters (YARN-896) Update tokens of running applications Reset AM failure count to allow mulitple failures over a long time Need to access logs while application is running Need a way to show progress

Application Timeline Server (YARN-321, YARN-1530) Currently we have a JobHistoryServer for MapReduce history Generic history server Gives information even while job is running YARN supports generic applications and thus it needs a generic history server

Application Timeline Server Store and serve generic data like when containers ran, container logs Apps post app-specific events e.g. MapReduce Attempt Succeeded/Failed Pluggable framework-specific UIs Pluggable storage backend Default LevelDB

Disk scheduling (YARN-2139 ) Disk as a resource in addition to CPU and Memory Expressed as virtual disk similar to vcore for cpu Dominant resource fairness can handle this on the scheduling side Use CGroups blkio controller for enforcement

Reservation-based Scheduling (YARN-1051)

Reservation-based Scheduling

FUTURE FEATUREs

Container Resizing (YARN-1197) Change container’s resource allocation Very useful for frameworks like Spark that schedule multiple tasks within a container Follow same paths as for acquiring and releasing containers

Admin labels (YARN-796) Admin tags nodes with labels (e.g. GPU) Applications can include labels in container requests I want a GPU Application Master NodeManager [Windows] NodeManager [GPU, beefy]

Container Delegation (YARN-1488) Problem: single process wants to run work on behalf of multiple users. Want to count resources used against users that use them. E.g. Impala or HDFS caching

Container Delegation (YARN-1488) Solution: let apps “delegate” their containers to other containers on the same node. Delegated container never runs Framework container gets its resources Framework container responsible for fairness within itself

Questions?

Thank You! Anubhav Dhoot, Software Engineer, Cloudera adhoot@cloudera.com