Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Table of Contents

Hadoop Yarn key architecture principles

1. Introduction to Yarn

2. Yarn Architecture

3. Yarn task operation process

4. Characteristics and usage scenarios of three Yarn resource schedulers

---

Hadoop Yarn key architecture principles

1. Introduction to Yarn

Apache Hadoop Yarn (Yet Another Reasource Negotiator) is a resource manager used after Hadoop 2.x, which can provide a unified resource management platform for upper-level applications.

2. Yarn Architecture

Yarn is mainly composed of ResourceManager, NodeManager, ApplicationMaster, and Container.

• ResourceManager

ResourceManager is the central manager in the Yarn cluster, responsible for resource allocation and scheduling of the entire cluster. ResourceManager is responsible for monitoring the status of NodeManager nodes, aggregating cluster resources, processing resource requests submitted by Clients, and starting and monitoring ApplicationMaster for each Application.

• NodeManager

NodeManager is responsible for managing resources (such as memory, CPU, etc.) on each node and reporting to ResourceManager. When ResourceManager assigns a container to NodeManager, NodeManager is responsible for starting the container and monitoring the container operation. In addition, NodeManager will receive AplicationMaster commands to start containers for each application.

• ApplicationMaster

Each application running in Yarn will start a corresponding ApplicationMaster, which is responsible for applying for resources from ResourceManager and managing application tasks. ApplicationMaster is essentially a container, which is scheduled by ResourceManager and started by NodeManager. After ApplicationMaster is started, it will apply for resources from ResourceManager to run the application. After ResourceManager allocates container resources, ApplicationMaster will connect to the corresponding NodeManager to notify the startup of Container and manage the tasks running on Container.

• Container

Container is the basic execution unit in Yarn, used to run application tasks. It is a virtual environment that contains application code, dependencies, and resources required for operation (memory, CPU, disk, network). Each container is assigned to ApplicationMaster by ResourceManager and started and managed by NodeManager on the corresponding node. The resource usage of the container is monitored by NodeManager and reported to ResourceManager when necessary.

The core of Yarn is to separate the resource management and task scheduling functions of JobTracker in MR1, which are implemented by ResourceManager and ApplicationMaster processes respectively. ResourceManager is responsible for resource management and scheduling of the entire cluster; ApplicationMaster is responsible for application task scheduling, task monitoring and fault tolerance, etc.

3. Yarn task operation process

1. When the client submits an MR task to Yarn, the MR task resources (Split, resource configuration, Jar package information) will first be uploaded to HDFS.

2. The client requests ResourceManager to start ApplicationMaster.

3. ResourceManager will select a relatively less busy NodeManager node and notify the node to start ApplicationMaster (Container).

4. After ApplicationMaster is started, it will download MR task resource information from HDFS to the local computer, and then apply for resources from ResourceManager to start MR Task.

5. ResourceManager returns the resource list to ApplicationMaster.

6. ApplicationMaster then notifies the corresponding NodeManager to start the Container

7. After the Container is started, it will be reversely registered with the ApplicationMaster.

8. ApplicationMaster sends the Task to the Container for execution. The Task executes the business logic of the code we wrote.

4. Characteristics and usage scenarios of three Yarn resource schedulers

• FIFO Scheduler (First-In-Fist-Out Scheduler), the simplest scheduler in Yarn. FIFO Scheduler puts submitted applications into a first-in-first-out queue in the order they are submitted. When allocating resources, it first allocates resources to the application at the head of the queue. After the resource requirements of the application at the head are met, it allocates resources to the next application, and so on. When this scheduler schedules resources, it is possible that an application with a large resource requirement occupies all cluster resources, causing other applications to be blocked.

• Capacity SchedulerIt is the default resource scheduler in Yarn, allowing multiple tenants to safely share a large cluster. In the Capacity scheduler, multiple resource queues can be configured, and the minimum and maximum available resource ratios can be specified for each resource queue. When allocating resources, idle resources are preferentially allocated to the queue with the lowest "actual resources/budget resources" ratio. Each resource queue adopts the FIFO scheduling strategy.

• Fair SchedulerIt is a resource scheduling method that fairly allocates YARN resources to each application. This scheduling method allows all applications to obtain equal resource shares over time. Its design goal is to allocate fair resources to all applications according to defined parameters.

The differences between the three resource schedulers in Yarn: FIFO, Capacity, and Fair are as follows:

---

• 📢博客主页：https://lansonli.blog.csdn.net
• 📢欢迎点赞 👍 收藏 ⭐留言 📝 如有错误敬请指正！
• 📢本文由 Lansonli 原创，首发于 CSDN博客🙉
• 📢停下休息的时候不要忘了别人还在奔跑，希望大家抓紧时间学习，全力奔赴更美好的生活✨