yarn 怎么查看有多个job在跑_flink on yarn 模式下提示yarn资源不足问题分析

背景

在实时计算平台上通过YarnClient向yarn上提交flink任务时一直卡在那里,并在client端一直输出如下日志:

(YarnClusterDescriptor.java:1036)- Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

看到这个的第一反应是yarn上的资源分配问题,于是来到yarn控制台,发现Cluster Metrics中Apps Pending项为1。what?新提交的job为什么会处于pending状态了?

1. 先确定cpu和内存情况如下:

d0735c0410bd152a46147b478eef7706.png

可以看出cpu和内存资源充足,没有发现问题。

2. 查看调度器的使用情况

集群中使用的调度器的类型如下图:

be57efb63b0b7551e81957b991c3a727.png

可以看到,集群中使用的是Capacity Scheduler调度器,也就是所谓的容量调度,这种方案更适合多租户安全地共享大型集群,以便在分配的容量限制下及时分配资源。采用队列的概念,任务提交到队列,队列可以设置资源的占比,并且支持层级队列、访问控制、用户限制、预定等等配置。但是,对于资源的分配占比调优需要更多的经验处理。但它不会出现在使用FIFO Scheduler时会出现的有大任务独占资源,会导致其他任务一直处于 pending 状态的问题。

3. 查看任务队列的情况

e4e1ffaabf59e9fa9d0adb6fcc6663c4.png

从上图中可以看出Configured Minimum User Limit Percent的配置为100%,由于集群目前相对较小,用户队列没有做租户划分,用的都是default队列,从图中可以看出使用的容量也只有38.2%,队列中最多可存放10000个application,而实际的远远少于10000,貎似这里也看不出来什么问题。

4. 查看resourceManager的日志

日志内容如下:

2020-11-26 19:33:46,669 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 3172020-11-26 19:33:48,874 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 317 submitted by user root2020-11-26 19:33:48,874 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1593338489799_03172020-11-26 19:33:48,874 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root    IP=x.x.x.x    OPERATION=Submit Application Request    TARGET=ClientRMService    RESULT=SUCCESS    APPID=application_1593338489799_03172020-11-26 19:33:48,874 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1593338489799_0317 State change from NEW to NEW_SAVING on event=START2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1593338489799_03172020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1593338489799_0317 State change from NEW_SAVING to SUBMITTED on event=APP_NEW_SAVED2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1593338489799_0317 user: root leaf-queue of parent: root #applications: 162020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Accepted application application_1593338489799_0317 from user: root, in queue: default2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1593338489799_0317 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1593338489799_0317_0000012020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1593338489799_0317_000001 State change from NEW to SUBMITTED2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit2020-11-26 19:33:48,877 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1593338489799_0317 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@6c0d5b4d, leaf-queue: default #user-pending-applications: 1 #user-active-applications: 15 #queue-pending-applications: 1 #queue-active-applications: 15

从日志中可以看到一个Application在yarn上进行资源分配的完整流程,只是这个任务因为一些原因进入了pending队列而已,与我们要查找的问题相关的日志主要是如下几行:

2020-11-26 19:33:48,875 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not starting application as amIfStarted exceeds amLimit2020-11-26 19:33:48,877 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1593338489799_0317 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@6c0d5b4d, leaf-queue: default #user-pending-applications: 1 #user-active-applications: 15 #queue-pending-applications: 1 #queue-active-applications: 15

没错,问题就出来not starting application as amIfStarted exceeds amLimit,那么这个是什么原因引起的呢,我们看下stackoverflow[1]上的解释:

ba4409ea3f232384e0da6acbcd022b7c.png

那么yarn.scheduler.capacity.maximum-am-resource-percent参数的真正含义是什么呢?国语意思就是集群中可用于运行application master的资源比例上限,这通常用于限制并发运行的应用程序数目,它的默认值为0.1。

查看了下集群上目前的任务总数有15个左右,每个任务分配有一个约1G左右的jobmanager(jobmanager为Application master类型的application),占15G左右,而集群上的总内存为144G,那么15>144 * 0.1 ,从而导致jobmanager的创建处于pending状态。

5. 解决验证

修改capacity-scheduler.xml的yarn.scheduler.capacity.maximum-am-resource-percent配置为如下:

    yarn.scheduler.capacity.maximum-am-resource-percent    0.5  

除了动态减少队列数目外,capacity-scheduler.xml的其他配置的修改是可以动态更新的,更新命令为:

yarn rmadmin -refreshQueues

执行命令后,在resourceManager的日志中可以看到如下输出:

2020-11-27 09:37:56,340 INFO org.apache.hadoop.conf.Configuration: found resource capacity-scheduler.xml at file:/work/hadoop-2.7.4/etc/hadoop/capacity-scheduler.xml2020-11-27 09:37:56,356 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Re-initializing queues...---------------------------------------------------------------------------2020-11-27 09:37:56,371 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue mappings, override: false2020-11-27 09:37:56,372 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root    IP=x.x.x.x    OPERATION=refreshQueues    TARGET=AdminServicRESULT=SUCCESS

仔细查看日志可以看到更新已经成功,在平台上重新发布任务显示成功,问题解决。

yarn Queue的配置

Resource Allocation

PropertyDescription
yarn.scheduler.capacity..capacityQueue capacity in percentage (%) as a float (e.g. 12.5) OR as absolute resource queue minimum capacity. The sum of capacities for all queues, at each level, must be equal to 100. However if absolute resource is configured, sum of absolute resources of child queues could be less than it’s parent absolute resource capacity. Applications in the queue may consume more resources than the queue’s capacity if there are free resources, providing elasticity.
yarn.scheduler.capacity..maximum-capacityMaximum queue capacity in percentage (%) as a float OR as absolute resource queue maximum capacity. This limits the elasticity for applications in the queue. 1) Value is between 0 and 100. 2) Admin needs to make sure absolute maximum capacity >= absolute capacity for each queue. Also, setting this value to -1 sets maximum capacity to 100%.
yarn.scheduler.capacity..minimum-user-limit-percentEach queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. If a third user submits an application, no single user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed. The default is 100. Value is specified as a integer.
yarn.scheduler.capacity..user-limit-factorThe multiple of the queue capacity which can be configured to allow a single user to acquire more resources. By default this is set to 1 which ensures that a single user can never take more than the queue’s configured capacity irrespective of how idle the cluster is. Value is specified as a float.
yarn.scheduler.capacity..maximum-allocation-mbThe per queue maximum limit of memory to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration yarn.scheduler.maximum-allocation-mb. This value must be smaller than or equal to the cluster maximum.
yarn.scheduler.capacity..maximum-allocation-vcoresThe per queue maximum limit of virtual cores to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration yarn.scheduler.maximum-allocation-vcores. This value must be smaller than or equal to the cluster maximum.
yarn.scheduler.capacity..user-settings..weightThis floating point value is used when calculating the user limit resource values for users in a queue. This value will weight each user more or less than the other users in the queue. For example, if user A should receive 50% more resources in a queue than users B and C, this property will be set to 1.5 for user A. Users B and C will default to 1.0.

Resource Allocation using Absolute Resources configuration

CapacityScheduler supports configuration of absolute resources instead of providing Queue capacity in percentage. As mentioned in above configuration section for yarn.scheduler.capacity..capacity and yarn.scheduler.capacity..max-capacity, administrator could specify an absolute resource value like [memory=10240,vcores=12]. This is a valid configuration which indicates 10GB Memory and 12 VCores.

Running and Pending Application Limits

The CapacityScheduler supports the following parameters to control the running and pending applications:

PropertyDescription
yarn.scheduler.capacity.maximum-applications / yarn.scheduler.capacity..maximum-applicationsMaximum number of applications in the system which can be concurrently active both running and pending. Limits on each queue are directly proportional to their queue capacities and user limits. This is a hard limit and any applications submitted when this limit is reached will be rejected. Default is 10000. This can be set for all queues with yarn.scheduler.capacity.maximum-applications and can also be overridden on a per queue basis by setting yarn.scheduler.capacity..maximum-applications. Integer value expected.
yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity..maximum-am-resource-percentMaximum percent of resources in the cluster which can be used to run application masters - controls number of concurrent active applications. Limits on each queue are directly proportional to their queue capacities and user limits. Specified as a float - ie 0.5 = 50%. Default is 10%. This can be set for all queues with yarn.scheduler.capacity.maximum-am-resource-percent and can also be overridden on a per queue basis by setting yarn.scheduler.capacity..maximum-am-resource-percent

更多配置

参考:https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

如果想要不重启集群来动态刷新hadoop配置可尝试如下方法:

1、刷新hdfs配置

在两个(以三节点的集群为例)namenode节点上执行:

hdfs dfsadmin -fs hdfs://node1:9000 -refreshSuperUserGroupsConfigurationhdfs dfsadmin -fs hdfs://node2:9000 -refreshSuperUserGroupsConfiguration

2、刷新yarn配置

在两个(以三节点的集群为例)namenode节点上执行:

yarn rmadmin -fs hdfs://node1:9000 -refreshSuperUserGroupsConfigurationyarn rmadmin -fs hdfs://node2:9000 -refreshSuperUserGroupsConfiguration

参考

•https://stackoverflow.com/questions/33465300/why-does-yarn-job-not-transition-to-running-state•https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html•https://stackoverflow.com/questions/29917540/capacity-scheduler•https://cloud.tencent.com/developer/article/1357111•https://cloud.tencent.com/developer/article/1194501

References

[1] stackoverflow: https://stackoverflow.com/questions/33465300/why-does-yarn-job-not-transition-to-running-state

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/455325.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

MPEG-2TS码流编辑的原理及其应用(转载

[作者:辽宁电视台 赵季伟] 在当今数字媒体不断发展、新媒体业务不断涌现 的前提下,实践证明襁褓中的新媒体只有两种经营方略可供选择:或是购买并集成整套节目,或是低成本深加工新节目,再不可能去按照传统生产模式…

Python中的yield详解

阅读别人的python源码时碰到了这个yield这个关键字,各种搜索终于搞懂了,在此做一下总结: 通常的for…in…循环中,in后面是一个数组,这个数组就是一个可迭代对象,类似的还有链表,字符串&#xf…

shell循环结构之while循环

while循环 1) while CONDITION; dostatementstatement<改变循环条件真假的语句>done 编写脚本&#xff0c;计算1---100的和 #!/bin/bash#sum0i1while [ $i -le 100 ]; dolet sum$sum$ilet i$i1doneecho $sum2) while true; do statementstatementdone #!/bin/bash#while …

python 管道队列_关于python:Multiprocessing-管道与队列

Python的多处理程序包中的队列和管道之间的根本区别是什么&#xff1f;在什么情况下应该选择一种&#xff1f; 什么时候使用Pipe()有优势&#xff1f; 什么时候使用Queue()有优势&#xff1f;Pipe()只能有两个端点。Queue()可以有多个生产者和消费者。何时使用它们如果需要两个…

pip默认使用国内镜像地址

很多小伙伴在ubuntu系统下,使用pip安装会很慢 以为安装源在国外服务器上面 今天小编就教大家配置成让pip默认从国内源中寻找安装包 首先CtrlAltT打开终端 进入家目录 cd ~在家目录中创建一个文件夹,命名为.pip mkdir .pip进入目录,并创建一个名为pip.conf的文件 cd .pip…

“大型票务系统”和“实物电商系统”的数据库选型

讨论请移步至&#xff1a;http://www.zhiliaotech.com/ideajam/idea/detail/423 相关文章&#xff1a; 《今天你买到票了吗&#xff1f;——从铁道部12306.cn站点漫谈电子商务站点的“海量事务快速处理”系统》 不能简单套用“实物电商系统”对“大型票务系统”做需求分析 “大…

FLV文件格式(Z)(转载)

刚才在看一些关于demux的东西&#xff0c;在处理flv格式的文件的时候&#xff0c;由于自己对flv文件的格式不了解&#xff0c;所以就比较云头转向&#xff0c;正好看到了一篇讲述flv文件格式的文章&#xff0c;写的比较明白&#xff0c;所以就转过来了。O(∩_∩)O~flv头文件比较…

mysql-5.7中的innodb_buffer_pool_prefetching(read-ahead)详解

一、innodb的read-ahead是什么&#xff1a; 所谓的read-ahead就是innodb根据你现在访问的数据&#xff0c;推测出你接下来可能要访问的数据&#xff0c;并把它们(可能要访问的数据)读入 内存。 二、read-ahead是怎么做到的&#xff1a; 1、总的来说read-ahead利用的是程序的局部…

python compare excel_python简单操作excle的方法

Python操作Excle文件&#xff1a;使用xlwt库将数据写入Excel表格&#xff0c;使用xlrd 库从Excel读取数据。从excle读取数据存入数据库1、导入模块&#xff1a;import xlrd2、打开excle文件&#xff1a;data xlrd.open_workbook(excel.xls)3、获取表、行/列值、行/列数、单元值…

collections系列

class Counter(dict):  Counter类继承dict类、继承了dict的所有功能计数器&#xff1a; 例&#xff1a;import collections obj collections.Counter(sdkasdioasdjoasjdoasd) print(obj)得&#xff1a;Counter({s: 5, d: 5, a: 4, o: 3, j: 2, k: 1, i: 1}) 拿到前几位&…

Python中的虚拟环境-virtualenv

更低层次: virtualenv virtualenv 是一个创建隔绝的Python环境的 工具。virtualenv创建一个包含所有必要的可执行文件的文件夹&#xff0c;用来使用Python工程所需的包。 它可以独立使用&#xff0c;代替Pipenv。 通过pip安装virtualenv&#xff1a; $ pip install virtual…

mp4文件格式解析(一)

原文地址&#xff1a;mp4文件格式解析&#xff08;一&#xff09;作者&#xff1a;可下人间目前MP4的概念被炒得很火&#xff0c;也很乱。最开始MP4指的是音频&#xff08;MP3的升级版&#xff09;&#xff0c;即MPEG-2 AAC标准。随后MP4概念被转移到视频上&#xff0c;对应的是…

shiro身份验证测试

2019独角兽企业重金招聘Python工程师标准>>> 一、登录验证 1、首先在shiro.ini里准备一些用户身份/凭据&#xff0c;后面这里会使用数据库代替&#xff0c;如&#xff1a; [users] [main] #realm jdbcRealmcom.learnging.system.shiro.ShiroRealm securityManager…

shell if多个条件判断_萌新关于Excel VBA中IF条件判断语句的一点心得体会

作者:金人瑞 《Excel VBA175例无理论纯实战教程》学员最近正在学习郑广学老师的VBA 175例教程&#xff0c;这是一篇新手向的文章&#xff0c;也是一个新手的总结&#xff0c;高手可以批评文章中的不足之处&#xff0c;也可以无视&#xff0c;VBA中的IF判断, 判断一般起到控制作…

Django笔记01-基础:一个完美主义的web框架

浅谈Web框架 一,什么是框架? 软件框架就是为实现或完成某种软件开发时,提供了一些基础的软件产品, 框架的功能类似于基础设施,提供并实现最为基础的软件架构和体系 通常情况下我们依据框架来实现更为复杂的业务程序开发 一个字,框架就是程序的骨架 二,框架的优缺点 可重…

mysql存储引擎的一点学习心得总结

首先我们应该了解mysql中的一个重要特性——插件式存储引擎&#xff0c;从名字就能够看出在mysql中&#xff0c;用户能够依据自己的需求随意的选择存储引擎。实际上也是这样。即使在同一个数据库中。不同的表也能够使用不同的存储引擎。Mysql中支持的存储引擎有非常多种&#x…

常见音视频格式(转载)

Contents 1 MPEG 系列 1.1 MPEG-1 1.2 MPEG-2 1.3 MPEG-4 1.4 MPEG-4 AVC 1.5 MPEG Audio Layer 1/2 1.6 MPEG Audio Layer 3 1.7 MPEG-2 AAC 1.8 MPEG-4 AAC 1.9 MPEG-4 aacPlus 1.10 MPEG-4 VQF 1.11 mp3PRO 1.12 MP3 Surround 2 DVD系列 2.1 Dolby Digital AC3 2.2 Dolby D…

编程语言难度排名_谷歌排名第一的编程语言,小学生拿来做答题,分分钟钟搞定高难度算法!...

点击上方蓝色文字关注我们吧谷歌排名第一的编程语言时什么&#xff1f;毫无疑问&#xff1a;肯定是 Python。 也难怪&#xff0c;作为大数据时代和人工智能时代的必备语言&#xff0c;Python 的优点太多了&#xff0c;语言简洁、易学、开发效率高、可移植性强...... 另外&#…

poj 2484 A Funny Game

题目&#xff1a;http://poj.org/problem?id2484 一&#xff0c;题意&#xff1a; n个硬币围成一个圈&#xff0c;Alice与Bob轮流从圈中取硬币。每次能够取一枚或者连续的两枚。 硬币取走后留下的空位不用填补&#xff0c;空位相隔的两个硬币视为不相邻。Alice第一个開始取。 …

58到家MySQL军规升级版

一、基础规范 表存储引擎必须使用InnoDB 表字符集默认使用utf8&#xff0c;必要时候使用utf8mb4 解读&#xff1a; &#xff08;1&#xff09;通用&#xff0c;无乱码风险&#xff0c;汉字3字节&#xff0c;英文1字节 &#xff08;2&#xff09;utf8mb4是utf8的超集&#…