YARN-site.xml相关:
yarn.scheduler.minimum-allocation-mb =1024
yarn.scheduler.maximum-allocation-mb =8192
yarn.nodemanager.vmem-pmem-ratio =2.1
yarn.nodemanager.resource.memory.mb =24576
Map-site.xml相关参数
mapreduce.map.java.opts =Xmx1024m
mapreduce.map.memory.mb =1536
mapreduce.reduce.java.opts =Xmx2560m
mapreduce.reduce.memory.mb =3072
map container的内存("mapreduce.map.memory.mb")被设置为1536mb 。am为其申请了2048m的内存,因为am的最小分配(yarn.scheduler.minimum-allocation-mb)是1024.如果map task对内存超过了2048m,nm会kill掉该task。reduce类似。
When a mapreduce job completes you will see several counters dumped at the end of the job.
The three memory counters below show how much physical memory was allocated vs virtual memory.
默认情况下,yarn.nodemanager.vmem-pmem-ratio被设置为2.1,这意味着,每个map或者task任务只能使用2.1倍("mapreduce.reduce.memory.mb") or ("mapreduce.map.memory.mb") 大小的虚拟内存,否则,会被nm杀掉。
例如下面的的报错信息:
Current usage: 2.1gb of 2.0gb physical memory used; 1.6gb of 3.15gb virtual memory used.
Killing container.