日志报错（WARN类型最后执行成功可以忽略）：

19/04/15 12:35:37 INFO memory.MemoryStore: Will not store rdd_2_5119/04/15 12:35:37 WARN memory.MemoryStore: Not enough space to cache rdd_2_51 in memory! (computed 1109.7 MB so far)19/04/15 12:35:37 INFO memory.MemoryStore: Memory use = 627.7 MB (blocks) + 1109.7 MB (scratch space shared across 1 tasks(s)) = 1737.4 MB. Storage limit = 2004.6 MB.19/04/15 12:35:37 WARN storage.BlockManager: Persisting block rdd_2_51 to disk instead.19/04/15 13:01:46 INFO zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x331abd57 to fwqml016.zh:2181,fwqml018.zh:2181,fwqml009.zh:218119/04/15 13:01:49 INFO memory.MemoryStore: Will not store rdd_2_5119/04/15 13:01:49 WARN memory.MemoryStore: Not enough space to cache rdd_2_51 in memory! (computed 1109.7 MB so far)19/04/15 13:01:49 INFO memory.MemoryStore: Memory use = 627.7 MB (blocks) + 1109.7 MB (scratch space shared across 1 tasks(s)) = 1737.4 MB. Storage limit = 2004.6 MB.

webUI页面报错：

异常提示内存空间不足

解决方法：

1.将"spark.yarn.executor.memoryOverhead"设置为最大值，可以考虑一下4096。这个数值一般都是2的次幂。

2.将rdd进行重新分区，这里可以考虑200k。在spark2.3的版本中，rdd成为了dateframe格式的数据。

3.将"executor.cores"从8设置为4。将core的个数调小，防止cpu不足。

4.将"executor.memory"从8g设置为12g。将内存调大。

 

一般引起内存的问题都可使用该参数进行调整，

如常见异常：

WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 2.2 GB of 2.1 GB virtual memory used. Consider boosting spark.yarn.executor.memoryOverhead.

当executor的内存使用大于executor-memory与executor.memoryOverhead的加和时，Yarn会干掉这些executor，

修改上面描述参数也可以解决该问题

 

 

最初导致异常提交的配置为：

spark-submit \
--master yarn \
--deploy-mode cluster \
--class com.data.filter.ArticleFilter \
--jars $(echo ./lib/*.jar | tr ' ' ',') \
--num-executors 50 \
--executor-cores 2 \
--conf spark.yarn.queue=etl \
--executor-memory 4g \
--driver-memory 1g \
--conf spark.kryoserializer.buffer.max=2000 \
--conf spark.akka.frameSize=500 \
--conf spark.sql.shuffle.partitions=100 \
--conf spark.default.parallelism=100 \
--conf spark.storage.memoryFraction=0.3 \
--conf spark.shuffle.memoryFraction=0.7 \
--conf spark.shuffle.safetyFraction=0.8 \
--conf spark.shuffle.spill=true \
--conf spark.yarn.queue=etl \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
./data-filter.jar

修改Executor-memory后执行成功的参数，修改为 --executor-memory 8g 

spark-submit \
--master yarn \
--deploy-mode cluster \
--class com.data.filter.ArticleFilter \
--jars $(echo ./lib/*.jar | tr ' ' ',') \
--num-executors 50 \
--executor-cores 2 \
--conf spark.yarn.queue=etl \
--executor-memory 8g \
--driver-memory 1g \
--conf spark.kryoserializer.buffer.max=2000 \
--conf spark.akka.frameSize=500 \
--conf spark.sql.shuffle.partitions=100 \
--conf spark.default.parallelism=100 \
--conf spark.storage.memoryFraction=0.3 \
--conf spark.shuffle.memoryFraction=0.7 \
--conf spark.shuffle.safetyFraction=0.8 \
--conf spark.shuffle.spill=true \
--conf spark.yarn.queue=etl \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
./data-filter.jar

或修改--conf spark.yarn.executor.memoryOverhead=4096参数同样也执行成功了

spark-submit \
--master yarn \
--deploy-mode cluster \
--class com.data.filter.ArticleFilter \
--jars $(echo ./lib/*.jar | tr ' ' ',') \
--num-executors 50 \
--executor-cores 2 \
--conf spark.yarn.queue=etl \
--executor-memory 4g \
--driver-memory 1g \
--conf spark.yarn.executor.memoryOverhead=4096 \
--conf spark.kryoserializer.buffer.max=2000 \
--conf spark.akka.frameSize=500 \
--conf spark.sql.shuffle.partitions=100 \
--conf spark.default.parallelism=100 \
--conf spark.storage.memoryFraction=0.3 \
--conf spark.shuffle.memoryFraction=0.7 \
--conf spark.shuffle.safetyFraction=0.8 \
--conf spark.shuffle.spill=true \
--conf spark.yarn.queue=etl \
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
./data-filter.jar