ERROR YarnScheduler: Lost executor 111 hostname: Container killed by YARN for exceeding memory limits 14.0 GB of 14 GB physical memory used. Consider boost in spark.yarn.executor.memoryOverhead.
spark.executor.memory + spark.yarn.executor.memoryOverhead =15G
memory和memoryOverhead加起来有个最大值限制，增大memoryOverhead的值，则相应memory的值要减小
解决方法：
1.增大memoryOverhead的值
- 减少可用的Core的数量 N, 使并行任务数减少，从而减少Overhead开销
--conf spark.executor.cores=3 <比原来小的值>
3.增大partition的数目。
dataframe.repartition((numPartitions, *cols)
