learn spark

内容来源：spark source code
1: spark 输入数据的默认task 个数：
解答：分如下情况：
Rdd:
Hadoopfile 计算分片，传递了一个参数 parallelism
Sc.parallelize() 默认值是：spark.default.parallelism
Local mode: number of cores on the local machine

Paste_Image.png

Mesos fine grained mode: 8

Paste_Image.png

Others: total number of cores on all executor nodes or 2, whichever is larger
Because:YarnSchedulerBackend 继承自CoarseGrainedSchedulerBackend

Paste_Image.png

Spark 2.0 中大量使用的Dataset
ExecutedCommandExec

2： 慎用 groupBykey ，可能导致oom

• Note: As currently implemented, groupByKey must be able to hold all the key-value pairs for any
• key in memory. If a key has too many values, it can result in an [[OutOfMemoryError]].
  It’s recommended to use PairRDDFunctions.aggregateByKey