RDD的存储机制：
-
其数据分布存储在多台机器上，都是以block的形式存储在服务器上。每个Executor都会启动一个BlockManagerSlave，并且管理一部分block;block的元数据保存在Driver的BlockManagerMaster上，当BlockManagerSlave产生block之后就会向BlockManagerMaster注册该block。RDD的partion是一个逻辑数据块，对应相应的物理数据块block。具体流程如下：
RDD数据存储机制
1.分区列表性
/**
* Implemented by subclasses to return the set of partitions in this RDD. This method will only
* be called once, so it is safe to implement a time-consuming computation in it.
*
* The partitions in this array must satisfy the following property:
* `rdd.partitions.zipWithIndex.forall { case (partition, index) => partition.index == index }`
*/
protected def getPartitions: Array [Partition]
2.依赖其他RDD，血缘关系
/**
* Implemented by subclasses to return how this RDD depends on parent RDDs. This method will only
* be called once, so it is safe to implement a time-consuming computation in it.
*/
protected def getDependencies: Seq [Dependency[_]]= deps
3.每个分区都有一个函数计算
/**
* :: DeveloperApi ::
* Implemented by subclasses to compute a given partition.
*/
@DeveloperApi
def compute(split: Partition, context: TaskContext): Iterator[T]
4.key-value的分区器
/** Optionally overridden by subclasses to specify how they are partitioned. */
@transient val partitioner: Option[Partitioner] = None
5.每个分区都有一个分区位置列表
/**
* Optionally overridden by subclasses to specify placement preferences.
* 可选的，输入参数是 split分片，返回的是一组优先的节点位置。例如hdfs的block位置
*/
protected def getPreferredLocations(split: Partition): Seq[String] = Nil
