1 spark核心概念
1 Application
基于Spark的应用程序 =1driver+n executors
User program built on Spark.
Consists of a driver program and executors on the cluster.
例如：一个py脚本，或者pyshark/spark-shell
2 Application jar
在java或者scala开发中会有，python中较为少见
A jar containing the user's Spark application. In some cases users will want to create an "uber jar" containing their application along with its dependencies. The user's jar should never include Hadoop or Spark libraries, however, these will be added at runtime.
3 Driver program
运行一个main方法并创建一个sc
The process running the main() function of the application and creating the SparkContext
4 Cluster manager
获取外部资源的，可以指定需要的资源量
An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN)
一个app需要申请，driver的内存，executors的内存，
5 Deploy mode
部署模式——driver 运行的位置
Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.
6 Worker node
standalone 相当于slave节点，yarn相当于node manager
Any node that can run application code in the cluster
7 Executor
A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.
在不同的app中是不共用的。
8 Task
A unit of work that will be sent to one executor
map、shuffle
9 Job
一个action对应一个job
A parallel computation consisting of multiple tasks that gets spawned in response to a Spark action (e.g. save, collect); you'll see this term used in the driver's logs.
10 stage
一个stage的边界是从某个地方取数据开始到suffle结束
Each job gets divided into smaller sets of tasks called stagesthat depend on each other (similar to the map and reduce stages in MapReduce); you'll see this term used in the driver's logs.
2 spark 运行架构以及注意事项
【重要】
Spark applications run as independent sets of processes on a cluster, coordinated by the SparkContext object in your main program (called the driver program).
Specifically, to run on a cluster, the SparkContext can connect to several types ofcluster managers (either Spark’s own standalone cluster manager, Mesos or YARN), which allocate resources（分配资源） across applications.
Step1 Once connected, Spark acquires executors on nodes in the cluster, which are processes that run computations and store data for your application.
完成链接后，spark的任务就会链接到executors，executors会负责处理任务和数据存储
Step2 Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors.
接下来，SC会把代码发送到executors下
Step3 Finally, SparkContext sends tasks to the executors to run.
SparkContext 会把 tasks发送到executors 去执行。相当于是具体的任务执行的命令。
Each application gets its own executor processes, which stay up for the duration of the whole application and run tasks in multiple threads. This has the benefit of isolating applications from each other, on both the scheduling side (each driver schedules its own tasks) and executor side (tasks from different applications run in different JVMs). However, it also means that data cannot be shared across different Spark applications (instances of SparkContext) without writing it to an external storage system.
每一个都是独立的线程，应用程序之间是隔离的。数据不能跨应用程序共享。
Spark is agnostic to the underlying cluster manager. As long as it can acquire executor processes, and these communicate with each other, it is relatively easy to run it even on a cluster manager that also supports other applications (e.g. Mesos/YARN).
spark不关心底层运行的节点
The driver program must listen for and accept incoming connections from its executors throughout its lifetime (e.g., see spark.driver.port in the network config section). As such, the driver program must be network addressable from the worker nodes.
有监听机制，防止运行错误
Because the driver schedules tasks on the cluster, it should be run close to the worker nodes, preferably on the same local area network. If you’d like to send requests to the cluster remotely, it’s better to open an RPC to the driver and have it submit operations from nearby than to run a driver far away from the worker nodes.
driver要尽可能近的靠近节点，想要发送远程请求需要开一个RPC
3 spark和hadoop重要概念区分
hadoop
1 一个MR程序-一个Job
2 一个Job =1个N个Task（Map/Reduce）
3 一个Task对对应于一个进程
4 Task运行时开启进程，Task执行完毕后销毁进程，对于多个Task，开销较大 （JVM共享对其无效）
Spark
1 Application= Driver（main方法创建SparkContext）+Executor
2 一个Application = 0~n个Job
3 一个Job = 一个Action
4 一个Job =1~n个Stage
5 一个Stage = 1~n个Task
6 一个Task 有一个线程，多个Task可以并行的方式运行在一个Executor进程中
4 Spark Cache详情
调用方法：rdd.cache()/persist()
如果启动这个机制，会把storageLevel改正memory only
使用lazy机制：没有遇到action 不提交作业
如果一个RDD在后续的计算中会被使用，建议Cache
from pyspark import StorageLevel
lines= sc.textFile("file:///home/hadoop/data/page_views.dat")
lines.count()
lines.cache()
lines.count()
lines.persist(StorageLevel.MEMORY_ONLY_2)
如果做了持久化，这样读取只有一次
rdd.unprisist()是非lazy的
RDD Persistence
One of the most important capabilities in Spark is persisting (or caching) a dataset in memory across operations. When you persist an RDD, each node stores any partitions of it that it computes in memory and reuses them in other actions on that dataset (or datasets derived from it). This allows future actions to be much faster (often by more than 10x). Caching is a key tool for iterative algorithms and fast interactive use.
cache方法底层是persist，缓存是有容错机制的，支持多副本。
如果内存较小可以用序列化的方式，较为节省内存。
persist的StorageLevel参数：MEMORY_ONLY, MEMORY_ONLY_2, MEMORY_AND_DISK, MEMORY_AND_DISK_2, DISK_ONLY, and DISK_ONLY_2.
持久化策略：内存，单副本，序列化
ps 查看存储情况：// 192.168.199.102:4040
5 Spark Lineage详情
Lineage：RDD的依赖关系
在容错机制中，按照partition计算，无需全部计算。
问1：如果没有持久化，是不是如果第二个partition丢失了，还是的从磁盘里读取后再计算？
问2：持久化的计算消耗大么？
6 Spark Dependancy
narrow窄依赖-pipline-abe
一个父RDD的分区最多被一个字RDD使用
——map，filiter，union，join with inputs co-partitioned
wide宽依赖-shuffle
一个父RDD会被子RDD使用多次
——groupbykey on non-partitioned data，join with inputs not co-partitioned，repartition，coclesce，cogroup，reducebykey
The reduceByKey operation generates a new RDD where all values for a single key are combined into a tuple - the key and the result of executing a reduce function against all values associated with that key.
The Shuffle is an expensive operation since it involves disk I/O, data serialization, and network I/O. T
区别
如果是窄依赖，如果错了只需要重算部分父RDD，宽依赖要重新计算所有RDD。
问：reducebykey之后会重新分配数据，那么partition输出的partition应该分几块？
写代码作业
lines = sc.textFile
words = flatMap
pairs = map(word,1)
reduceByKey
sc.textFile().flatmap().map().reduceByKey()
可以看到两个stage
【作业】：官网suffle内容阅读
tips
进程——process
线程——threads
血源——lineage
