hadoop集群搭建完毕，spark-standalone集群搭建结束后，能正常启动
bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://hadoop002:7077 \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10
这时候开始配置历史服务：
spark-shell停止后，集群监控hadoop002:4040就看不到历史任务的运行情况，所以开发时配置历史服务器记录任务运行情况。
修改【spark-defaults.conf.template】去掉template，并加上spark.eventLog.enabled true spark.eventLog.dir hdfs://hadoop002:8020/directory #directory要保证事先存在于hdfs上
修改【spark-env.sh】文件，添加日志配置
export SPARK_HISTORY_OPTS=" -Dspark.history.ui.port=18080 -Dspark.history.fs.logDirectory=hdfs://hadoop002:8020/directory -Dspark.history.retainedApplications=30"
[root]分发配置：standlone目录下，xsync conf/
[childe-h]启动hadoop集群，并保证directory目录的存在sbin/start-dfs.sh #存在则不用创建。首次创建后，之后对dfs进行-format格式化才需要再次创建 hadoop fs -mkdir /directory
[childe-h]启动集群：standlone目录下，
sbin/start-all.sh sbin/start-history-server.sh #启动历史服务器
启动报错
21/5/19 16:30:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/5/19 16:30:33 ERROR cluster.StandaloneSchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
21/5/19 16:30:33 ERROR netty.Inbox: Ignoring error
org.apache.spark.SparkException: Exiting due to error from cluster scheduler: Master removed our application: FAILED
at org.apache.spark.scheduler.TaskSchedulerImpl.error(TaskSchedulerImpl.scala:459)
at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.dead(StandaloneSchedulerBackend.scala:139)
at ...
19/11/15 16:28:31 ERROR spark.SparkContext: Error initializing SparkContext.
...
有个worker日志，内容如下：
java.io.IOException: Failed to create directory /soft/spark/work/app-...
at org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.applyOrElse(Worker.scala:450)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
...
worker无法创建日志，到目录下一看，worker是root所属文件，而其他是自己用户的所属文件。
两种解决方法。
一个是把这个文件改用户，改成要使用的用户
chown xxx.xxx worker -R
另一个是修改spark-env.sh文件并分发，我在这个文件中指定了work目录，如下
# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
# - SPARK_NO_DAEMONIZE Run the proposed command in the foreground. It will not output a PID file.
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
#主节点的IP
export SPARK_MASTER_IP=b1
#主节点的端口号，用来与worker通信
export SPARK_MASTER_PORT=7077
#每一个worker进程所能管理的核数
export SPARK_WORKER_CORES=2
#每一个worker进程所能管理的内存数
export SPARK_WORKER_MEMORY=1G
#worker的工作目录区
export SPARK_WORKER_DIR=/usr/local/spark/spark-standlone
