Spark Standalone Cluster环境搭建

Spark Standalone Cluster环境搭建过程中的一些命令代码有必要记录一下，正好看到以下这篇文章转载过来了。

1.在master虚拟机设置spark-env.sh

（1）复制模板文件来创建spark-env.sh

cp /usr/local/spark/conf/spark-env.sh.template /usr/local/spark/conf/spark-env.sh

（2）修改spark-env.sh

sudo vim /usr/local/spark/conf/spark-env.sh

export SPARK_MASTER_IP=master

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=128m

export SPARK_WORKER_INSTANCES=4

2.复制spark程序到data1、data2、data3

（1）复制spark程序到data1

ssh data1

sudo mkdir /usr/local/spark

sudo chown hduser:hduser /usr/local/spark

exit

sudo scp -r /usr/local/spark hduser@data1:/usr/local

（2）复制spark程序到data2

ssh data2

sudo mkdir /usr/local/spark

sudo chown hduser:hduser /usr/local/spark

exit

sudo scp -r /usr/local/spark hduser@data2:/usr/local

（3）复制spark程序到data3

ssh data3

sudo mkdir /usr/local/spark

sudo chown hduser:hduser /usr/local/spark

exit

sudo scp -r /usr/local/spark hduser@data3:/usr/local

3.在master虚拟机编辑slaves文件

1.修改slaves文件

sudo vim /usr/local/spark/conf/slaves

data1

data2

data3

4.启动Spark Standalone Cluster

/usr/local/spark/sbin/start-all.sh

或

/usr/local/spark/sbin/start-master.sh

/usr/local/spark/sbin/start-slaves.sh

5.在spark standalone运行pyspark

pyspark --master spark://master:7077 --num-executors 1 --total-executor-cores 3 --executor-memory 512m

---------------------

作者：剑海风云

来源：CSDN

原文：https://blog.csdn.net/nanxiaotao/article/details/90482958