Spark 1.6.3安装
设置主机名
[root@localhost ~]# hostnamectl set-hostname sam01
修改hosts文件，加入主机名
[root@sam01 ~]# vi /etc/hosts
127.0.0.1 sam01 sam01
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
查看主机名
[root@sam01 ~]# hostname
sam01
修改当前用户.bash_profile
[sam@localhost ~]$ vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export JAVA_HOME=$HOME/tools/jdk1.7.0_80
export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=$HOME/tools/scala-2.11.8
export MAVEN_HOME=$HOME/tools/apache-maven-3.3.9
export SBT_HOME=$HOME/tools/sbt
export HADOOP_HOME=$HOME/tools/hadoop-2.6.5
export HADOOP_CONF_DIR=$HOME/tools/hadoop-2.6.5/etc/hadoop
export SPARK_HOME=$HOME/tools/spark-1.6.3-bin-hadoop2.6
PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$SCALA_HOME/bin:$JAVA_HOME/bin:$MAVEN_HOME/bin:$SBT_HOME/bin:$PATH
export PATH
使配置文件立即生效
[sam@localhost ~]$ source .bash_profile
检查环境
[sam@localhost ~]$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
[sam@localhost ~]$ scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
[sam@localhost ~]$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /home/sam/tools/apache-maven-3.3.9
Java version: 1.7.0_80, vendor: Oracle Corporation
Java home: /home/sam/tools/jdk1.7.0_80/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"
[sam@localhost ~]$ sbt -batch sbt-version
[info] Loading project definition from /home/sam/project
[info] Set current project to sam (in build file:/home/sam/)
[info] 0.13.15
修改Spark配置文件
[sam@localhost conf]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/conf
[sam@localhost conf]$ cp spark-env.sh.template spark-env.sh
在spark-env.sh最后添加下面
export SCALA_HOME=/home/sam/tools/scala-2.11.8
export SPARK_MASTER_IP=sam01
export SPARK_WORKER_MEMORY=1G
export JAVA_HOME=/home/sam/tools/jdk1.7.0_80
配置slaves,在slaves最后添加主机名
[sam@localhost conf]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/conf
[sam@localhost conf]$ cp slaves.template slaves
sam01
启动master
[sam@localhost sbin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/sbin
[sam@sam01 sbin]$ ./start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/sam/tools/spark-1.6.3-bin-hadoop2.6/logs/spark-sam-org.apache.spark.deploy.master.Master-1-sam01.out
启动worker
[sam@localhost sbin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/sbin
[sam@localhost sbin]$ ./start-slaves.sh spark://sam01:7077
sam01: starting org.apache.spark.deploy.worker.Worker, logging to /home/sam/tools/spark-1.6.3-bin-hadoop2.6/logs/spark-sam-org.apache.spark.deploy.worker.Worker-1-sam01.out
启动Shell
[sam@sam01 bin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/bin
[sam@sam01 bin]$ ./spark-shell --master spark://sam01:7077
访问http://sam01:8080/
(如不能通过主机名访问，可修改hosts文件加入ip与主机名对应关系)查看状态
ssh配置
[sam@sam01 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sam/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sam/.ssh/id_rsa.
Your public key has been saved in /home/sam/.ssh/id_rsa.pub.
The key fingerprint is:
aa:a9:c1:26:7b:3c:25:a7:f2:1a:c3:10:5d:50:8a:2b sam@sam01
The key's randomart image is:
+--[ RSA 2048]----+
| .oo |
| o o |
|o o |
| o |
|E S |
|+.. o . |
|+o+= . |
|.*=. o |
|o=ooo |
+-----------------+
[sam@sam01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub sam@sam01
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sam@sam01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'sam@sam01'"
and check to make sure that only the key(s) you wanted were added.
[sam@sam01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub sam@sam02
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sam@sam02's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'sam@sam02'"
and check to make sure that only the key(s) you wanted were added.
