一、简介
Spark是UC Berkeley AMPLab开发的类MapRed计算框架。MapRed框架适用于batch job，但是由于它自身的框架限制，第一，pull-based heartbeat作业调度。第二，shuffle中间结果全部落地disk，导致了高延迟，启动开销很大。
而Spark是为迭代式，交互式计算所生的。第一，它采用了actor model的akka作为通讯框架。第二，它使用了RDD分布式内存，操作之间的数据不需要dump到磁盘上，而是通过RDD Partition分布在各个节点内存中，极大的提高了数据间的流转，同时RDD之间维护了血统关系，一旦RDD fail掉了，能通过父RDD自动重建，保证了fault tolerance。
而且在Spark之上有丰富的应用，比如Shark，Spark Streaming，MLBase。我们在生产环境中已经使用Shark来作为Hive的一种补充，它共享了hive 的metastore，serde，使用方式也和hive几乎一样，如果data input size不是很大的情况下，相同语句确实比hive会快很多。
二、安装部署
-
下载安装配置Scala
[root@master ~]# wget https://downloads.lightbend.com/scala/2.12.2/scala-2.12.2.tgz [root@master spark]# tar xvf scala-2.12.2.tgz -C /usr/local/program/scala/ #在etc/profile中增加环境变量SCALA_HOME，并使之生效： vim /etc/profile export SCALA_HOME=/usr/local/program/scala/scala-2.12.2 export PATH=$PATH:$SCALA_HOME/bin [root@master spark]# . /etc/profile -
下载安装配置Spark
#因为我现有的hadoop是2.7.1版本，故... [root@master spark]# wget https://d3kbcqa49mib13.cloudfront.net/spark-2.1.1-bin-hadoop2.7.tgz [root@master spark]# tar xvf spark-2.1.1-bin-hadoop2.7.tgz -C /usr/local/program/spark/ #在etc/profile中增加环境变量SPARK_HOME，并使之生效： export SPARK_HOME=/usr/local/program/spark/spark-2.1.1-bin-hadoop2.7 export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin [root@master spark]# . /etc/profile #在m1上配置Spark，修改spark-env.sh配置文件 #进入spark的conf目录 [root@master spark]# cd /usr/local/program/spark/spark-2.1.1-bin-hadoop2.7/conf/ [root@master conf]# cp spark-env.sh.template spark-env.sh [root@master conf]# cat spark-env.sh export SCALA_HOME=/usr/local/program/scala/scala-2.12.2 export HADOOP_HOME=/home/hadoop/hadoop-2.7.3 export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64 export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop #export SPARK_JAR=/usr/local/program/spark/ export SPARK_MASTER_IP=master #修改conf/slaves文件，将计算节点的主机名添加到该文件，一行一个 # 这里应该包含master，将master也同时作为一个计算节点 [root@master conf]# cat slaves slave01 slave02 slave03 slave04 配置ssh免密码登陆
-
复制到集群节点
[root@master conf]# scp /etc/profile slave01:/etc/ [root@master conf]# scp -r /usr/local/program/spark/spark-2.1.1-bin-hadoop2.7 slave02:/usr/local/program/spark/ [root@master conf]# scp -r /usr/local/program/scala/scala-2.12.2/ slave02:/usr/local/program/scala/ -
启动master和slaves
[root@master conf]# start-master.sh [root@master conf]# start-slaves.sh -
通过web端口访问spark
http://master:8080
三、 运行简单的example
-
单机运行
#计算圆周率 [root@master spark-2.1.1-bin-hadoop2.7]# ./bin/run-example SparkPi 10 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 17/06/05 19:19:00 INFO SparkContext: Running Spark version 2.1.1 17/06/05 19:19:00 WARN SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0 17/06/05 19:19:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/06/05 19:19:00 INFO SecurityManager: Changing view acls to: root 17/06/05 19:19:00 INFO SecurityManager: Changing modify acls to: root 17/06/05 19:19:00 INFO SecurityManager: Changing view acls groups to: 17/06/05 19:19:00 INFO SecurityManager: Changing modify acls groups to: 17/06/05 19:19:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() ... 17/06/05 19:19:02 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 0.761265 s Pi is roughly 3.143967143967144 -
spark-shell的简单使用
[root@master spark-2.1.1-bin-hadoop2.7]# spark-shell scala> val s=sc.textFile("hdfs://master:9000/user/hadoop/test/Temperature.txt") s: org.apache.spark.rdd.RDD[String] = hdfs://master:9000/user/hadoop/test/Temperature.txt MapPartitionsRDD[3] at textFile at <console>:24 scala> s.count res1: Long = 11 [hadoop@slave02 ~]$ hdfs dfs -cat test/Temperature.txt 2015,1,24 2015,3,56 2015,1,3 2015,2,-43 2015,4,5 2015,3,46 2014,2,64 2015,1,4 2015,1,21 2015,2,35 2015,2,0 -
集群提交作业
[hadoop@master ~]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 /usr/local/program/spark/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar 100 17/06/06 16:03:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/06/06 16:03:22 INFO client.RMProxy: Connecting to ResourceManager at master/10.10.18.229:8032 17/06/06 16:03:22 INFO yarn.Client: Requesting a new application from cluster with 4 NodeManagers 17/06/06 16:03:22 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) ... 17/06/06 16:04:33 INFO yarn.Client: Application report for application_1494595290830_0061 (state: RUNNING) 17/06/06 16:04:34 INFO yarn.Client: Application report for application_1494595290830_0061 (state: RUNNING) 17/06/06 16:04:35 INFO yarn.Client: Application report for application_1494595290830_0061 (state: RUNNING) 17/06/06 16:04:36 INFO yarn.Client: Application report for application_1494595290830_0061 (state: FINISHED) 17/06/06 16:04:36 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.10.19.232 ApplicationMaster RPC port: 0 queue: default start time: 1496736259866 final status: SUCCEEDED tracking URL: http://master:8088/proxy/application_1494595290830_0061/ user: hadoop 17/06/06 16:04:36 INFO util.ShutdownHookManager: Shutdown hook called 17/06/06 16:04:36 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-6039cb14-8084-404e-b970-633dff4dd086
