Spark本地开发环境搭建

spark-logo-trademark.png

环境介绍

  • win10
  • java-1.8.0_131
  • scala-2.11.6
  • hadoop-2.9.1
  • spark-2.3.2-bin-hadoop2.7

在eclipse中创建一个springboot工程，并导入相关依赖

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.qunce</groupId>
  <artifactId>spark-study-one</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.2.RELEASE</version>
  </parent>
  <properties>
    <spark.version>2.3.2</spark.version>
    <scala.version>2.11</scala.version>
    <project.build.sourceEncoding>utf-8</project.build.sourceEncoding>
  <dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
     <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.9.1</version>
        <exclusions>
            <exclusion>
                 <groupId>io.netty</groupId>
                 <artifactId>netty</artifactId>
            </exclusion>
        </exclusions>
     </dependency>
     <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.9.1</version>
        <exclusions>
            <exclusion>
                 <groupId>io.netty</groupId>
                 <artifactId>netty</artifactId>
            </exclusion>
        </exclusions>
     </dependency>
     <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.9.1</version>
     </dependency>
     <dependency>
        <groupId>jdk.tools</groupId>
        <artifactId>jdk.tools</artifactId>
        <version>1.8</version>
        <scope>system</scope>
        <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
     </dependency>
     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.3.2</version>
        <exclusions>
            <exclusion>
                 <groupId>io.netty</groupId>
                 <artifactId>netty</artifactId>
            </exclusion>
        </exclusions>
     </dependency>
     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.3.2</version>
     </dependency>
     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
     </dependency>
     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>2.3.2</version>
     </dependency>
     <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.3.2</version>
     </dependency>
  </dependencies>
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.17.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>
  <!-- eclipse中使用maven创建项目JDK版本默认为1.5的 解决办法 -->
  <profiles>
    <profile>
        <id>jdk-1.8</id>
        <activation>
            <activeByDefault>true</activeByDefault>
            <jdk>1.8</jdk>
        </activation>
        <properties>
            <maven.compiler.source>1.8</maven.compiler.source>
            <maven.compiler.target>1.8</maven.compiler.target>
            <maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
        </properties>
    </profile>
  </profiles>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-install-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-install</id>
                        <phase>install</phase>
                        <goals>
                            <goal>install</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            
             <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
  </properties>

下面对pom文件进行说明

  1. eclipse默认的java版本为1.5，需要加入以下配置。
  <profiles>
    <profile>
        <id>jdk-1.8</id>
        <activation>
            <activeByDefault>true</activeByDefault>
            <jdk>1.8</jdk>
        </activation>
        <properties>
            <maven.compiler.source>1.8</maven.compiler.source>
            <maven.compiler.target>1.8</maven.compiler.target>
            <maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
        </properties>
    </profile>
  </profiles>
  1. 由于多个组件中都使用了netty，导致冲突，所有需要将版本较低的netty排除掉。
        <exclusions>
            <exclusion>
                 <groupId>io.netty</groupId>
                 <artifactId>netty</artifactId>
            </exclusion>
        </exclusions>

还需要将netty升级。

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.17.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>
  1. 解决打的jar包里面没有相关依赖。
             <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </plugin>
  1. 由于eclipse默认的编码环境为GBK，需要修改为UTF-8。我的报了一下错误。
maven [ERROR] 26206
[ERROR] To see the full stack trace of the errors, re-run Maven with the –e
  1. 运行环境还需要添加jdk中的jar包
     <dependency>
        <groupId>jdk.tools</groupId>
        <artifactId>jdk.tools</artifactId>
        <version>1.8</version>
        <scope>system</scope>
        <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
     </dependency>

编写测试代码

package com.qunce;

import java.util.Arrays;
import java.util.Iterator;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

/**
 * 集群测试的wordcount程序
 * @author m1896
 *
 */
public class WordCountLocal {

    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("WordCountLocal").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);
        JavaRDD<String> lines =sc.textFile("G:\\StockDetail.txt");
        JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {

            private static final long serialVersionUID = 1L;

            @Override
            public Iterator<String> call(String line) throws Exception {
                return  Arrays.asList(line.split(",")).iterator();
            }
        });
        
        JavaPairRDD<String, Integer> pairs = words.mapToPair(
                new PairFunction<String, String, Integer>() {

                    private static final long serialVersionUID = 1L;

                    @Override
                    public Tuple2<String, Integer> call(String word) throws Exception {
                        return new Tuple2<String, Integer>(word, 1);
                    }
                    
                }
        );
        
        JavaPairRDD<String, Integer> wordCounts = pairs.reduceByKey(
                new Function2<Integer, Integer, Integer>() {
                    
                    private static final long serialVersionUID = 1L;

                    @Override
                    public Integer call(Integer v1, Integer v2) throws Exception {
                        return v1 + v2;
                    }
                }
        );
        
        wordCounts.foreach(new VoidFunction<Tuple2<String,Integer>>() {
            
            private static final long serialVersionUID = 1L;

            @Override
            public void call(Tuple2<String, Integer> wordCount) throws Exception {
                System.out.println(wordCount._1 + " appeared " + wordCount._2 + " times.");
            }
        });
        
        sc.close();
    }
}

单词统计结果(部分)，运行成功。

HMJSL00008939 appeared 9 times.
05525262010402 appeared 23 times.
HMJSL00004251 appeared 21 times.
HMJSL00010319 appeared 8 times.
2010.96 appeared 2 times.
HMJSL00005220 appeared 18 times.
04324304410102 appeared 46 times.
YA214315200101 appeared 39 times.
SSSL00000720 appeared 16 times.
HMJSL00011009 appeared 11 times.
RMSL00014263 appeared 5 times.
SSSL00001199 appeared 17 times.
1648.8 appeared 1 times.
GHSL00001512 appeared 2 times.
HMJSL00013060 appeared 3 times.
ZYSL00000917 appeared 25 times.
HMJSL00008603 appeared 26 times.

总结

搭建的过程中，可能会遇到很多其他问题。网上都有成熟经验。

路漫漫其修远兮 吾将上下而求索

©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。

推荐阅读更多精彩内容

  • Spring Cloud
    Spring Cloud为开发人员提供了快速构建分布式系统中一些常见模式的工具（例如配置管理，服务发现，断路器，智...
    卡卡罗2017阅读 135,067评论 19赞 139
  • Spring boot参考指南
    Spring Boot 参考指南 介绍 转载自:https://www.gitbook.com/book/qbgb...
    毛宇鹏阅读 46,985评论 6赞 342
  • 2018-05-15
    【亿邦日报&运动日】 5月15号 星期二 夏天来临了，一群小仙女穿上自己漂亮的仙女裙，但却有一群人因自己的身材而烦...
    亿邦成长阅读 315评论 0赞 0
  • 《幸福的方法》心得笔记--30304--大禹
    《幸福的方法》心得笔记 易效能@时间管理天使28班 3.0教练团 云南--大禹 哈弗心理学博士泰勒・本-沙哈尔写的...
    大禹_QJ阅读 662评论 4赞 4
  • 2017.09.28悠悠上学记
    今天的大户外从大理大学出发一路上山，开启了孩子们的探索之旅，一路上孩子们发现了很多新奇的植物和昆虫，各种蘑菇，木耳...
    姚姚_c8e3阅读 166评论 0赞 0