使用的ide是idea
packagecom.fsl
importorg.apache.spark.{SparkConf,SparkContext}
/**
* Created by Juvenile on 2017/2/24.
* 统计字符出现次数
*/
objectWordCountScala {
defmain(args: Array[String]) {
if(args.length <1) {
System.err.println("Usage: ")
System.exit(1)
}
valconf =newSparkConf()
valsc =newSparkContext(conf)
valline = sc.textFile(args(0))
line.flatMap(_.split("\t")).map((_,1)).reduceByKey(_ + _).collect.foreach(println)
sc.stop
}
}
导出成jar包。
上次到集群服务器: mySparkTest-1.0.jar
进入spark文件夹下执行 spark-submit --name WordCountScala --class com.fsl.WordCountScala --master spark://master:7077 --executor-memory 1G --total-executor-cores 1 /root/mySparkTest-1.0.jar hdfs://master:9000/user/data.txt 即可
