Spark入门（二）：打包程序、提交任务到集群

1.环境工具

1.1环境

系统 centos jdk 1.8.0_144 scala 2.11.8 hadoop 2.7.3 spark 2.1.0

1.2打包工具

IDEA + sbt

2.打包

2.1安装插件

需要预先安装scala插件，点击File ->Setting ->Plugins ->输入框输入scala->install
安装完成需要重启IDE

2.2创建项目

File -> New Project ->Scala -> SBT 选择相应版本 ->finish

2.3编写代码

build.sbt 添加spark相关依赖

name := "scalaWorkspace"version := "1.0"scalaVersion := "2.11.11"libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.1.0"

创建WordCount.scala，编写如下代码

import org.apache.spark.{SparkConf, SparkContext}object WordCount {  def main(args: Array[String]): Unit = {    val conf  = new SparkConf().setAppName("hdfs:master:9000/test/test.txt")    val sc = new SparkContext(conf)    val input = sc.textFile("derby.log")    val words = input.flatMap(line =>line.split(" "))    val count  = words.map(word => (word,1)).reduceByKey{ case (x,y) => x+y }    val out = count.saveAsTextFile("hdfs:master:9000/test/result")  }}

2.4打包

File -> Project Structure -> Aritifacts -> 点击+号 ->jar -> 第二个 -> 指定Module和 MainClass -> JAR files from libraries 选择第二个 ->点击ok

主题栏点击Build -> Build Aritifacts - Build

在工程目下out目录中生成相应jar包即打包成功

3.提交任务

3.1启动hadoop

#进入sbin目录cd $Hadoop_HOME/sbin #启动hadoop集群start-all.sh

3.2上传测试文件到hdfs

hadoop fs -put test.txt /test/test.txt

3.3上传程序jar包

是同filelize 或者sftp 或者 rz -y命令上传程序jar

3.4 提交任务

3.4.1启动Master

.$SPARK_HOME/sbin/start-master.sh 访问localhost:8080 获取spark://xxx:7077

3.4.2启动Worker

.$SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker spark://xxx:7077

3.4.3提交作业

.$SPARK_HOME/bin/spark-submit --master spark://xxx:7077 --class WordCount /xxx/wordCount.jar