连接笔记

来源：互联网发布：记录商品价格的软件编辑：程序博客网时间：2024/06/15 19:44

1:jdk,scala.msi
2:ssh,putty
3:IDEA

4:FILE-SETTING-PLUGINS-SCALA
5:FILE-PROJECT STRUCTURE-LIBRARIES-“+”-JAVA——-(SPARK-ASSEMBLY-HADOOP.JAR)

6:NEW-ADD jdk-ADD SCALA
7:BUILD JAR ->OUT(“D:\SPARK DISTRIBUTE\1j\out\artifacts\1j_jar”)

7:PUTTY(LINUX FILE ->HDFS FILE )-SSH(UPLOAD FILE)-“nbu@hadoop201:~/spark-1.2.0$ bin/spark-submit –master yarn-cluster://10.22.66.201:7077 –class scj /home/nbu/jhq/1j.jar” ——(class + “object NAME”)

WATCH:
1:HOST IP-NAME
2:10.22.66.201 nbu 123456

8.su -

9.流数据hdfs处理：数据必须是从hdfs文件到某个input文件夹

object CountTriangle {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName(“CountTriangle”)
val sc = new SparkContext(sparkConf)
val window = sc.textFile(“/jhq/”)
// var x
val edgevv=window.flatMap(_.split(” “)).map(x=>(x.split(“;”)(1),x.split(“;”)(0))).reduceByKey((x,y)=>x+” “+y).collect()
if(edgevv(0)(0)==”0”)
//val edgevvadd=edgeadd.flatMap(x => List(x.split(“,”)(0)+”,”+ x.split(“,”)(1) , x.split(“,”)(1)+”,”+ x.split(“,”)(0))).map(x=>(x.split(“,”)(0),x.split(“,”)(1)))
// edgevv.reduceByKey((x,y)=>x+”,”+y)
/*def cnstrctNeighbor(x:String,y:String):String={

  return x}*/

// window.flatMap(.split(” “)).flatMap(x => List(x.split(“,”)(0)+”,”+x.split(“,”)(1),.split(“,”)(1)+”,”+_.split(“,”)(0)))
sc.stop()
}
}

0 0