2 大数据实战系列-spark shell wordcount

1 启动spark shell

cd /home/data/app/hadoop/spark-2.1.1-bin-hadoop2.7/bin./spark-shell --master spark://shulaibao2:7077 --executor-memory 512m --driver-memory 4540m

初始化sc->SparkContext   spark->SparkSession

2 创建hdfs数据源

• 2.1创建hdfs文件夹

Hadoop fs - mkdir -p /home/hadoop/upload/test

• 2.2 上传数据源到hdfs

Hadoop fs -put /home/data/app/hadoop/hadoop-2.8.0/etc/hadoop/core-site.xml /home/hadoop/upload/test

• 2.3 验证hdfs文件列表

Hadoop fs -ls /home/hadoop/upload/test

3 wordcount

Scala-> spark shell:

scala>val rdd=sc.textFile("hdfs://shulaibao2:9010/home/hadoop/upload/test/core-site.xml")scala>rdd.cache()scala>val wordcount=rdd.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_)scala>wordcount.take(10)scala>val wordsort=wordcount.map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1))scala>wordsort.take(10)

Python -> spark submit:

spark = SparkSession.builder.appName("WordCountAPP").getOrCreate()    lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])    print(lines.collect())    counts = lines.flatMap(lambda x: x.split(' ')).map(lambda x: (x, 1)).reduceByKey(add)    output = counts.collect()    for (word, count) in output:        print("%s: %i" % (word, count))    spark.stop()

java version:
备注：楼主也是java程序员，但使用sprak确实不适合使用java开发。
例如：

List<Tuple2<String, Integer>> output = counts.collect();    for (Tuple2<?,?> tuple : output) {      System.out.println(tuple._1() + ": " + tuple._2());}

• 元组、列表都是scala封装jar不容易抓到本质的数据结构
• Java做数据分析代码冗长_.split(” “)或者lambda函数，java需要实现FlatMapFunction接口