Spark2.0: Structured Streaming

来源：互联网发布：易语言安装包源码编辑：程序博客网时间：2024/05/21 20:24

启动nc：

nc -lk 9999

启动spark-shell：

val lines = spark.readStream.     | format("socket").     | option("host","localhost").     | option("port",9999).     | load();lines: org.apache.spark.sql.DataFrame = [value: string]val words = lines.as[String].flatMap(_.split(" "))val wordCounts = words.groupBy("value").count()val query = wordCounts.writeStream.     | outputMode("complete").     | format("console").     | start();query: org.apache.spark.sql.streaming.StreamingQuery = org.apache.spark.sql.execution.streaming.Streamin          gQueryWrapper@22cf6c9cquery.awaitTermination()-------------------------------------------Batch: 0-------------------------------------------+-----+-----+|value|count|+-----+-----+|hello|    1|| word|    1|+-----+-----+-------------------------------------------Batch: 1-------------------------------------------+-----+-----+|value|count|+-----+-----+|hello|    2|| word|    1||world|    1|+-----+-----+-------------------------------------------Batch: 2-------------------------------------------+-----+-----+|value|count|+-----+-----+|hello|    2||  bye|    1|| word|    1||world|    2||   ny|    1|+-----+-----+-------------------------------------------Batch: 3-------------------------------------------+------+-----+| value|count|+------+-----+| hello|    2||   bye|    1||  word|    1||sounds|    1|| world|    2||  good|    1||    ny|    1|+------+-----+-------------------------------------------Batch: 4-------------------------------------------+------+-----+| value|count|+------+-----+| hello|    2||   bye|    3||  word|    1||sounds|    1|| world|    2||  good|    1||    ny|    1|+------+-----+

阅读全文

0 0