程序博客网 > 淘宝的主营类目怎么改

spark streaming的NetworkWordCount实例理解

来源：互联网 发布：淘宝的主营类目怎么改 编辑：程序博客网 时间：2024/05/20 17:39

NetworkWordCount代码

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/** Licensed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> the Apache Software Foundation (ASF) under one <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> more* contributor license agreements. See the NOTICE file distributed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span>* this work <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> additional information regarding copyright ownership.* The ASF licenses this file <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> You under the Apache License, Version <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.0</span>* (the <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"License"</span>); you may not <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">use</span> this file except <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> compliance <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span>* the License. You may obtain a copy <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> the License at** http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//www.apache.org/licenses/LICENSE-2.0</span>** Unless required by applicable law <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> agreed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> writing, software* distributed under the License is distributed on an <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AS IS"</span> BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> implied.* See the License <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> the specific language governing permissions <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span>* limitations under the License.*/package org.apache.spark.examples.streamingimport org.apache.spark.SparkConfimport org.apache.spark.streaming.{Seconds, StreamingContext}import org.apache.spark.storage.StorageLevel/*** Counts words <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> UTF8 encoded, '\n' delimited text received from the network every second.** Usage: NetworkWordCount <hostname> <port>* <hostname> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <port> describe the TCP server that Spark Streaming would connect <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> receive data.** To run this on your local machine, you need <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> first run a Netcat server* `$ nc -lk <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>`* <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span> run the example* `$ bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>`*/object NetworkWordCount {    def main(args: Array[String]) {        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.length < <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>) {        System.err.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Usage: NetworkWordCount <hostname> <port>"</span>)        System.exit(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)    }    StreamingExamples.setStreamingLogLevels()    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Create the context with a 1 second batch size</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 创建SparkConf实例</span>    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"NetworkWordCount"</span>)    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">///创建Spark Streaming Context，每隔1秒钟处理一批数据，那么这一秒收集的数据存放在哪，如何将收集的数据推送出去？是生产者主动推出去还是消费者每隔1秒钟来拉取一次数据</span>    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ssc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(sparkConf, Seconds(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Create a socket stream on target ip:port and count the</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// words in input stream of \n delimited text (eg. generated by 'nc')</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Note that no duplication in storage level only for running locally.</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Replication necessary in distributed scenario for fault tolerance.</span>    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> lines = ssc.socketTextStream(args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>).toInt, StorageLevel.MEMORY_AND_DISK_SER)    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//flatMap是把将每一行使用空格做分解，那么words对应的数据结构是怎么样的？</span>        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">///words是个集合，每个集合元素依然是个集合，这个集合存放单词</span>        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> words = lines.flatMap(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" "</span>))    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> wordCounts = words.map(x => (x, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)).reduceByKey(_ + _)    wordCounts.print()    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//启动计算作业</span>    ssc.start()    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//等待结束，什么时候结束作业，即触发什么条件会让作业执行结束</span>    ssc.awaitTermination()   }}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li></ul>

运行NetworkWordCount

  • 启动NetCat命令

    <code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[hadoop<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@hadoop</span> ~]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>nc -lk <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span><span class="hljs-constant" style="box-sizing: border-box;">This</span> is a book </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

    启动后，可以在后面输入文本，比如This is a book

  • 在另外一个终端，Spark提交NetworkWordCount任务

    <code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./bin/run-example streaming<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.NetworkWordCount</span> localhost <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

启动后，发现每隔一秒钟，打印一行日志，例如

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953009000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953010000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953011000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">12</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953012000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953013000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953014000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953015000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953016000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953017000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">18</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953018000</span> ms   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"> ///Added jobs是个什么概念？不就是一个Job吗</span>    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">19</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953019000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953020000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953021000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953022000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953023000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">24</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953024000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953025000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">26</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953026000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">27</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953027000</span> ms    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">28</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953028000</span> ms</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>

  • 在nc -lk运行的终端，输入文本，发现Spark作业没有将文本输出 

  • 不过通过nc -lk输入一行文本后，控制台会显示如下信息

    <code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.MemoryStore: ensureFreeSpace(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>) called <span class="hljs-operator" style="box-sizing: border-box;">with</span> curMem=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">91198</span>, maxMem=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">280248975</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.MemoryStore: Block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span> stored <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">bytes</span> <span class="hljs-operator" style="box-sizing: border-box;">in</span> memory (estimated size <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11.0</span> B, free <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">267.2</span> MB)<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954148000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.BlockManagerInfo: Added input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span> <span class="hljs-operator" style="box-sizing: border-box;">in</span> memory <span class="hljs-command" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">on</span> <span class="hljs-title" style="box-sizing: border-box;">localhost</span>:<span class="hljs-title" style="box-sizing: border-box;">57786</span> (<span class="hljs-title" style="box-sizing: border-box;">size</span>: <span class="hljs-title" style="box-sizing: border-box;">11</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-title" style="box-sizing: border-box;">B</span>, <span class="hljs-title" style="box-sizing: border-box;">free</span>: <span class="hljs-title" style="box-sizing: border-box;">267</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span> <span class="hljs-title" style="box-sizing: border-box;">MB</span>)</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.BlockManagerMaster: Updated info <span class="hljs-operator" style="box-sizing: border-box;">of</span> block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO receiver.BlockGenerator: Pushed block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

  • 关闭nc命令的执行，此时，Spark报错，提示9999端口连接不上

    <code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">///相对于数据源而言，Spark是Receiver，所以Spark Streaming有Receiver这个模块<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Stopped receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">37</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.JobScheduler</span>: Added jobs for time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420952977000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.JobScheduler</span>: Added jobs for time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420952978000</span> ms///重提开始Receiver<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Starting receiver again<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Starting receiver<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Called receiver onStart<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverTracker</span>: Registered receiver for stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> from akka://sparkDriver<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Receiver started again<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO dstream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span>: Connecting to localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> WARN receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Restarting receiver with delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span> ms: Error connecting to localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span><span class="hljs-label" style="box-sizing: border-box;">java.net.ConnectException:</span> Connection refused    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.PlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.socketConnect</span>(Native Method)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.doConnect</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">339</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connectToAddress</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">182</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocksSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(SocksSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">392</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">579</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">528</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span>.<init>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">425</span>)    at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span>.<init>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">208</span>)    at org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.streaming</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.dstream</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.receive</span>(SocketInputDStream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.scala</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">71</span>)    at org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.streaming</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.dstream</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span>$$anon$2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.run</span>(SocketInputDStream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.scala</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">57</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

关于接不到消息

下面的代码收不到消息

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkStreamingExample"</span>).setMaster(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

而下面的代码则能收到消息

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkStreamingExample"</span>).setMaster(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local[2]"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

原因来自于http://spark.apache.org/docs/latest/streaming-programming-guide.html： 
When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using a input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data. Hence, when running locally, always use “local[n]” as the master URL where n > number of receivers to run (see Spark Properties for information on how to set the master). 
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise the system will receive data, but not be able to process them.

关于Receiver的线程数

上面提到，有一个线程用于运行Receiver,实际中，可能需要多个线程运行Receiver： 
Receiving data over the network (like Kafka, Flume, socket, etc.) requires the data to deserialized and stored in Spark. If the data receiving becomes a bottleneck in the system, then consider parallelizing the data receiving. Note that each input DStream creates a single receiver (running on a worker machine) that receives a single stream of data. Receiving multiple data streams can therefore be achieved by creating multiple input DStreams and configuring them to receive different partitions of the data stream from the source(s). For example, a single Kafka input DStream receiving two topics of data can be split into two Kafka input streams, each receiving only one topic. This would run two receivers on two workers, thus allowing data to be received in parallel, and increasing overall throughput. These multiple DStream can be unioned together to create a single DStream. Then the transformations that was being applied on the single input DStream can applied on the unified stream. This is done as follows

<code class="hljs r has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">val numStreams = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>val kafkaStreams = (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> to numStreams).map { i => KafkaUtils.createStream(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">...</span>) }val unifiedStream = streamingContext.union(kafkaStreams)unifiedStream.print()</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

关于Spark内部处理数据的并行数

Cluster resources can be under-utilized if the number of parallel tasks used in any stage of the computation is not high enough. For example, for distributed reduce operations like reduceByKey and reduceByKeyAndWindow, the default number of parallel tasks is controlled by thespark.default.parallelism configuration property. You can pass the level of parallelism as an argument (see PairDStreamFunctions documentation), or set the spark.default.parallelism configuration property to change the default.

关于谁先启动：

如果SparkStreaming先启动而9999端口后启动，会发生什么样的情况？ 
首先，SparkStreaming每次到了时间点都会提示连接失败，也就是说，每次操作都是尝试新建连接。 
1. 实际上，当SparkStreaming每次时间间隔到了进行提交作业时，都会首先start Receiver，本次Job运行完成时再stop Receiver，如下是一个时间点在没有开启9999的一个情况

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[hadoop<span class="hljs-decorator" style="color: rgb(0, 102, 102); box-sizing: border-box;">@hadoop ~]$ cat spark.log | grep "2015-09-20 00:50:04"</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Slicing <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411402000</span> ms to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms (aligned to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411402000</span> ms <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms)[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting job: foreachRDD at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">44</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">238</span> finished: foreachRDD at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">44</span>, took <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.000036</span> s[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting job streaming job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> job set of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Finished job streaming job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> job set of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Total delay: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.012</span> s <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms (execution: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.000</span> s)[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">711</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">711</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">704</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">704</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing blocks of RDD BlockRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span>] at socketTextStream at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deleting batches ArrayBuffer()[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deleting batches ArrayBuffer(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411400000</span> ms)[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting receiver again[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting receiver[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Called receiver onStart[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Registered receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> akka://sparkDriver[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Receiver started again[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>[WARN ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logWarning:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">92</span>] Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span> ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Stopping receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> message: Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span>ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>: java.net.ConnectException: Connection refused[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Called receiver onStop[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deregistering receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>[ERROR] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logError:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">75</span>] Deregistered receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>: Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span>ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span> - java.net.ConnectException: Connection refused[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Stopped receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li></ul>

上面的日志有Starting receiver和Stopped Receiver的输出，同时，也有java.net.Connection refused的异常

  1. 虽然9999端口没有监听上，但是此时由于提交了spark streaming application，那么spark streaming application已经开始正常运行了，即上面的wordcount例子，words这个RDD为空，下面的代码更能说明问题：
<code class="hljs scala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> spark.examples.databricks.reference.apps.loganalysis<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.sql.SQLContext<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.{SparkContext, SparkConf}<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.streaming.{StreamingContext, Duration}<span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** * The LogAnalyzerStreamingSQL is similar to LogAnalyzerStreaming, except * it computes stats using Spark SQL. * * To feed the new lines of some logfile into a socket for streaming, * Run this command: *   % tail -f [YOUR_LOG_FILE] | nc -lk 9999 * * If you don't have a live log file that is being written to, * you can add test lines using this command: *   % cat ../../data/apache.access.log >> [YOUR_LOG_FILE] * * Example command to run: * % spark-submit *   --class "com.databricks.apps.logs.chapter1.LogAnalyzerStreaming" *   --master local[4] *   target/scala-2.10/spark-logs-analyzer_2.10-1.0.jar */</span><span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">object</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LogAnalyzerStreamingSQL</span> {</span>  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> WINDOW_LENGTH = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Duration(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> SLIDE_INTERVAL = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Duration(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> main(args: Array[String]) {    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Log Analyzer Streaming SQL in Scala"</span>)    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkContext(sparkConf)    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sqlContext = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SQLContext(sc)    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> sqlContext.createSchemaRDD    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> streamingContext = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(sc, SLIDE_INTERVAL)    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//首先应该监听上localhost:9999端口，如果不监听会发生什么情况？</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//下面的逻辑  println("No access com.databricks.app.logs received in this time interval")已经执行了</span>    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//同时日志中报出Connection Refused错误</span>    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> logLinesDStream = streamingContext.socketTextStream(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>)    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//转换成DStream[ApacheAccessLog]</span>    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> accessLogsDStream = logLinesDStream.map(ApacheAccessLog.parseLogLine).cache()    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> windowDStream = accessLogsDStream.window(WINDOW_LENGTH, SLIDE_INTERVAL)    windowDStream.foreachRDD(accessLogs => {      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (accessLogs.count() == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {        println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"No access com.databricks.app.logs received in this time interval"</span>)      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {        accessLogs.registerTempTable(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"TBL_ACCESS_LOG"</span>)        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Calculate statistics based on the content size.</span>        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> contentSizeStats = sqlContext          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT SUM(contentSize), COUNT(*), MIN(contentSize), MAX(contentSize) FROM TBL_ACCESS_LOG"</span>)          .first()        println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Content Size Avg: %s, Min: %s, Max: %s"</span>.format(          contentSizeStats.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) / contentSizeStats.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>),          contentSizeStats(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>),          contentSizeStats(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)))        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Compute Response Code to Count.</span>        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> responseCodeToCount = sqlContext          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT responseCode, COUNT(*) FROM TBL_ACCESS_LOG GROUP BY responseCode"</span>)          .map(row => (row.getInt(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), row.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))          .take(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""Response code counts: ${responseCodeToCount.mkString("[", ",", "]")}"""</span>)        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Any IPAddress that has accessed the server more than 10 times.</span>        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ipAddresses =sqlContext          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT ipAddress, COUNT(*) AS total FROM TBL_ACCESS_LOG GROUP BY ipAddress HAVING total > 10"</span>)          .map(row => row.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))          .take(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>)        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""IPAddresses > 10 times: ${ipAddresses.mkString("[", ",", "]")}"""</span>)        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> topEndpoints = sqlContext          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT endpoint, COUNT(*) AS total FROM TBL_ACCESS_LOG GROUP BY endpoint ORDER BY total DESC LIMIT 10"</span>)          .map(row => (row.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), row.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))          .collect()        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""Top Endpoints: ${topEndpoints.mkString("[", ",", "]")}"""</span>)      }    })    streamingContext.start()    streamingContext.awaitTermination()  }}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li></ul>

虽然9999端口没有开启监听，但是spark streaming application的console，定时的输出No access com.databricks.app.logs received in this time interval


0 0
淘宝的主营类目怎么改 淘宝的主营类目怎么改
原创粉丝点击
热门IT博客
mac如何下载qq游戏mac如何下载qq游戏
java 隐式参数java 隐式参数
日本蜡烛图技术知乎
民乐队服装大全淘宝
2009流行网络歌曲500首
淘宝网卖家客服
linux mplayer 分辨率
ios上python编辑器推荐
js 非严格模式
matlab分水岭算法
测颜色的软件
金山画王软件下载
手机淘宝卡券包怎么用
mysql数据库汉化版
微信淘宝客怎么做
淘宝上可以卖弹弓
淘宝口令是什么
暴力美学知乎
室内设计网络培训
美国新纳粹知乎
热门问题 老师的惩罚 人脸识别 我在镇武司摸鱼那些年 重生之率土为王 我在大康的咸鱼生活 盘龙之生命进化 天生仙种 凡人之先天五行 春回大明朝 姑娘不必设防，我是瞎子 dpf灯亮了怎么办 60岁社保没交满怎么办 老公素质太差怎么办 耿彦波退休了太原怎么办 宫腔镜三天后同房了怎么办 开关失灵关不了怎么办 灯的开关关不掉怎么办? 身上毛孔粗大怎么办呀 脸被牙膏灼伤怎么办 wifi要登录认证怎么办 本科没有选导师怎么办 预授权撤销了怎么办 软件连不上网怎么办 手机屏幕出现有道词典怎么办 孩子总觉得没错怎么办 孩子动手打父母怎么办 初中零基础高中怎么办 弟媳妇爱上我怎么办 孩子考不好家长怎么办 小高考不过关怎么办 小高考不过考生怎么办 江苏不过怎么办小高考 大小脸拍婚纱照怎么办 被大学开除学籍怎么办 大专不给你退学怎么办 大专退学后档案怎么办 多单位同时录取怎么办 毕业学校改名了怎么办 改名后所有档案怎么办 大学学校改名毕业证怎么办 洗碗下水道堵了怎么办 美国打朝鲜丹东怎么办 欧洲通票行程不够怎么办 飞机安检被拦下东西怎么办 火车站丢了东西怎么办 高铁乘务近视怎么办 高铁乘务员怀孕怎么办 小超市生意不好怎么办 企业数字证书过期了怎么办 盆栽植物烂根怎么办 盆栽果树烂根怎么办

程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里