Kafka+Storm+HDFS整合

对数据进行离线和实时分析，离线分析可以很容易地借助于Hive来实现统计分析，但是对于实时的需求Hive就不合适了。实时应用场景可以使用Storm，它是一个实时处理系统，它为实时处理类应用提供了一个计算模型，可以很容易地进行编程处理。为了统一离线和实时计算，一般情况下，我们都希望将离线和实时计算的数据源的集合统一起来作为输入，然后将数据的流向分别经由实时系统和离线分析系统，分别进行分析处理，这时我们可以考虑将数据源（如使用Flume收集日志）直接连接一个消息中间件，如Kafka，可以整合Flume+Kafka，Flume作为消息的Producer，生产的消息数据（日志数据、业务请求数据等等）发布到Kafka中，然后通过订阅的方式，使用Storm的Topology作为消息的Consumer，在Storm集群中分别进行如下两个需求场景的处理：

直接使用Storm的Topology对数据进行实时分析处理 
整合Storm+HDFS，将消息处理后写入HDFS进行离线分析处理 
实时处理，只要开发满足业务需要的Topology即可，不做过多说明。这里，我们主要从安装配置Kafka、Storm，以及整合Kafka+Storm、整合Storm+HDFS、整合Kafka+Storm+HDFS这几点来配置实践，满足上面提出的一些需求。配置实践使用的软件包如下所示：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">zookeeper-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.5</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span>kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tgz</span>apache-storm-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span>-incubating<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span>hadoop-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

程序配置运行所基于的操作系统为CentOS 5.11。 
Kafka安装配置

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">192.168.4.142   h1192.168.4.143   h2192.168.4.144   h3</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

在安装Kafka集群之前，这里没有使用Kafka自带的Zookeeper，而是独立安装了一个Zookeeper集群，也是使用这3台机器，保证Zookeeper集群正常运行。 
首先，在h1上准备Kafka安装文件，执行如下命令：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>wget http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//mirror.bit.edu.cn/apache/kafka/0.8.1.1/kafka_2.9.2-0.8.1.1.tgz</span>tar xvzf kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tgzln <span class="hljs-attribute" style="box-sizing: border-box;">-s</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafkachown <span class="hljs-attribute" style="box-sizing: border-box;">-R</span> kafka:kafka /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li></ul>

修改配置文件/usr/local/kafka/config/server.properties，修改如下内容：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">broker<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.id</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>zookeeper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>=h1:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>,h2:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>,h3:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>/kafk</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

这里需要说明的是，默认Kafka会使用ZooKeeper默认的/路径，这样有关Kafka的ZooKeeper配置就会散落在根路径下面，如果你有其他的应用也在使用ZooKeeper集群，查看ZooKeeper中数据可能会不直观，所以强烈建议指定一个chroot路径，直接在zookeeper.connect配置项中指定

<code class="hljs fix has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-attribute" style="box-sizing: border-box;">zookeeper.connect</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">h1:2181,h2:2181,h3:2181/kafka</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

而且，需要手动在ZooKeeper中创建路径/kafka，使用如下命令连接到任意一台ZooKeeper服务器：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">cd</span> /usr/local/zookeeperbin/zkCli.sh</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

在ZooKeeper执行如下命令创建chroot路径：

<code class="hljs sql has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">create</span> /kafka <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

这样，每次连接Kafka集群的时候（使用–zookeeper选项），也必须使用带chroot路径的连接字符串，后面会看到。 
然后，将配置好的安装文件同步到其他的h2、h3节点上：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scp <span class="hljs-attribute" style="box-sizing: border-box;">-r</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span> h2:/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>scp <span class="hljs-attribute" style="box-sizing: border-box;">-r</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span> h3:/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

最后，在h2、h3节点上配置，执行如下命令：’

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>ln <span class="hljs-attribute" style="box-sizing: border-box;">-s</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafkachown <span class="hljs-attribute" style="box-sizing: border-box;">-R</span> kafka:kafka /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

并修改配置文件/usr/local/kafka/config/server.properties内容如下所示：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">broker<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.id</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;"># 在h1修改</span>broker<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.id</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;"># 在h2修改</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

因为Kafka集群需要保证各个Broker的id在整个集群中必须唯一，需要调整这个配置项的值（如果在单机上，可以通过建立多个Broker进程来模拟分布式的Kafka集群，也需要Broker的id唯一，还需要修改一些配置目录的信息）。 
在集群中的h1、h2、h3这三个节点上分别启动Kafka，分别执行如下命令：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/kafka<span class="hljs-attribute" style="box-sizing: border-box;">-server</span><span class="hljs-attribute" style="box-sizing: border-box;">-start</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>sh /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/config/server<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>properties <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">&</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

可以通过查看日志，或者检查进程状态，保证Kafka集群启动成功。 
我们创建一个名称为my-replicated-topic5的Topic，5个分区，并且复制因子为3，执行如下命令：

<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">bin/kafka</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topics</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">sh</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">create</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">zookeeper</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h1:2181</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">,</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h2:2181</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">,</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h3:2181/kafka</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">replication</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">factor</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">3</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">partitions</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">5</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topic</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">my</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">replicated</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topic5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

查看创建的Topic，执行如下命令：

<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">bin/kafka</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topics</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">sh</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">describe</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">zookeeper</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h1:2181</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">,</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h2:2181</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">,</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">h3:2181/kafka</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topic</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">my</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">replicated</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topic5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Topic:my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     PartitionCount:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>     ReplicationFactor:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>     Configs:     Topic: my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     Partition: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Leader: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Replicas: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Isr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Topic: my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     Partition: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Leader: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Replicas: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>     Isr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Topic: my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     Partition: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>     Leader: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>     Replicas: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Isr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Topic: my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     Partition: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>     Leader: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Replicas: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>     Isr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>     Topic: my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span>     Partition: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>     Leader: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>     Replicas: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>     Isr: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li></ul>

上面Leader、Replicas、Isr的含义如下： 
Partition： 分区 
Leader ： 负责读写指定分区的节点 
Replicas ： 复制该分区log的节点列表 
Isr ： “in-sync” replicas，当前活跃的副本列表（是一个子集），并且可能成为Leader

我们可以通过Kafka自带的bin/kafka-console-producer.sh和bin/kafka-console-consumer.sh脚本，来验证演示如果发布消息、消费消息。 
在一个终端，启动Producer，并向我们上面创建的名称为my-replicated-topic5的Topic中生产消息，执行如下脚本：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/kafka<span class="hljs-attribute" style="box-sizing: border-box;">-console</span><span class="hljs-attribute" style="box-sizing: border-box;">-producer</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>sh <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>broker<span class="hljs-attribute" style="box-sizing: border-box;">-list</span> h1:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9092</span>,h2:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9092</span>,h3:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9092</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>topic my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

在另一个终端，启动Consumer，并订阅我们上面创建的名称为my-replicated-topic5的Topic中生产的消息，执行如下脚本：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/kafka<span class="hljs-attribute" style="box-sizing: border-box;">-console</span><span class="hljs-attribute" style="box-sizing: border-box;">-consumer</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>sh <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>zookeeper h1:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>,h2:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>,h3:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>/kafka <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>from<span class="hljs-attribute" style="box-sizing: border-box;">-beginning</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span>topic my<span class="hljs-attribute" style="box-sizing: border-box;">-replicated</span><span class="hljs-attribute" style="box-sizing: border-box;">-topic5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

可以在Producer终端上输入字符串消息行，然后回车，就可以在Consumer终端上看到消费者消费的消息内容。 
也可以参考Kafka的Producer和Consumer的Java API，通过API编码的方式来实现消息生产和消费的处理逻辑。

Storm安装配置

Storm集群也依赖Zookeeper集群，要保证Zookeeper集群正常运行。Storm的安装配置比较简单，我们仍然使用下面3台机器搭建：

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">192.168.4.142   h1192.168.4.143   h2192.168.4.144   h3</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>wget http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//mirror.bit.edu.cn/apache/incubator/storm/apache-storm-0.9.2-incubating/apache-storm-0.9.2-incubating.tar.gz</span>tar xvzf apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gzln <span class="hljs-attribute" style="box-sizing: border-box;">-s</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/stormchown <span class="hljs-attribute" style="box-sizing: border-box;">-R</span> storm:storm /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

然后，修改配置文件conf/storm.yaml，内容如下所示： 
storm.zookeeper.servers: 
- “h1” 
- “h2” 
- “h3” 
storm.zookeeper.port: 2181 
# 
nimbus.host: “h1”

supervisor.slots.ports: 
- 6700 
- 6701 
- 6702 
- 6703

storm.local.dir: “/tmp/storm”

将配置好的安装文件，分发到其他节点上：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scp <span class="hljs-attribute" style="box-sizing: border-box;">-r</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span> h2:/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>scp <span class="hljs-attribute" style="box-sizing: border-box;">-r</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span> h3:/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

最后，在h2、h3节点上配置，执行如下命令：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>ln <span class="hljs-attribute" style="box-sizing: border-box;">-s</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/stormchown <span class="hljs-attribute" style="box-sizing: border-box;">-R</span> storm:storm /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/apache<span class="hljs-attribute" style="box-sizing: border-box;">-storm</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-attribute" style="box-sizing: border-box;">-incubating</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

Storm集群的主节点为Nimbus，从节点为Supervisor，我们需要在h1上启动Nimbus服务，在从节点h2、h3上启动Supervisor服务：

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/storm nimbus &bin/storm supervisor &</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

为了方便监控，可以启动Storm UI，可以从Web页面上监控Storm Topology的运行状态，例如在h2上启动：

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/storm ui &</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

整合Kafka+Storm

消息通过各种方式进入到Kafka消息中间件，比如可以通过使用Flume来收集日志数据，然后在Kafka中路由暂存，然后再由实时计算程序Storm做实时分析，这时我们就需要将在Storm的Spout中读取Kafka中的消息，然后交由具体的Spot组件去分析处理。实际上，apache-storm-0.9.2-incubating这个版本的Storm已经自带了一个集成Kafka的外部插件程序storm-kafka，可以直接使用，例如我使用的Maven依赖配置，如下所示：

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.storm<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>storm-core<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>0.9.2-incubating<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">scope</span>></span>provided<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">scope</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.storm<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>storm-kafka<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>0.9.2-incubating<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.kafka<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>kafka_2.9.2<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>0.8.1.1<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusions</span>></span>          <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusion</span>></span>               <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.zookeeper<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>               <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>zookeeper<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>          <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusion</span>></span>          <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusion</span>></span>               <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>log4j<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>               <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>log4j<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>          <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusion</span>></span>     <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">exclusions</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li></ul>

代码

<code class="hljs java has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> org.shirdrn.storm.examples;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Arrays;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.HashMap;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Iterator;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Map;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Map.Entry;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.concurrent.atomic.AtomicInteger;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.commons.logging.Log;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.commons.logging.LogFactory;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> storm.kafka.BrokerHosts;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> storm.kafka.KafkaSpout;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> storm.kafka.SpoutConfig;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> storm.kafka.StringScheme;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> storm.kafka.ZkHosts;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.Config;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.LocalCluster;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.StormSubmitter;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.generated.AlreadyAliveException;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.generated.InvalidTopologyException;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.spout.SchemeAsMultiScheme;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.task.OutputCollector;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.task.TopologyContext;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.topology.OutputFieldsDeclarer;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.topology.TopologyBuilder;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.topology.base.BaseRichBolt;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.tuple.Fields;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.tuple.Tuple;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> backtype.storm.tuple.Values;<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">MyKafkaTopology</span> {</span>     <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">KafkaWordSplitter</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">BaseRichBolt</span> {</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> Log LOG = LogFactory.getLog(KafkaWordSplitter.class);          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> serialVersionUID = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">886149197481637894</span>L;          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> OutputCollector collector;          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">prepare</span>(Map stormConf, TopologyContext context,                    OutputCollector collector) {               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.collector = collector;                        }          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">execute</span>(Tuple input) {               String line = input.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>);               LOG.info(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"RECV[kafka -> splitter] "</span> + line);               String[] words = line.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\\s+"</span>);               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(String word : words) {                    LOG.info(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"EMIT[splitter -> counter] "</span> + word);                    collector.emit(input, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Values(word, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>));               }               collector.ack(input);          }          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">declareOutputFields</span>(OutputFieldsDeclarer declarer) {               declarer.declare(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"count"</span>));                   }     }     <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">WordCounter</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">extends</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">BaseRichBolt</span> {</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> Log LOG = LogFactory.getLog(WordCounter.class);          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> serialVersionUID = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">886149197481637894</span>L;          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> OutputCollector collector;          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> Map<String, AtomicInteger> counterMap;          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">prepare</span>(Map stormConf, TopologyContext context,                    OutputCollector collector) {               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.collector = collector;                   <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counterMap = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> HashMap<String, AtomicInteger>();          }          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">execute</span>(Tuple input) {               String word = input.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>);               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> count = input.getInteger(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>);               LOG.info(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"RECV[splitter -> counter] "</span> + word + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" : "</span> + count);               AtomicInteger ai = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counterMap.get(word);               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span>(ai == <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) {                    ai = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> AtomicInteger();                    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counterMap.put(word, ai);               }               ai.addAndGet(count);               collector.ack(input);               LOG.info(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"CHECK statistics map: "</span> + <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counterMap);          }          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">cleanup</span>() {               LOG.info(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"The final result:"</span>);               Iterator<Entry<String, AtomicInteger>> iter = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">this</span>.counterMap.entrySet().iterator();               <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span>(iter.hasNext()) {                    Entry<String, AtomicInteger> entry = iter.next();                    LOG.info(entry.getKey() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t:\t"</span> + entry.getValue().get());               }          }          <span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@Override</span>          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">declareOutputFields</span>(OutputFieldsDeclarer declarer) {               declarer.declare(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"count"</span>));                   }     }     <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">throws</span> AlreadyAliveException, InvalidTopologyException, InterruptedException {          String zks = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h1:2181,h2:2181,h3:2181"</span>;          String topic = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-replicated-topic5"</span>;          String zkRoot = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/storm"</span>; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// default zookeeper root configuration for storm</span>          String id = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>;          BrokerHosts brokerHosts = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> ZkHosts(zks);          SpoutConfig spoutConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SpoutConfig(brokerHosts, topic, zkRoot, id);          spoutConf.scheme = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SchemeAsMultiScheme(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StringScheme());          spoutConf.forceFromStart = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>;          spoutConf.zkServers = Arrays.asList(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> String[] {<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h1"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h2"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h3"</span>});          spoutConf.zkPort = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>;          TopologyBuilder builder = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> TopologyBuilder();          builder.setSpout(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka-reader"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> KafkaSpout(spoutConf), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Kafka我们创建了一个5分区的Topic，这里并行度设置为5</span>          builder.setBolt(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word-splitter"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> KafkaWordSplitter(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>).shuffleGrouping(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka-reader"</span>);          builder.setBolt(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word-counter"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> WordCounter()).fieldsGrouping(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word-splitter"</span>, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>));          Config conf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Config();          String name = MyKafkaTopology.class.getSimpleName();          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span> && args.length > <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {               <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Nimbus host name passed from command line</span>               conf.put(Config.NIMBUS_HOST, args[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]);               conf.setNumWorkers(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>);               StormSubmitter.submitTopologyWithProgressBar(name, conf, builder.createTopology());          } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {               conf.setMaxTaskParallelism(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>);               LocalCluster cluster = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> LocalCluster();               cluster.submitTopology(name, conf, builder.createTopology());               Thread.sleep(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">60000</span>);               cluster.shutdown();          }     }}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li></ul>

上面程序，在本地调试（使用LocalCluster）不需要输入任何参数，提交到实际集群中运行时，需要传递一个参数，该参数为Nimbus的主机名称。 
通过Maven构建，生成一个包含依赖的single jar文件（不要把Storm的依赖包添加进去），例如storm-examples-0.0.1-SNAPSHOT.jar，在提交Topology程序到Storm集群之前，因为用到了Kafka，需要拷贝一下依赖jar文件到Storm集群中的lib目录下面：

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/scala<span class="hljs-attribute" style="box-sizing: border-box;">-library</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.9</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/metrics<span class="hljs-attribute" style="box-sizing: border-box;">-core</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/snappy<span class="hljs-attribute" style="box-sizing: border-box;">-java</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.5</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/zkclient<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.3</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/log4j<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.15</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/slf4j<span class="hljs-attribute" style="box-sizing: border-box;">-api</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span>cp /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/kafka/libs/jopt<span class="hljs-attribute" style="box-sizing: border-box;">-simple</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.2</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/storm/lib<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">/</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

然后，就可以提交我们开发的Topology程序了：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/storm jar /home/storm/storm-examples-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span>-SNAPSHOT<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.jar</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shirdrn</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.examples</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.MyKafkaTopology</span> h1</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

可以通过查看日志文件（logs/目录下）或者Storm UI来监控Topology的运行状况。如果程序没有错误，可以使用前面我们使用的Kafka Producer来生成消息，就能看到我们开发的Storm Topology能够实时接收到并进行处理。 
上面Topology实现代码中，有一个很关键的配置对象SpoutConfig，配置属性如下所示：

<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">spoutConf.forceFromStart = <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">false</span>;</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

该配置是指，如果该Topology因故障停止处理，下次正常运行时是否从Spout对应数据源Kafka中的该订阅Topic的起始位置开始读取，如果forceFromStart=true，则之前处理过的Tuple还要重新处理一遍，否则会从上次处理的位置继续处理，保证Kafka中的Topic数据不被重复处理，是在数据源的位置进行状态记录。

整合Storm+HDFS

Storm实时计算集群从Kafka消息中间件中消费消息，有实时处理需求的可以走实时处理程序，还有需要进行离线分析的需求，如写入到HDFS进行分析。下面实现了一个Topology，代码如下所示：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shirdrn</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.examples</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.text</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DateFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.text</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SimpleDateFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Date</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Map</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Random</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.commons</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.logging</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Log</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.commons</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.logging</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.LogFactory</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.HdfsBolt</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DefaultFileNameFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DelimitedRecordFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileNameFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RecordFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileRotationPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimedRotationPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimedRotationPolicy</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimeUnit</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sync</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.CountSyncPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sync</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SyncPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Config</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.LocalCluster</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.StormSubmitter</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.generated</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AlreadyAliveException</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.generated</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.InvalidTopologyException</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spout</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SpoutOutputCollector</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.task</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TopologyContext</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.OutputFieldsDeclarer</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TopologyBuilder</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.base</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.BaseRichSpout</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tuple</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Fields</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tuple</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Values</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.utils</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Utils</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>public class StormToHDFSTopology {     public static class EventSpout extends BaseRichSpout {          private static final Log LOG = LogFactory<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getLog</span>(EventSpout<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private static final long serialVersionUID = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">886149197481637894</span>L<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private SpoutOutputCollector collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private Random rand<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private String[] records<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          @Override          public void open(Map conf, TopologyContext context,                    SpoutOutputCollector collector) {               this<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collector</span> = collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;    </span>               rand = new Random()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               records = new String[] {                         <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"10001     ef2da82d4c8b49c44199655dc14f39f6     4.2.1     HUAWEI G610-U00     HUAWEI     2     70:72:3c:73:8b:22     2014-10-13 12:36:35"</span>,                         <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"10001     ffb52739a29348a67952e47c12da54ef     4.3     GT-I9300     samsung     2     50:CC:F8:E4:22:E2     2014-10-13 12:36:02"</span>,                         <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"10001     ef2da82d4c8b49c44199655dc14f39f6     4.2.1     HUAWEI G610-U00     HUAWEI     2     70:72:3c:73:8b:22     2014-10-13 12:36:35"</span>               }<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }          @Override          public void nextTuple() {               Utils<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sleep</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               DateFormat df = new SimpleDateFormat(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"yyyy-MM-dd_HH-mm-ss"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               Date d = new Date(System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.currentTimeMillis</span>())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               String minute = df<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span>(d)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               String record = records[rand<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.nextInt</span>(records<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span>)]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               LOG<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.info</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"EMIT[spout -> hdfs] "</span> + minute + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" : "</span> + record)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               collector<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.emit</span>(new Values(minute, record))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }          @Override          public void declareOutputFields(OutputFieldsDeclarer declarer) {               declarer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.declare</span>(new Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"minute"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"record"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;         </span>          }     }     public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException, InterruptedException {          // use <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"|"</span> instead of <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span> for field delimiter          RecordFormat format = new DelimitedRecordFormat()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFieldDelimiter</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" : "</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          // sync the filesystem after every <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>k tuples          SyncPolicy syncPolicy = new CountSyncPolicy(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          // rotate files           FileRotationPolicy rotationPolicy = new TimedRotationPolicy(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.0</span>f, TimeUnit<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.MINUTES</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          FileNameFormat fileNameFormat = new DefaultFileNameFormat()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withPath</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/storm/"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withPrefix</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"app_"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withExtension</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">".log"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          HdfsBolt hdfsBolt = new HdfsBolt()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFsUrl</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://h1:8020"</span>)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFileNameFormat</span>(fileNameFormat)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withRecordFormat</span>(format)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withRotationPolicy</span>(rotationPolicy)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withSyncPolicy</span>(syncPolicy)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          TopologyBuilder builder = new TopologyBuilder()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setSpout</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"event-spout"</span>, new EventSpout(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setBolt</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs-bolt"</span>, hdfsBolt, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fieldsGrouping</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"event-spout"</span>, new Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"minute"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          Config conf = new Config()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          String name = StormToHDFSTopology<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getSimpleName</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          if (args != null && args<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span> > <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.put</span>(Config<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.NIMBUS</span>_HOST, args[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>])<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setNumWorkers</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               StormSubmitter<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.submitTopologyWithProgressBar</span>(name, conf, builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.createTopology</span>())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          } else {               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMaxTaskParallelism</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               LocalCluster cluster = new LocalCluster()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               cluster<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.submitTopology</span>(name, conf, builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.createTopology</span>())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               Thread<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sleep</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">60000</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               cluster<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shutdown</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }     }}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li></ul>

上面的处理逻辑，可以对HdfsBolt进行更加详细的配置，如FileNameFormat、SyncPolicy、FileRotationPolicy（可以设置在满足什么条件下，切出一个新的日志，如可以指定多长时间切出一个新的日志文件，可以指定一个日志文件大小达到设置值后，再写一个新日志文件），更多设置可以参考storm-hdfs，。 
上面代码在打包的时候，需要注意，使用storm-starter自带的Maven打包配置，可能在将Topology部署运行的时候，会报错，可以使用maven-shade-plugin这个插件，如下配置所示：

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugin</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.maven.plugins<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>maven-shade-plugin<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>1.4<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">createDependencyReducedPom</span>></span>true<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">createDependencyReducedPom</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">executions</span>></span>        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">execution</span>></span>            <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">phase</span>></span>package<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">phase</span>></span>            <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goals</span>></span>                <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goal</span>></span>shade<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goal</span>></span>            <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goals</span>></span>            <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>                <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformers</span>></span>                    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer</span>                            <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">implementation</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"</span>/></span>                    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer</span>                            <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">implementation</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"</span>></span>                        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">mainClass</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">mainClass</span>></span>                    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer</span>></span>                <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformers</span>></span>            <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span>        <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">execution</span>></span>    <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">executions</span>></span><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugin</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li></ul>

整合Kafka+Storm+HDFS

上面分别对整合Kafka+Storm和Storm+HDFS做了实践，可以将后者的Spout改成前者的Spout，从Kafka中消费消息，在Storm中可以做简单处理，然后将数据写入HDFS，最后可以在Hadoop平台上对数据进行离线分析处理。下面，写了一个简单的例子，从Kafka消费消息，然后经由Storm处理，写入到HDFS存储，代码如下所示：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">package org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shirdrn</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.examples</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Arrays</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.util</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Map</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.commons</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.logging</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Log</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.commons</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.logging</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.LogFactory</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.HdfsBolt</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DefaultFileNameFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DelimitedRecordFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileNameFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RecordFormat</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.FileRotationPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimedRotationPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rotation</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimedRotationPolicy</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TimeUnit</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sync</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.CountSyncPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.bolt</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sync</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SyncPolicy</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import storm<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.BrokerHosts</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import storm<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.KafkaSpout</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import storm<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SpoutConfig</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import storm<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.StringScheme</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import storm<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ZkHosts</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Config</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.LocalCluster</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.StormSubmitter</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.generated</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AlreadyAliveException</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.generated</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.InvalidTopologyException</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spout</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SchemeAsMultiScheme</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.task</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.OutputCollector</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.task</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TopologyContext</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.OutputFieldsDeclarer</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.TopologyBuilder</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topology</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.base</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.BaseRichBolt</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tuple</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Fields</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tuple</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Tuple</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>import backtype<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tuple</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Values</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>public class DistributeWordTopology {     public static class KafkaWordToUpperCase extends BaseRichBolt {          private static final Log LOG = LogFactory<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getLog</span>(KafkaWordToUpperCase<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private static final long serialVersionUID = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5207232012035109026</span>L<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private OutputCollector collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          @Override          public void prepare(Map stormConf, TopologyContext context,                    OutputCollector collector) {               this<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collector</span> = collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;              </span>          }          @Override          public void execute(Tuple input) {               String line = input<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getString</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.trim</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               LOG<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.info</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"RECV[kafka -> splitter] "</span> + line)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               if(!line<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.isEmpty</span>()) {                    String upperLine = line<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.toUpperCase</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>                    LOG<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.info</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"EMIT[splitter -> counter] "</span> + upperLine)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>                    collector<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.emit</span>(input, new Values(upperLine, upperLine<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span>()))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               }               collector<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ack</span>(input)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }          @Override          public void declareOutputFields(OutputFieldsDeclarer declarer) {               declarer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.declare</span>(new Fields(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"line"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"len"</span>))<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;         </span>          }     }     public static class RealtimeBolt extends BaseRichBolt {          private static final Log LOG = LogFactory<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getLog</span>(KafkaWordToUpperCase<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private static final long serialVersionUID = -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4115132557403913367</span>L<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          private OutputCollector collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          @Override          public void prepare(Map stormConf, TopologyContext context,                    OutputCollector collector) {               this<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collector</span> = collector<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;              </span>          }          @Override          public void execute(Tuple input) {               String line = input<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getString</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.trim</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               LOG<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.info</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"REALTIME: "</span> + line)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               collector<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ack</span>(input)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }          @Override          public void declareOutputFields(OutputFieldsDeclarer declarer) {          }     }     public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException, InterruptedException {          // Configure Kafka          String zks = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h1:2181,h2:2181,h3:2181"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          String topic = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my-replicated-topic5"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          String zkRoot = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/storm"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; // default zookeeper root configuration for storm</span>          String id = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          BrokerHosts brokerHosts = new ZkHosts(zks)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          SpoutConfig spoutConf = new SpoutConfig(brokerHosts, topic, zkRoot, id)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          spoutConf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.scheme</span> = new SchemeAsMultiScheme(new StringScheme())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          spoutConf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.forceFromStart</span> = false<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          spoutConf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.zkServers</span> = Arrays<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.asList</span>(new String[] {<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h1"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h2"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"h3"</span>})<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          spoutConf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.zkPort</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          // Configure HDFS bolt          RecordFormat format = new DelimitedRecordFormat()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFieldDelimiter</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; // use "\t" instead of "," for field delimiter</span>          SyncPolicy syncPolicy = new CountSyncPolicy(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; // sync the filesystem after every 1k tuples</span>          FileRotationPolicy rotationPolicy = new TimedRotationPolicy(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.0</span>f, TimeUnit<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.MINUTES</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; // rotate files</span>          FileNameFormat fileNameFormat = new DefaultFileNameFormat()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withPath</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/storm/"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withPrefix</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"app_"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withExtension</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">".log"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">; // set file name format</span>          HdfsBolt hdfsBolt = new HdfsBolt()                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFsUrl</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs://h1:8020"</span>)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withFileNameFormat</span>(fileNameFormat)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withRecordFormat</span>(format)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withRotationPolicy</span>(rotationPolicy)                  <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.withSyncPolicy</span>(syncPolicy)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          // configure & build topology          TopologyBuilder builder = new TopologyBuilder()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setSpout</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka-reader"</span>, new KafkaSpout(spoutConf), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setBolt</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"to-upper"</span>, new KafkaWordToUpperCase(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shuffleGrouping</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"kafka-reader"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setBolt</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hdfs-bolt"</span>, hdfsBolt, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shuffleGrouping</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"to-upper"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setBolt</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"realtime"</span>, new RealtimeBolt(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shuffleGrouping</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"to-upper"</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          // submit topology          Config conf = new Config()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          String name = DistributeWordTopology<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.getSimpleName</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          if (args != null && args<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span> > <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {               String nimbus = args[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.put</span>(Config<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.NIMBUS</span>_HOST, nimbus)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setNumWorkers</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               StormSubmitter<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.submitTopologyWithProgressBar</span>(name, conf, builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.createTopology</span>())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          } else {               conf<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMaxTaskParallelism</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               LocalCluster cluster = new LocalCluster()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               cluster<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.submitTopology</span>(name, conf, builder<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.createTopology</span>())<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               Thread<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sleep</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">60000</span>)<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>               cluster<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shutdown</span>()<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;</span>          }     }}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li><li style="box-sizing: border-box; padding: 0px 5px;">147</li><li style="box-sizing: border-box; padding: 0px 5px;">148</li><li style="box-sizing: border-box; padding: 0px 5px;">149</li><li style="box-sizing: border-box; padding: 0px 5px;">150</li></ul>

上面代码中，名称为to-upper的Bolt将接收到的字符串行转换成大写以后，会将处理过的数据向后面的hdfs-bolt、realtime这两个Bolt各发一份拷贝，然后由这两个Bolt分别根据实际需要（实时/离线）单独处理。 
打包后，在Storm集群上部署并运行这个Topology：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/storm jar ~/storm-examples-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span>-SNAPSHOT<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.jar</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.shirdrn</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.examples</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.DistributeWordTopology</span> h1</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><div class="save_code tracking-ad" data-mod="popu_249" style="box-sizing: border-box; position: absolute; height: 60px; right: 30px; top: 5px; color: rgb(255, 255, 255); cursor: pointer; z-index: 2;"><a target=_blank target="_blank" style="color: rgb(202, 0, 0); box-sizing: border-box;"><img src="http://static.blog.csdn.net/images/save_snippets_01.png" style="border: none; box-sizing: border-box; max-width: 100%;" alt="" /></a></div>

可以通过Storm UI查看Topology运行情况，可以查看HDFS上生成的数据。