Storm源码阅读（一）：使用

来源：互联网发布：mac桌面工具栏不见了编辑：程序博客网时间：2024/05/21 17:26

Storm项目主页http://storm-project.net/，github主页https://github.com/nathanmarz/storm

storm源码是clojure和java的混合体

安装JRE

Clojure 运行在 JRE (JAVA Runtime Environment) 之上，因此，你需要先安装 JRE。

安装lein

Clojure项目管理工具：Leiningen，类似于maven。leiningen使用maven仓库进行包管理

http://leiningen-win-installer.djpowell.net/

lein基本操作

$ lein new [TEMPLATE] NAME # generate a new projectskeleton

$ lein test [TESTS] # run the tests in the TESTSnamespaces, or all tests

$ lein repl # launch an interactive REPL session

$ lein run -m my.namespace # run the -main function of anamespace

$ lein uberjar # package the project and dependencies asstandalone jar

$ lein deps //下载依赖到storm/lib

$ lein compile //编译生成storm.jar

$ lein install //安装到本地.m2库

$ lein version //查看version版本

$ lein uberjar //生成一个适合提交给storm集群的jar包

$ lein pom //生成pom.xml

类似pom文件，工程根目录下的 project.clj 内容类似

(defproject myproject"0.5.0-SNAPSHOT"

  :description"A project for doing things."

  :license"Eclipse Public License 1.0"

  :url"http://github.com/technomancy/myproject"

  :dependencies [[org.clojure/clojure"1.5.1"]]

  :plugins [[lein-ring"0.4.5"]])

Lein pom命令：根据project.clj生成pom.xml，导入eclipse。

安装

zeromq

$ wgethttp://download.zeromq.org/zeromq-2.1.7.tar.gz

$ tar -xzf zeromq-2.1.7.tar.gz

$ cdzeromq-2.1.7

$ ./configure

$ make

$ sudo make install

jzmq

$git clone https://github.com/nathanmarz/jzmq.git

$cd jzmq

$./autogen.sh

$./configure

$make

$sudo make install

Zookeeper配置

#zoo.cfg

server.1=192.168.119.24:5888:6888

server.2=192.168.119.25:5888:6888

server.3=192.168.119.26:5888:6888

storm配置

#storm.yaml

storm.zookeeper.servers:

- "192.168.119.24"

- "192.168.119.25"

- "192.168.119.26"

storm.zookeeper.port: 5181

nimbus.host: "192.168.119.24"

启动storm

storm nimbus #24

storm ui #24

storm supervisor #24, 25, 26

http://192.168.119.24:8080

生产环境使用supervisord保证出错退出时重启

storm-starter

storm-starters提供了storm的使用例子。如果不研究storm的实现，只是使用，可以直接按照storm-starter的例子编写代码。

$ git clone git@github.com:nathanmarz/storm-starter.git

$ cdstorm-starter

$ lein deps

$ lein compile

$ lein uberjar

$./storm jar storm-starter-0.0.1-SNAPSHOT-standalone.jarstorm.starter.ExclamationTopology exclamation-topology

Exception in thread "main"java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException:java.net.ConnectException: Connection refused

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:36)

atbacktype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:69)

at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:40)

at storm.starter.ExclamationTopology.main(ExclamationTopology.java:60)

Caused by:org.apache.thrift7.transport.TTransportException: java.net.ConnectException:Connection refused

at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)

at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)

at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:34)

... 4 more

Caused by: java.net.ConnectException:Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:529)

at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

... 6 more

需要在客户端机器上配置 ~/.storm/storm.yaml

[root@localhost bin]# mkdir ~/.storm

[root@localhost bin]# cp../conf/storm.yaml ~/.storm/storm.yaml

0 [main] INFO backtype.storm.StormSubmitter -Jar not uploaded to master yet. Submitting jar...

16 [main]INFO backtype.storm.StormSubmitter - Uploading topology jarstorm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar to assigned location:storm-local/nimbus/inbox/stormjar-178ac806-5246-4d28-8adc-9032eaf300eb.jar

536 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar toassigned location:storm-local/nimbus/inbox/stormjar-178ac806-5246-4d28-8adc-9032eaf300eb.jar

536 [main] INFO backtype.storm.StormSubmitter -Submitting topology exclamation-topology in distributed mode with conf{"topology.workers":3,"topology.debug":true}

982 [main] INFO backtype.storm.StormSubmitter -Finished submitting topology: exclamation-topology

Topology界面

成功！！！

Storm-release(部署目录)主要结构

|-- bin

| |-- storm

| |-- storm-local

| | |-- nimbus

| | | |-- inbox

| | | `-- stormdist

| | | `-- exclamation-topology-1-1376451279

| | | |-- stormcode.ser

| | | |-- stormconf.ser

| | | `-- stormjar.jar

| | |-- supervisor

| | | |-- isupervisor

| | | | |-- 1376449252196

| | | |-- localstate

| | | | |-- 1376458241022

| | | |-- stormdist

| | | | `-- exclamation-topology-1-1376451279

| | | | `-- stormjar.jar

| | | `-- tmp

| | `-- workers

| | `--eb0ad8a8-bf76-41e0-bfa2-84bcb2ab42b9

| | |-- heartbeats

| | | |-- 1376458241416

| | | |-- 1376458241416.version

| | `-- pids

| | `-- 10633

|-- conf

| `--storm.yaml

|-- lib

| `-- zookeeper-3.3.3.jar

|-- log4j

| `-- storm.log.properties

|-- logs

| |-- nimbus.log

| |-- supervisor.log

| |-- ui.log

| `-- worker-6703.log

|-- public

| |-- css

| | |-- bootstrap-1.1.0.css

| `-- js

| |-- jquery-1.6.2.min.js

`-- storm-0.8.2.jar

因为没有设置”storm.local.dir”，所以默认使用bin/storm-local目录。