Storm0.9.4安装

环境：三台虚拟机，系统是CentOS6.5

1.关闭防火墙，配置hosts，添加集群中主机和IP的映射关系

?

1

2

3

4

5

6

[grid@hadoop4 ~]$cat /etc/hosts

127.0.0.1   localhost

::1         localhost

192.168.0.106    hadoop4

192.168.0.107    hadoop5

192.168.0.108    hadoop6

2.安装Java（JDK6或以上版本），配置JAVA_HOME 、CLASSPATH环境变量

?

1

2

3

4

5

6

[grid@hadoop4 ~]$cat .bash_profile

JAVA_HOME=/usr/java/jdk1.7.0_72

JRE_HOME=$JAVA_HOME/jre

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

exportJAVA_HOME JRE_HOME PATH CLASSPATH

3.安装python
先确定你系统自带的Python版本，如果是2.6.6或者以上的不需要安装

?

1

2

3

4

5

[grid@hadoop4 ~]$ python

Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2

Type "help","copyright", "credits" or "license"for more information.

>>>

4.搭建Zookeeper集群

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

##下载解压##

[grid@hadoop4 ~]$ wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

[grid@hadoop4 ~]$tar -axf zookeeper-3.4.6.tar.gz

##修改配置文件##

[grid@hadoop4 ~]$cd zookeeper-3.4.6/conf/

[grid@hadoop4 conf]$cp -p zoo_sample.cfg zoo.cfg

[grid@hadoop4 conf]$vi zoo.cfg

# The number of milliseconds of each tick

tickTime=2000  ##服务器心跳时间，单位毫秒

# The number of ticks that the initial

# synchronization phase can take

initLimit=10  ##投票选择新leader的初始化时间

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5  ##leader与follower心跳检测最大容忍时间，响应超过syncLimit*tickTime，leader认为follwer死掉，从服务器列表中删除follwer

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/home/grid/zookeeper-3.4.6/data ##数据目录，需要手动创建

# dataLogDir=   ##日志目录，不指定将使用和dataDir相同的设置

# the port at which the clients will connect

clientPort=2181##监听client连接的端口

##server.id=host:port:port，id是一个数字，表示这是第几号server，这个id也会被写到myid文件中；host是zookeeper服务器ip或主机名；第一个port是leader与follwer通讯所使用的端口；第二个port是选举leader时所使用的端口

server.1=hadoop4:2888:3888

server.2=hadoop5:2888:3888

server.3=hadoop6:2888:3888

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

  

##手动创建数据目录##

[grid@hadoop4 conf]$cd /home/grid/zookeeper-3.4.6

[grid@hadoop4 zookeeper-3.4.6]$mkdir data

##分发zookeeper##

[grid@hadoop4 zookeeper-3.4.6]$cd ..

[grid@hadoop4 ~]$scp -rp zookeeper-3.4.6 grid@hadoop5:/home/grid/

[grid@hadoop4 ~]$scp -rp zookeeper-3.4.6 grid@hadoop6:/home/grid/

##在数据目录下创建myid文件，写入id号，用来标识当前主机##

[grid@hadoop4 ~]$echo "1"> zookeeper-3.4.6/data/myid

[grid@hadoop5 ~]$echo "2"> zookeeper-3.4.6/data/myid

[grid@hadoop6 ~]$echo "3"> zookeeper-3.4.6/data/myid

##启动zookeeper##

[grid@hadoop4 ~]$ zookeeper-3.4.6/bin/zkServer.sh start

[grid@hadoop5 ~]$ zookeeper-3.4.6/bin/zkServer.sh start

[grid@hadoop6 ~]$ zookeeper-3.4.6/bin/zkServer.sh start

##查看zookeeper状态##

[grid@hadoop4 ~]$ zookeeper-3.4.6/bin/zkServer.sh status

JMX enabled by default

Using config: /home/grid/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

[grid@hadoop5 ~]$ zookeeper-3.4.6/bin/zkServer.sh status

JMX enabled by default

Using config: /home/grid/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: leader

[grid@hadoop6 ~]$ zookeeper-3.4.6/bin/zkServer.sh status

JMX enabled by default

Using config: /home/grid/zookeeper-3.4.6/bin/../conf/zoo.cfg

Mode: follower

5.安装Storm

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

##下载解压##

[grid@hadoop4 ~]$ wget http://mirrors.cnnic.cn/apache/storm/apache-storm-0.9.4/apache-storm-0.9.4.tar.gz

[grid@hadoop4 ~]$tar -zxf apache-storm-0.9.4.tar.gz

[grid@hadoop4 ~]$mv apache-storm-0.9.4 storm-0.9.4

  

##修改配置项##

[grid@hadoop4 conf]$ vim storm.yaml

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

  

########### These MUST be filled in for a storm configuration

##集群使用的Zookeeper集群地址

storm.zookeeper.servers:

    -"hadoop4"

    -"hadoop5"

    -"hadoop6"

storm.zookeeper.port: 2181

#

##集群的Nimbus机器的地址

nimbus.host: "hadoop4"

##Nimbus和Supervisor迚程用于存储少量状态，如jars、 confs等的本地磁盘目录，需要提前创建该目录并给以足够的访问权限

storm.local.dir:"/home/grid/storm-0.9.4/data"

##对于每个Supervisor工作节点，需要配置该工作节点可以运行的worker数量。每个worker占用一个单独的端口用于接收消息，该配置选项即用于定义哪些端口是可被worker使用。默认情况下，每个节点上可运行4个workers，分别在6700、 6701、 6702和6703端口上。

supervisor.slots.ports:

    - 6700

    - 6701

    - 6702

    - 6703

#

#

# ##### These may optionally be filled in:

#    

## List of custom serializations

# topology.kryo.register:

#     - org.mycompany.MyType

#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer

#

## List of custom kryo decorators

# topology.kryo.decorators:

#     - org.mycompany.MyDecorator

#

## Locations of the drpc servers

# drpc.servers:

#     - "server1"

#     - "server2"

  

## Metrics Consumers

# topology.metrics.consumer.register:

#   - class: "backtype.storm.metric.LoggingMetricsConsumer"

#     parallelism.hint: 1

#   - class: "org.mycompany.MyMetricsConsumer"

#     parallelism.hint: 1

#     argument:

#       - endpoint: "metrics-collector.mycompany.org"

## 有关的其他配置项可以参看：https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml ##

  

##创建数据目录##

[grid@hadoop4 conf]$cd /home/grid/storm-0.9.4/

[grid@hadoop4 storm-0.9.4]$mkdir data

  

##分发Storm##

[grid@hadoop4 ~]$scp -rp storm-0.9.4/ grid@hadoop5:/home/grid/

[grid@hadoop4 ~]$scp -rp storm-0.9.4/ grid@hadoop6:/home/grid/

  

##编辑环境变量##

[grid@hadoop4 ~]$ vim .bash_profile

exportSTORM_HOME=/home/grid/storm-0.9.4

exportPATH=$PATH:$STORM_HOME/bin

[grid@hadoop4 ~]$source .bash_profile

  

##启动Storm（确保zookeeper已经启动）##

[grid@hadoop4 ~]$ storm nimbus & ##在主节点上运行Nimbus后台程序

[grid@hadoop5 ~]$ storm supervisor & ##在工作节点上运行Supervisor后台程序

[grid@hadoop6 ~]$ storm supervisor &

[grid@hadoop4 ~]$ storm ui & ##在主节点上运行UI程序，启动后可以在浏览器上输入http://主节点的ip:port （默认8080端口）

[grid@hadoop4 ~]$ storm logviewer & ##在主节点上运行LogViewer程序，启动后在UI上通过点击相应的Woker来查看对应的工作日志

  

[grid@hadoop4 ~]$ jps

2959 QuorumPeerMain

3310 logviewer

3414 Jps

3228 nimbus

3289 core

[grid@hadoop5 ~]$ jps

2907 QuorumPeerMain

3215 Jps

3154 supervisor

[grid@hadoop6 ~]$ jps

3248 Jps

2935 QuorumPeerMain

3186 supervisor