zookeeper+activemq配置消息中间件集群 服务器配置

好久没有更新博客了。最近由于项目并发请求压力比较大，所以着手改进架构，引入消息中间件集群作为一个缓冲消息队列。

需求：

1、将大量的WebService请求报文发送到mq集群之中，并保持消息先后顺序

2、保证每个消息的可靠性

3、维护MQ服务器的可扩展性

综合考虑，决定使用Apache的 activemq。接触的时候，新的activemq 5.10已经出，所以，本项目使用的是activemq5.10。activemq5.10要求jdk6+，由于项目立项比较老，所以将项目整体迁移到jdk1.7进行部署，并引入activemq服务器。OK，废话不多说，开始。

首先，选三台服务器。

      之所以选三台，是因为zookeeper推荐最低三台配置，这样可以保持最大的可用服务器数，具体的这里不多说，可以查阅zookeeper的安装配置，以后有机会，详细讲解zookeeper。

我这里选择三台linux服务器，ip地址分别为：192.168.120.241,192.168.120.242,192.168.120.171。系统均使用centos6.4

1、创建账号

      useradd amqbroker;  //默认会创建amqbroker的群组，并且amqbroker会自动加入amqbroker群组。

      passwd amqbroker;修改密码

      cd /home/amqbroker  切换目录

2、下载zookeeper

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz

      tar -xvf  zookeeper-3.3.6.tar.gz

修改配置，mv zoo_sample.cfg zoo.cfg

     vi zoo.cfg,内容如下：

  1 # The number of milliseconds of each tick  2 tickTime=2000  3 # The number of ticks that the initial   4 # synchronization phase can take  5 initLimit=10  6 # The number of ticks that can pass between   7 # sending a request and getting an acknowledgement  8 syncLimit=5  9 # the directory where the snapshot is stored. 10 # do not use /tmp for storage, /tmp here is just  11 # example sakes. 12  13 dataDir=/home/amqbroker/zkdir/data 14 dataLogDir=/home/amqbroker/zkdir/log 15 # the port at which the clients will connect 16 clientPort=2181 17 # the maximum number of client connections. 18 # increase this if you need to handle more clients 19 #maxClientCnxns=60 20 # 21 # Be sure to read the maintenance section of the  22 # administrator guide before turning on autopurge. 23 # 24 # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance 25 # 26 # The number of snapshots to retain in dataDir 27 #autopurge.snapRetainCount=3 28 # Purge task interval in hours 29 # Set to "0" to disable auto purge feature 30 #autopurge.purgeInterval=1 31  32  33 ##three servers of this cluster 34 server.1=192.168.120.241:2888:3888 35 server.2=192.168.120.242:2888:3888 36 server.3=192.168.120.171:2888:3888

 dataDir=/home/amqbroker/zkdir/data
 dataLogDir=/home/amqbroker/zkdir/log  这两行说明了，需要创建一个文件夹，分别存放data以及log，所以，在/home/amqbroker下创建文件夹zkdir，也可以创建在其他地方，只要配置中指定就好

在zkdir/data下，创建myid文件，并写入与ip地址相称的服务器编号，比如，192.168.120.241,写入 1;echo 1>myid;

其他两台机器配置相同，不过myid文件写入相应的id号

启动服务：

cd /home/amqbroker/zookeeper-3.3.6/bin

./zkServer.sh start

可以使用zookeeper自带的客户端测试：

./zkCli.sh -server 192.168.120.241:2181

其他两台一样配置

3、下载activemq-5.10

wget  http://archive.apache.org/dist/activemq/5.10.1/apache-activemq-5.10.1-bin.tar.gz

tar -xvf apache-activemq-5.10.1-bin.tar.gz

vi apache-activemq-5.10.1/conf/activemq.xml

<pre name="code" class="java">  1 <!--  2     Licensed to the Apache Software Foundation (ASF) under one or more  3     contributor license agreements.  See the NOTICE file distributed with  4     this work for additional information regarding copyright ownership.  5     The ASF licenses this file to You under the Apache License, Version 2.0  6     (the "License"); you may not use this file except in compliance with  7     the License.  You may obtain a copy of the License at  8   9     http://www.apache.org/licenses/LICENSE-2.0 10  11     Unless required by applicable law or agreed to in writing, software 12     distributed under the License is distributed on an "AS IS" BASIS, 13     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14     See the License for the specific language governing permissions and 15     limitations under the License. 16 --> 17 <!-- START SNIPPET: example --> 18 <beans 19   xmlns="http://www.springframework.org/schema/beans" 20   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 21   xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd 22   http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> 23  24     <!-- Allows us to use system properties as variables in this configuration file --> 25     <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> 26         <property name="locations"> 27             <value>file:${activemq.conf}/credentials.properties</value> 28         </property> 29     </bean> 30  31    <!-- Allows accessing the server log --> 32     <!-- 33     <bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery" 34           lazy-init="false" scope="singleton" 35           init-method="start" destroy-method="stop"> 36     </bean> 37     --> 38     <!-- 39         The <broker> element is used to configure the ActiveMQ broker. 40     --> 41     <broker xmlns="http://activemq.apache.org/schema/core" brokerName="job-broker" dataDirectory="${activemq.data}"> 42  43         <destinationPolicy> 44             <policyMap> 45               <policyEntries>

 46              <!-- Set the following policy on all queues using the '>' wildcard --> 47                 <policyEntry queue="QUEUE.SH.LTE.IMPORT"> 48                     <deadLetterStrategy> 49                             <!--Use the prefix 'DLQ.' for the destination name, and make the DLQ a queue rather than a topic --> 50                         <individualDeadLetterStrategy queuePrefix="SH.LTE.DLQ." useQueueForQueueMessages="true" /> 51                     </deadLetterStrategy> 52                 </policyEntry> 53                 <policyEntry queue="QUEUE.STTTTT"> 54                     <deadLetterStrategy> 55                          <!--Use the prefix 'DLQ.' for the destination name, and make the DLQ a queue rather than a topic --> 56                           <individualDeadLetterStrategy queuePrefix="TTT.LTE.SH.DLQ." useQueueForQueueMessages="true" /> 57                       </deadLetterStrategy> 58                   </policyEntry> 59                 <policyEntry topic=">" > 60                     <!-- The constantPendingMessageLimitStrategy is used to prevent 61                          slow topic consumers to block producers and affect other consumers 62                          by limiting the number of messages that are retained 63                          For more information, see: 64  65                          http://activemq.apache.org/slow-consumer-handling.html 66  67                     --> 68                   <pendingMessageLimitStrategy> 69                     <constantPendingMessageLimitStrategy limit="1000"/> 70                   </pendingMessageLimitStrategy> 71                 </policyEntry> 72               </policyEntries> 73             </policyMap> 74         </destinationPolicy> 75  76  77         <!-- 78             The managementContext is used to configure how ActiveMQ is exposed in 79             JMX. By default, ActiveMQ uses the MBean server that is started by 80             the JVM. For more information, see: 81  82             http://activemq.apache.org/jmx.html 83         --> 84         <managementContext> 85             <managementContext createConnector="false"/> 86         </managementContext> 87  88         <!-- 89             Configure message persistence for the broker. The default persistence

 89             Configure message persistence for the broker. The default persistence 90             mechanism is the KahaDB store (identified by the kahaDB tag). 91             For more information, see: 92  93             http://activemq.apache.org/persistence.html 94         --> 95  96      <!-- 97      <persistenceAdapter> 98          <kahaDB directory="${activemq.data}/kahadb"/> 99          </persistenceAdapter>100      -->101      <persistenceAdapter>102         <replicatedLevelDB 103              directory="${activemq.data}/leveldb"104              replicas="3"105              bind="tcp://0.0.0.0:0"106              zkAddress="192.168.120.171:2181,192.168.120.241:2181,192.168.120.242:2181"107              zkPassword=""108              hostname="192.168.120.241"109              sync="local_disk"110              zkPath="/activemq/leveldb-stores"/>111      </persistenceAdapter>112 113           <!--114             The systemUsage controls the maximum amount of space the broker will115             use before disabling caching and/or slowing down producers. For more information, see:116             http://activemq.apache.org/producer-flow-control.html117           -->118           <systemUsage>119             <systemUsage>120                 <memoryUsage>121                     <memoryUsage percentOfJvmHeap="70" />122                 </memoryUsage>123                 <storeUsage>124                     <storeUsage limit="100 gb"/>125                 </storeUsage>126                 <tempUsage>127                     <tempUsage limit="50 gb"/>128                 </tempUsage>129             </systemUsage>130         </systemUsage>131 132         <!--133             The transport connectors expose ActiveMQ over a given protocol to

134             clients and other brokers. For more information, see:135 136             http://activemq.apache.org/configuring-transports.html137         -->138         <transportConnectors>139             <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->140             <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104    857600"/>141             <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=1048576    00"/>142             <transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=1048    57600"/>143             <transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=1048576    00"/>144             <transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"    />145         </transportConnectors>146 147         <!-- destroy the spring context on shutdown to stop jetty -->148         <shutdownHooks>149             <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />150         </shutdownHooks>151 152     </broker>153 154     <!--155         Enable web consoles, REST and Ajax APIs and demos156         The web consoles requires by default login, you can disable this in the jetty.xml file157 158         Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details159     -->160     <import resource="jetty.xml"/>161 162 </beans>

以上是192.168.120.241的activemq的activemq.xml配置文件配置。因为使用的是zookeeper，所以注意persistenceAdapter这个节点配置，如果有zkpassword，则填写，否则不要填写。按照上边的配置，zkpassword是不需要写的。

还有broker配置中，有一些policyEntity，是定制的，比如DLQ的配置，是因为针对不同的队列，重发机制的最大次数重发完以后，如果还有异常，则进入该队列定制的DLQ之中，详细的一些参数设置，请参考activemq官方文档。

另外，配置中应该有安全配置，目前没有写入，后续在说

另外两台服务器的配置与该文档相似，唯一不同就是    hostname="192.168.120.241"这里，需要写入各自的ip地址

启动activemq服务器

cd /home/amqbroker/activemq***/bin

./activemq start

三台服务器启动类似

然后，在浏览器地址栏里输入：

http://192.168.120.241:8161/admin/queues.jsp

http://192.168.120.242:8161/admin/queues.jsp

http://192.168.120.171:8161/admin/queues.jsp

因为使用zookeeper做负载均衡，三台只有一台是master，其他两台处于等待状态，所以只有其中一台提供服务，但一旦这台服务器宕机以后，会有另外一台顶替上来，所以其他几个ip地址是打不开的，只有一台能打开

在客户端使用的时候，使用 failover进行配置的，如下：

##使用replicaDB方式来搭建activemq，使用zookeeper集群进行负载均衡以及数据同步
mq.broker.url=failover:(tcp://192.168.120.241:61616,tcp://192.168.120.242:61616,tcp://192.168.120.171:61616)?initialReconnectDelay=1000

在下一节说spring配置的时候再讲