kafka集群安装

1.zookeeper下载

 wget -c http://apache.fayea.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz

2.分别在两台台服务器上安装kafka：

tar zxvf kafka_2.11-0.9.0.0.tgz

修改每台服务器的config/server.properties

broker.id： 唯一，填数字，本文中分别为0/1

host.name：唯一，填服务器IP

zookeeper.connect=192.168.40.134:2181,192.168.40.132:2181

先启动zookeeper服务:（zookeeper安装详情见http://blog.csdn.net/shandadadada/article/details/48303019）

再在每台机器上执行：

bin/kafka-server-start.sh config/server.properties  &

注释：& 是指退出命令行仍运行

启动完毕，可用jps查看进程

root@iZ94uoxhu56Z:/home/kafka/kafka_2.11-0.9.0.0# jps24548 QuorumPeerMain20532 Bootstrap3878 Kafka3955 Jps27832 Bootstrap

3.创建topic

root@iZ94uoxhu56Z:/home/kafka/kafka_2.11-0.9.0.0# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic mykafka Created topic "mykafka".

查看topic

root@iZ94uoxhu56Z:/home/kafka/kafka_2.11-0.9.0.0# bin/kafka-topics.sh --list --zookeeper localhost:2181mykafkatest-topic

查看详细信息：

root@iZ94uoxhu56Z:/home/kafka/kafka_2.11-0.9.0.0# bin/kafka-topics.sh --describe --zookeeper 192.168.40.134:2181Topic:mykafka   PartitionCount:1        ReplicationFactor:2     Configs:        Topic: mykafka  Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0Topic:test-topic        PartitionCount:1        ReplicationFactor:1     Configs:        Topic: test-topic       Partition: 0    Leader: 0       Replicas: 0     Isr: 0

发送消息：

bin/kafka-console-producer.sh --broker-list 192.168.40.134:9092 --topic mykafka

接受消息：

bin/kafka-console-consumer.sh --zookeeper 192.168.40.133:2181 --topic mykafka --from-beginningsfsaasdgsfaasfdasadf13

去zk上看kafka集群

[zk: localhost:2181(CONNECTED) 5] ls /  [admin, zookeeper, consumers, config, controller, zk-fifo, storm, brokers, controller_epoch]  [zk: localhost:2181(CONNECTED) 6] ls /brokers   ----> 查看注册在zk内的kafka  [topics, ids]  [zk: localhost:2181(CONNECTED) 7] ls /brokers/ids  [112, 110, 111]  [zk: localhost:2181(CONNECTED) 8] ls /brokers/ids/112  []  [zk: localhost:2181(CONNECTED) 9] ls /brokers/topics   [test]  [zk: localhost:2181(CONNECTED) 10] ls /brokers/topics/test   [partitions]  [zk: localhost:2181(CONNECTED) 11] ls /brokers/topics/test/partitions  [2, 1, 0]  [zk: localhost:2181(CONNECTED) 12] 

遇到的问题：
1.启动kafka的时候报内存不足错误

你可以通过编辑调整JVM堆大小kafka-server-start.sh，zookeeper-server-start.sh等：

export KAFKA_HEAP_OPTS=”-Xmx1G -Xms1G”

的-Xms参数指定的最小堆大小。让你的服务器至少开始，试着改变它使用较少的内存。因为你只有512M，你应该改变最大堆大小（-Xmx）太：

export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"

我不知道是什么的最小内存要求卡夫卡在默认的配置文件是你可能需要调整的消息的大小在卡夫卡得到它运行。

2.brokerid不匹配问题

kafka.common.InconsistentBrokerIdException: Configured brokerId 1 doesn't match stored brokerId 0 in meta.properties        at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)        at kafka.server.KafkaServer.startup(KafkaServer.scala:175)        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)        at kafka.Kafka$.main(Kafka.scala:67)        at kafka.Kafka.main(Kafka.scala)[2015-12-11 09:49:20,175] INFO shutting down (kafka.server.KafkaServer)

doesn’t match stored brokerId 0 in meta.properties

错误的原因是log.dirs目录下的meta.properties中配置的broker.id和配置目录下的server.properties中的broker.id不一致了，解决问题的方法是将两者修改一致后再重启。

3.客户端连接kafka报错：

[2015-06-02 16:23:04,375] WARN Fetching topic metadata with correlation id 0 for topics [Set(eventbustopic)] from broker [id:1,host:SOME_HOST,port:9092] failed (kafka.client.ClientUtils$)java.nio.channels.ClosedChannelException                                           at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)               at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)            at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)    at kafka.producer.SyncProducer.send(SyncProducer.scala:113)                    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)            at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)            at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)         [2015-06-02 16:23:04,515] WARN Fetching topic metadata with correlation id 0 for topics [Set(eventbustopic)] from broker [id:0,host:SOME_HOST,port:9092] failed (kafka.client.ClientUtils$)java.nio.channels.ClosedChannelException                                           at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)               at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)            at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)    at kafka.producer.SyncProducer.send(SyncProducer.scala:113)                    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)            at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)            at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

是由于Kafka server.properties中的advertised.host.name 属性没有配置。这个属性代表客户端访问Kafka的地址。如果没配置默认是主机名。

bin/kafka-console-producer.sh --broker-list 114.215.173.49:9092,112.74.88.53:9092 --topic mykafka

bin/kafka-console-consumer.sh --zookeeper 112.74.88.53:2181,115.29.175.58:2181 --topic mykafka --from-beginning