安装配置kafka集群

以下为生产环境安装文档

1. 下载kafka并安装
wget https://archive.apache.org/dist/kafka/0.8.1/kafka_2.9.2-0.8.1.tgz
tar zxvf kafka_2.9.2-0.8.1.tgz
mv kafka_2.9.2-0.8.1 /usr/local/kafka


2. 修改配置文件
cd /usr/local/kafka
vim config/server.properties
broker.id=1            #每个节点用不同的值 如1/2/3
host.name=172.16.100.21         #绑定kafka本机IP地址
log.dirs=/data/kafka
zookeeper.connect=172.16.100.30:2181,172.16.100.31:2181,172.16.100.32:2181/kafka #配置zookeeper的地址，后面的kafka字符串是和zk里面创建的节点是一致的


3. 复制kafka到另外节点
scp -r /usr/local/kafka 172.16.100.22:/usr/local
mkdir -p /data/kafka


4. 配置环境变量
echo "export KAFKA_HOME=/usr/local/kafka" >> /etc/profile.d/app.sh
echo "export PATH=\$KAFKA_HOME/bin:\$PATH" >> /etc/profile.d/app.sh
source /etc/profile

5. 启动和停止
启动kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /dev/null &
停止kafka
/usr/local/kafka/bin/kafka-server-stop.sh

添加自启动（注意这里不要最后的&）
echo "source /etc/profile && /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties > /dev/null" >> /etc/rc.local

在zookeeper中查看
zkCli.sh
get /kafka/brokers/ids/1
get /kafka/brokers/ids/2

创建Topic
创建Topic(replication-factor是副本数量)
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 172.16.100.30:2181/kafka --replication-factor 3 --partitions 2 --topic mykafka
查看Topic
/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 172.16.100.30:2181/kafka
查看Topic详细
/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 172.16.100.30:2181/kafka

控制台测试(分别在两台kafka上面测试一下)
生产者
/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 172.16.100.22:9092 --topic mykafka
消费者
/usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 172.16.100.30/kafka --topic mykafka --from-beginning

6. 安装kafka monitor
mkdir /usr/local/kafka-offset-console
cd /usr/local/kafka-offset-console
mkdir logs
https://github.com/quantifind/KafkaOffsetMonitor/releases/tag/v0.2.0
wget https://github.com/quantifind/KafkaOffsetMonitor/releases/download/v0.2.0/KafkaOffsetMonitor-assembly-0.2.0.jar

vim monitor-start.sh
添加

#!/bin/bashexport JAVA_HOME=/usr/java/jdk1.7.0_79export MONITOR_HOME=/usr/local/kafka-offset-console$JAVA_HOME/bin/java -Xms512M -Xmx512M -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=512m -cp $MONITOR_HOME/KafkaOffsetMonitor-assembly-0.2.0.jar \     com.quantifind.kafka.offsetapp.OffsetGetterWeb \    --zk 172.16.100.30:2181,172.16.100.31:2181,172.16.100.32:2181/kafka\    --port 8086\    --refresh 10.seconds \    --retain 7.days 1>$MONITOR_HOME/logs/stdout.log 2>$MONITOR_HOME/logs/stderr.log &

chmod 755 monitor-start.sh
启动monitor
/usr/local/kafka-offset-console/monitor-start.sh
添加自启动

echo "/usr/local/kafka-offset-console/monitor-start.sh" >> /etc/rc.local

以下的不再使用，仅供参考

1. 安装zookeeper
http://blog.csdn.net/u013619834/article/details/41316957

连接zookeeper并添加节点
zkCli.sh -server 192.168.1.111:2181
create /kafka ""


2. 下载kafka并安装（注意如果kafka和zookeeper不在同一服务器，也要先安装jdk，我这里因为测试少用机器，所以是在一个上面安装的zk和kafka）
cd /usr/local/src
wget http://mirrors.cnnic.cn/apache/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgz
tar zxvf kafka_2.10-0.8.1.1.tgz
mv kafka_2.10-0.8.1.1 /usr/local/kafka

下载slf4j并复制slf4j-nop-1.7.2.jar
cd /usr/local/src
wget http://www.slf4j.org/dist/slf4j-1.7.2.tar.gz
tar zxvf slf4j-1.7.2.tar.gz
cd slf4j-1.7.2
cp slf4j-nop-1.7.2.jar /usr/local/kafka/libs


3. 修改配置文件
cd /usr/local/kafka
vim config/server.properties
broker.id=1            #每个节点用不同的值 如1/2/3
host.name=192.168.1.111         #绑定kafka本机IP地址
log.dirs=/data/kafka/data
zookeeper.connect=192.168.1.111:2181,192.168.1.112:2181,192.168.1.113:2181/kafka #配置zookeeper的地址，后面的kafka字符串是和zk里面创建的节点是一致的


复制到其他的节点并修改broker.id和host.name
scp -r kafka 192.168.1.112:/usr/local
scp -r kafka 192.168.1.113:/usr/local



4. 启动broker
创建目录
mkdir -p /data/kafka/data

启动kafka broker
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
停止kafka broker
/usr/local/kafka/bin/kafka-server-stop.sh
添加broker自启动
echo "/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &" >> /etc/rc.local



5. 创建Topic
创建Topic(replication-factor是副本数量)
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.1.111:2181/kafka --replication-factor 3 --partitions 2 --topic mykafka
查看Topic
/usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 192.168.1.111:2181/kafka
查看Topic详细
/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.1.111:2181/kafka


6. 控制台测试(分别在两台kafka上面测试一下)
生产者
/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 192.168.1.111:9092 --topic mykafka
消费者
/usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper 192.168.1.111:2181/kafka --topic mykafka --from-beginning