Kafka详细教程：下载、安装、配置与集群部署

1、下载

[root@node1 ~]# wget http://mirror.bit.edu.cn/apache/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgz--2017-10-28 09:16:07--  http://mirror.bit.edu.cn/apache/kafka/0.11.0.1/kafka_2.11-0.11.0.1.tgzResolving mirror.bit.edu.cn (mirror.bit.edu.cn)... 202.204.80.77, 2001:da8:204:2001:250:56ff:fea1:22Connecting to mirror.bit.edu.cn (mirror.bit.edu.cn)|202.204.80.77|:80... connected.HTTP request sent, awaiting response... 200 OKLength: 42031343 (40M) [application/octet-stream]Saving to: ‘kafka_2.11-0.11.0.1.tgz’100%[==============================================================================================>] 42,031,343  1.11MB/s   in 33s    2017-10-28 09:16:40 (1.22 MB/s) - ‘kafka_2.11-0.11.0.1.tgz’ saved [42031343/42031343][root@node1 ~]# 

2、安装

（1）解压缩

[root@node1 ~]# tar -zxvf kafka_2.11-0.11.0.1.tgz -C /opt

（2）进入根目录

[root@node1 ~]# cd /opt/kafka_2.11-0.11.0.1[root@node1 kafka_2.11-0.11.0.1]# lsbin  config  libs  LICENSE  NOTICE  site-docs[root@node1 kafka_2.11-0.11.0.1]# 

Kafka目录介绍
- /bin 操作kafka的可执行脚本，还包含windows下脚本
- /config 配置文件所在目录
- /libs 依赖库目录
- /logs 日志数据目录，目录kafka把server端日志分为5种类型，分为:server,request,state，log-cleaner，controller

3、配置

（1）统一修改配置文件

[root@node1 kafka_2.11-0.11.0.1]# vi config/server.properties

• brokerID唯一标示，集群中不能有重复的ID

# The id of the broker. This must be set to a unique integer for each broker.broker.id=1

• 打开监听

listeners=PLAINTEXT://:9092

• 修改日志目录

log.dirs=/var/log/kafka-logs

• 修改 zookeeper.connect

zookeeper.connect=node1:2181,node2:2181,node3:2181

4、分发

（1）分发软件包

[root@node1 ~]# scp -r /opt/kafka_2.11-0.11.0.1/ node2:/opt[root@node1 ~]# scp -r /opt/kafka_2.11-0.11.0.1/ node3:/opt

（2）修改node2 的配置

[root@node2 ~]# vi /opt/kafka_2.11-0.11.0.1/config/server.properties

修改内容

broker.id=2

（3）修改node3 的配置

[root@node3 ~]# vi /opt/kafka_2.11-0.11.0.1/config/server.properties

修改内容

broker.id=3

5、启动kafka

（1）先将zookeeper集群启动
（2）再启动kafka
启动node1

[root@node1 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &[1] 2576[root@node1 kafka_2.11-0.11.0.1]# 

启动node2

[root@node2 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &[1] 2526[root@node2 kafka_2.11-0.11.0.1]# 

启动node3

[root@node3 kafka_2.11-0.11.0.1]# bin/kafka-server-start.sh config/server.properties &[1] 7650[root@node3 kafka_2.11-0.11.0.1]# 

（3）查看进程

[root@node1 kafka_2.11-0.11.0.1]# jps2576 Kafka2551 QuorumPeerMain2874 Jps[root@node1 kafka_2.11-0.11.0.1]# 

[root@node2 kafka_2.11-0.11.0.1]# jps2499 QuorumPeerMain2821 Jps2526 Kafka[root@node2 kafka_2.11-0.11.0.1]# 

[root@node3 kafka_2.11-0.11.0.1]# jps7617 QuorumPeerMain7650 Kafka7945 Jps[root@node3 kafka_2.11-0.11.0.1]#