kafka集群搭建整理

1.Download and install kafka

wget https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1/kafka_2.10-0.8.1.1.tgztar zvxf kafka_2.10-0.8.1.1.tgzcd kafka_2.10-0.8.1.1

---

2.Start zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

---

3.Start kafka

bin/kafka-server-start.sh config/server.properties

---

4.Create topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 --topics test

---

5.Open producer，send message

bin/kafka-console-producer.sh  --broler-list localhost:9092 --topic test

---

6.Open consumer, receive message

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

---

7.Build a multi broker cluster

7.1 Copy setting files from the first broker

cp config/server.properties config/server-1.propertiescp config/server.properties config/server-2.properties

7.2 Add code below to every new broker

 for broker1

broker.id = 1port = 9093log.dir = /tmp/kafka-log-1

  for broker2

broker.id = 2port = 9094log.dir = /tmp/kafka-log-2

==++In the cluster, broker.id pointed to only one certain node. In case overwrite the data, we have to set different port and log file for each broker in the same mechine.++==

7.3 Now we have three node and one already be started, we just need to start the tow new nodes.

bin/kafka-server-start.sh config/server-1.propertiesbin/kafka-server-start.sh config/server-2.properties

7.4 Create a topic which has three copies

bin/kafka-topics.sh --create --zookeeper lcoalhost:2181 --replication-factor 3 --partition 1 --topic my-replicated-topic

---

8. Now we have a cluster with three nodes, but how do we know information about each node? Run describe topics command and you will see.

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

Topic:my-replication-topic PartitionCount:1 Configs:

Topic:my-replication-topic Partition:1 Leader:0 Replicas:2,0,1 Isr:0,1,2

Notification: The first row show information of all the partitions. Below the first row, every partition has a row to show detail of its own information. Since there is only one partition for the topic, there is only one row.

Leader: Manage meaage reading and meaage writing

Replicas：List all broker

Isr: List all the broker which is in servicing

9. Test the fault-durable capbility of kafka

From information above we know that broker 0 is the leader, then we open this broker and send messages

bin/kafka-console-producer.sh  --broler-list localhost:9092 --topic my-replicated-topic

And we open a consumer to receive the message

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic --from-beginning 

Then we kill broker 0

netstat -tnlp

find broker 0（localhost:9092）,remember PID, and kill it

kill -9 PID

And use command ‘describe’ we can see kafka chose another broker as the new leader and we use this new leader to send massege , the consumer can receive the message normally.