Kafka Study : Understand the Partitions and replicates

来源:互联网 发布:网络舆情监测方法 编辑:程序博客网 时间:2024/06/04 23:15
test configurations:
  1. start zookeeper at port 2181
  2. start kafka instance 0 at port 9092 (using default server.properties).
    • kafka_folder] bin/kafka-server-start.sh
  3. start kafka instance 1
    • create new server1.properteis, change the id=1, port=9093 and log directory;
    • bin/kafka-server-start.sh config/server1.properties
  4. start kafka instance 2
    • create new server2.properteis, change the id=2, port=9094 and log directory;
    • bin/kafka-server-start.sh config/server2.properties
  5. create topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-replicated-test-topic
  6. check the topic partion and replicates: 
          kafka-topics.sh --describe --topic my-replicated-test-topic --zookeeper localhost:2181
    • Topic:my-replicated-test-topic  PartitionCount:3        ReplicationFactor:2     Configs:
    •        Topic: my-replicated-test-topic Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
    •         Topic: my-replicated-test-topic Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
    •        Topic: my-replicated-test-topic Partition: 2    Leader: 2       Replicas: 2,0   Isr: 2,0

  1. publish message through different kafka instance:
    • kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-test-topic
      • 1st message
      • 2nd message
      • 3rd message
    • kafka-console-producer.sh --broker-list localhost:9093 --topic my-replicated-test-topic
      • 4th message
      • 5th message
      • 6th message
    • kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-test-topic
      • 7th message
      • 8th message
      • 9th message
  2. check the storage:                   kafka instance 0kafka instance 1kafka instance 2my-replicated-test-topic-replicate-0:
    2nd message, 6th message, 8th messagemy-replicated-test-topic-replicate-0:
    2nd message, 6th message, 8th message  my-replicated-test-topic-replicate-1:
    1st message, 5th message, 7th messagemy-replicated-test-topic-replicate-1:
    1st message, 5th message, 7th messagemy-replicated-test-topic-replicate-2:
    3rd message, 4th message, 9th message my-replicated-test-topic-replicate-2:
    3rd message, 4th message, 9th message 
  3. consumer from beginning from any broker:
  4. kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-test-topic --from-beginning
  5. 1st message
  6. 2nd message
  7. 5th message
  8. 7th message
  9. 6th message
  10. 8th message
  11. 3rd message
  12. 4th message
  13. 9th message
  14. we can see that: the order from one partition was assured, but from different partitions is not assured
0 0
原创粉丝点击