kafka常用操作

来源：互联网发布：JavaScript target 编辑：程序博客网时间：2024/06/04 23:00

1.启动集群：

export KAFKA_HEAP_OPTS=" -Xms3g -Xmx3g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 " nohup ./kafka-server-start.sh -daemon ../config/server.propertiesnohup ./kafka-server-start.sh ../config/server.properties >/dev/null 2>&1 &

2.停止集群：

./kafka-server-stop.sh

3.创建topic：

./kafka-topics.sh --create --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/kafka --replication-factor 2 --partitions 10 --topic testtopic

zookeeper指定其中一个节点即可(10.129.142.46:2181)

4.查看全部topic：

./kafka-topics.sh --list --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/testkafka

查看某个topic的详细信息:

./kafka-topics.sh --describe --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/testkafka --topic itil_topic_4038

5.发送消息(在一个终端启动Producer)：

./kafka-console-producer.sh --broker-list 10.49.133.77:9092,10.49.133.76:9092,10.49.133.75:9092 --topic itil_topic_4097

ctrl+c退出发送

6.接收消息(在一个终端启动Consumer)：

./kafka-console-consumer.sh --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/testkafka --topic itil_topic_4097 --from-beginning

7.删除一个topic:

./kafka-topics.sh  --delete --zookeeper 10.129.142.46:2181/kafka  --topic test_topic

配置文件中必须delete.topic.enable=true，否则只会标记为删除，而不是真正删除。

8.修改topic:

./kafka-topics.sh --alter  --zookeeper 10.129.142.46:2181/kafka --topic test_topic  --partitions 4 ./kafka-topics.sh —alter --zookeeper 10.129.142.46:2181/kafka  --topic test_topic --config key=value ./kafka-topics.sh —alter --zookeeper 10.129.142.46:2181/kafka  --topic test_topic --deleteConfig key (http://blog.jobbole.com/99195/)

9.kafka重新分区：

./kafka-reassign-partitions.sh --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/appnews_kafka2 --reassignment-json-file result.json --execute./kafka-reassign-partitions.sh --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/appnews_kafka2 --topics-to-move-json-file topic-to-move.json --broker-list "1,2,3,4,5" --generate{"topics": [{"topic": "itil_topic_2954"},{"topic": "itil_topic_3825"},{"topic": "itil_topic_3295"}], "version":1}

10.调整分片副本数

准备json文档addReplicas.json：

{    "version": 1,    "partitions": [        {            "topic": "itil_topic_4499",            "partition": 0,            "replicas": [                2,                1            ]        },        {            "topic": "itil_topic_4499",            "partition": 1,            "replicas": [                1,                2            ]        }, {            "topic": "itil_topic_4499",            "partition": 6,            "replicas": [                2,                1            ]        }, {            "topic": "itil_topic_4499",            "partition": 9,            "replicas": [                2,                1            ]        }    ]}

执行：

./kafka-reassign-partitions.sh --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/appnews_kafka3 --reassignment-json-file addReplicas.json --execute

查看执行情况：

./kafka-reassign-partitions.sh --zookeeper 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/appnews_kafka3 --reassignment-json-file addReplicas.json --verify

11.配置

相关URL：

http://kafka.apache.org/documentation.html#topic-config
https://m.oschina.net/blog/413649

12.kafka系统工具
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
查看topic的latest offset

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.49.133.77:9092,10.49.133.76:9092,10.49.133.75:9092 --topic itil_topic_2954 --time -1./kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect 10.129.142.46:2181/kafka --output-file abc.txt./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper 10.129.142.46:2181/kafka --topic itil_topic_2954 --broker-info  --group videoplay_3295./kafka-run-class.sh kafka.tools.SimpleConsumerShell --broker-list 10.49.133.77:9092,10.49.133.76:9092,10.49.133.75:9092 --topic itil_topic_3295 --partition 3./kafka-run-class.sh kafka.tools.VerifyConsumerRebalance  --group videoplay_3295 --zookeeper.connect 10.129.142.46:2181/kafka

12.C语言API:
https://github.com/edenhill/librdkafka
http://docs.confluent.io/2.0.0/clients/librdkafka/classRdKafka_1_1Conf.html
链接里加上：-lrdkafka -lz -lpthread -lrt.

13监控

KafkaOffsetMonitor启动命令：

nohup java -cp KafkaOffsetMonitor-assembly-0.2.1.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 10.129.142.46:2181,10.166.141.46:2181,10.166.141.47:2181/testkafka --port 80 --refresh 10.seconds --retain 2.days &

14.同类文章推荐：

Kafka集群操作指南:http://blog.jobbole.com/99195/

0 0