Kafka集群动态修改复制因子以及数据留存时间

之前用的kafka的默认配置数据留存配置的是留存7天，并且在创建kafka topic的时候指定的副本数量为3，但是后面发现数据量太大了，一天有200G的数据，如果设置副本数为3份，然后还保留7天，那么存储最大会占用接近5个T的数据，所以决定把数据复制因子改为2，然后将数据留存时间改为12个小时。

以为修改复制因子一条语句就搞定，结果发现还有点麻烦，操作如下，仅供参考：

• 1.查看TOPIC的详细信息：

/home/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --topic newiot002 --describe --zookeeper 10.104.123.231:2181,10.104.138.20:2181,10.104.54.172:2181

结果如下：

Topic:newiot002 PartitionCount:28   ReplicationFactor:3 Configs:    Topic: newiot002    Partition: 0    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3    Topic: newiot002    Partition: 1    Leader: 2   Replicas: 2,3,1 Isr: 2,3,1    Topic: newiot002    Partition: 2    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2    Topic: newiot002    Partition: 3    Leader: 1   Replicas: 1,3,2 Isr: 1,3,2    Topic: newiot002    Partition: 4    Leader: 2   Replicas: 2,1,3 Isr: 2,1,3    Topic: newiot002    Partition: 5    Leader: 3   Replicas: 3,2,1 Isr: 3,2,1    Topic: newiot002    Partition: 6    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3    Topic: newiot002    Partition: 7    Leader: 2   Replicas: 2,3,1 Isr: 2,3,1    Topic: newiot002    Partition: 8    Leader: 3   Replicas: 3,1,2 Isr: 3,1,2    Topic: newiot002    Partition: 9    Leader: 1   Replicas: 1,3,2 Isr: 1,3,2    Topic: newiot002    Partition: 10   Leader: 2   Replicas: 2,1,3 Isr: 2,1,3    Topic: newiot002    Partition: 11   Leader: 3   Replicas: 3,2,1 Isr: 3,2,1    Topic: newiot002    Partition: 12   Leader: 1   Replicas: 1,2,3 Isr: 1,2,3    Topic: newiot002    Partition: 13   Leader: 2   Replicas: 2,3,1 Isr: 2,3,1    Topic: newiot002    Partition: 14   Leader: 3   Replicas: 3,1,2 Isr: 3,1,2    Topic: newiot002    Partition: 15   Leader: 1   Replicas: 1,3,2 Isr: 1,3,2    Topic: newiot002    Partition: 16   Leader: 2   Replicas: 2,1,3 Isr: 2,1,3    Topic: newiot002    Partition: 17   Leader: 3   Replicas: 3,2,1 Isr: 3,2,1    Topic: newiot002    Partition: 18   Leader: 1   Replicas: 1,2,3 Isr: 1,2,3    Topic: newiot002    Partition: 19   Leader: 2   Replicas: 2,3,1 Isr: 2,3,1    Topic: newiot002    Partition: 20   Leader: 3   Replicas: 3,1,2 Isr: 3,1,2    Topic: newiot002    Partition: 21   Leader: 1   Replicas: 1,3,2 Isr: 1,3,2    Topic: newiot002    Partition: 22   Leader: 2   Replicas: 2,1,3 Isr: 2,1,3    Topic: newiot002    Partition: 23   Leader: 3   Replicas: 3,2,1 Isr: 3,2,1    Topic: newiot002    Partition: 24   Leader: 1   Replicas: 1,2,3 Isr: 1,2,3    Topic: newiot002    Partition: 25   Leader: 2   Replicas: 2,3,1 Isr: 2,3,1    Topic: newiot002    Partition: 26   Leader: 3   Replicas: 3,1,2 Isr: 3,1,2    Topic: newiot002    Partition: 27   Leader: 1   Replicas: 1,3,2 Isr: 1,3,2

• 2.创建一个exchange-rep.json文件用于描述修改的备份

在 /home/kafka/kafka_2.11-0.10.1.1/config/ 目录下

{        "partitions":                [                {                        "topic": "newiot002",                        "partition": 0,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 1,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 2,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 3,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 4,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 5,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 6,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 7,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 8,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 9,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 10,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 11,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 12,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 13,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 14,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 15,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 16,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 17,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 18,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 19,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 20,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 21,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 22,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 23,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 24,                        "replicas": [1,2]                },                {                        "topic": "newiot002",                        "partition": 25,                        "replicas": [2,3]                },                {                        "topic": "newiot002",                        "partition": 26,                        "replicas": [3,1]                },                {                        "topic": "newiot002",                        "partition": 27,                        "replicas": [1,2]                }                ],        "version":1}

• 3.执行语句进行修改：

/home/kafka/kafka_2.11-0.10.1.1/bin/kafka-reassign-partitions.sh --zookeeper 10.104.123.231:2181,10.104.138.20:2181,10.104.54.172:2181 --reassignment-json-file /home/kafka/kafka_2.11-0.10.1.1/config/exchange-rep.json --execute

结果如下：

Current partition replica assignment{"version":1,"partitions":[{"topic":"newiot002","partition":17,"replicas":[3,2,1]},{"topic":"newiot002","partition":16,"replicas":[2,1,3]},{"topic":"newiot002","partition":11,"replicas":[3,2,1]},{"topic":"newiot002","partition":12,"replicas":[1,2,3]},{"topic":"newiot002","partition":3,"replicas":[1,3,2]},{"topic":"newiot002","partition":27,"replicas":[1,3,2]},{"topic":"newiot002","partition":23,"replicas":[3,2,1]},{"topic":"newiot002","partition":20,"replicas":[3,1,2]},{"topic":"newiot002","partition":4,"replicas":[2,1,3]},{"topic":"newiot002","partition":13,"replicas":[2,3,1]},{"topic":"newiot002","partition":21,"replicas":[1,3,2]},{"topic":"newiot002","partition":9,"replicas":[1,3,2]},{"topic":"newiot002","partition":8,"replicas":[3,1,2]},{"topic":"newiot002","partition":10,"replicas":[2,1,3]},{"topic":"newiot002","partition":14,"replicas":[3,1,2]},{"topic":"newiot002","partition":5,"replicas":[3,2,1]},{"topic":"newiot002","partition":19,"replicas":[2,3,1]},{"topic":"newiot002","partition":6,"replicas":[1,2,3]},{"topic":"newiot002","partition":2,"replicas":[3,1,2]},{"topic":"newiot002","partition":22,"replicas":[2,1,3]},{"topic":"newiot002","partition":25,"replicas":[2,3,1]},{"topic":"newiot002","partition":24,"replicas":[1,2,3]},{"topic":"newiot002","partition":1,"replicas":[2,3,1]},{"topic":"newiot002","partition":15,"replicas":[1,3,2]},{"topic":"newiot002","partition":26,"replicas":[3,1,2]},{"topic":"newiot002","partition":0,"replicas":[1,2,3]},{"topic":"newiot002","partition":18,"replicas":[1,2,3]},{"topic":"newiot002","partition":7,"replicas":[2,3,1]}]}Save this to use as the --reassignment-json-file option during rollbackSuccessfully started reassignment of partitions.

• 4.检查执行情况：

/home/kafka/kafka_2.11-0.10.1.1/bin/kafka-reassign-partitions.sh --zookeeper 10.104.123.231:2181,10.104.138.20:2181,10.104.54.172:2181 --reassignment-json-file /home/kafka/kafka_2.11-0.10.1.1/config/exchange-rep.json --verify

上面全部显示successful 说明都执行成功！

---

下面修改数据默认留存时间

1.刚开始用如下脚本进行修改：

/home/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --alter --zookeeper 10.104.123.231:2181,10.104.138.20:2181,10.104.54.172:2181 --topic newiot002 --config delete.retention.ms=43200000

然后提示该脚本的这个功能以及被弃用了，让我使用kafka-configs.sh脚本操作

然后我就用kafka-configs.sh这个脚本来操作

查看提示，且提示如下：

Add/Remove entity config for a topic, client, user or brokerOption              Description                            ------              -----------                            --add-config        Key Value pairs of configs to add.                           Square brackets can be used to group                       values which contain commas: 'k1=v1,                       k2=[v1,v2,v2],k3=v3'. The following                        is a list of valid configurations:                         For entity_type 'topics':                                    cleanup.policy                                                compression.type                                              delete.retention.ms                                           file.delete.delay.ms                                          flush.messages                                                flush.ms                                                      follower.replication.throttled.                             replicas                                                     index.interval.bytes                                          leader.replication.throttled.replicas                         max.message.bytes                                             message.format.version                                        message.timestamp.difference.max.ms                           message.timestamp.type                                        min.cleanable.dirty.ratio                                     min.compaction.lag.ms                                         min.insync.replicas                                           preallocate                                                   retention.bytes                                               retention.ms                                                  segment.bytes                                                 segment.index.bytes                                           segment.jitter.ms                                             segment.ms                                                    unclean.leader.election.enable                            For entity_type 'brokers':                                     follower.replication.throttled.rate                           leader.replication.throttled.rate                         For entity_type 'users':                                       producer_byte_rate                                            consumer_byte_rate                                        For entity_type 'clients':                                     producer_byte_rate                                            consumer_byte_rate                                        Entity types 'users' and 'clients' may                       be specified together to update                            config for clients of a specific                           user.                                --alter             Alter the configuration for the entity.--delete-config     config keys to remove 'k1,k2'          --describe          List configs for the given entity.     --entity-default    Default entity name for clients/users                        (applies to corresponding entity                           type in command line)                --entity-name       Name of entity (topic name/client                            id/user principal name/broker id)    --entity-type       Type of entity                                               (topics/clients/users/brokers)       --force             Suppress console prompts               --help              Print usage information.               --zookeeper <urls>  REQUIRED: The connection string for                          the zookeeper connection in the form                       host:port. Multiple URLS can be                            given to allow fail-over.

按照提示用如下脚本搞定：

/home/kafka/kafka_2.11-0.10.1.1/bin/kafka-configs.sh --alter --zookeeper 10.104.123.231:2181,10.104.138.20:2181,10.104.54.172:2181 --entity-name newiot002 --entity-type topics --add-config delete.retention.ms=43200000

返回

Updated config for entity: topic 'newiot002'.

但是发现数据并没有被删除，最后还是使用修改conf配置文件的方法，重启了一次kafka 搞定，但是在生产上面直接启停这种方法不好，具体不停 让配置生效下次我再试一试。