本文共 39502 字，大约阅读时间需要 131 分钟。
本文档介绍kafka_2.11_2.1.1升级到2.12-2.4.1的具体操作方法. 其他版本大同小异, 详见
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep "ReplicationFactor:[[:space:]]*1"
单副本Topic在rolling restart过程中必定会受到影响
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|while read line;do echo "$(awk -F'Topic:' '{print $NF}' <<< "${line}"|awk -F 'Partition:' '{print $1}'|sed 's/[ \t]*//g' ) Isr: $(awk -F'Isr:' '{print $NF}' <<< "${line}"|tr "," "\n"|sort -n|tr "\n" ","|sed 's/,$/\n/g;'|sed 's/[ \t]*//g' )";done|awk '{printf "%-64s%-4s%-23s\n",$1,$2,$3}'test.t_batch Isr:8,13,174 test.t_template Isr:8,13,174 test.t_user_apply Isr:8,13,174 test.t_user_order Isr:8,13,174 test.t_user Isr:8,13,174 test.t_user_job_record Isr:8,13,174 使用如下语句输出只有一个Isr的Topic. (if (length(a) == 1)bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|awk '{split($NF,a,",");slen=asort(a,tA);}{if (length(a) == 1){ {printf $2" : ";for (i = 1; i <= length(a); i++){if (i==length(a)&&i==1){printf tA[i];}else{printf tA[i]",";}}}{printf "\n";}}}'|awk '{printf "%-64s%-23s\n",$1,$3}' 只有一个Isr的在rolling restart过程中必定会受到影响
当上一步检查步骤存在问题Topic时, 需要针对这些Topic增加副本, 或等待Isr>1.
本次线上升级我们已知所有Topic都只有一个副本, 所以全部需要增加副本数
增加副本会触发数据复制, 可能导致网卡打满, IO增高, 还是会影响业务
Kafka 支持为已有 topic 的分区增加副本因子(replication factor)，具体的方法就是使用kafka-reassign-partitions.sh并且为 topic分区增加额外的副本
理论上不影响生产和消费, 实测也确实没影响
我们先为__consumer_offsets增加副本, 先describe一下
#bin/kafka-topics.sh --zookeeper localhost:2182 --topic __consumer_offsets --describeTopic: __consumer_offsets PartitionCount: 50 ReplicationFactor: 1 Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600 Topic: __consumer_offsets Partition: 0 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 1 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 2 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 3 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 4 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 5 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 6 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 7 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 8 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 9 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 10 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 11 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 12 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 13 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 14 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 15 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 16 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 17 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 18 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 19 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 20 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 21 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 22 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 23 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 24 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 25 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 26 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 27 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 28 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 29 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 30 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 31 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 32 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 33 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 34 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 35 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 36 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 37 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 38 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 39 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 40 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 41 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 42 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 43 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 44 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 45 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 46 Leader: 174 Replicas: 174 Isr: 174 Topic: __consumer_offsets Partition: 47 Leader: 8 Replicas: 8 Isr: 8 Topic: __consumer_offsets Partition: 48 Leader: 13 Replicas: 13 Isr: 13 Topic: __consumer_offsets Partition: 49 Leader: 174 Replicas: 174 Isr: 174
目前副本因子是1, 现在我们要把它重设成3, 创建json文件/tmp/incr_replica.json
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":1,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":2,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":3,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":4,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":5,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":6,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":7,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":8,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":9,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":10,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":11,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":12,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":13,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":14,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":15,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":16,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":17,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":18,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":19,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":20,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":21,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":22,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":23,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":24,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":25,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":26,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":27,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":28,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":29,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":30,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":31,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":32,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":33,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":34,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":35,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":36,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":37,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":38,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":39,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":40,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":41,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":42,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":43,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":44,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":45,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":46,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":47,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":48,"replicas":[13,174,8]},{"topic":"__consumer_offsets","partition":49,"replicas":[13,174,8]}]} [root@bj1-mysql-dba-prod-01 kafka_2.11-2.1.1]# bin/kafka-reassign-partitions.sh --zookeeper localhost:2182 --reassignment-json-file /tmp/incr_replica.json --executeCurrent partition replica assignment{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":22,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[174],"log_dirs":["any"]}]}Save this to use as the --reassignment-json-file option during rollbackSuccessfully started reassignment of partitions. bin/kafka-reassign-partitions.sh --zookeeper localhost:2182 --reassignment-json-file /tmp/incr_replica.json --verifyStatus of partition reassignment: Reassignment of partition __consumer_offsets-22 completed successfullyReassignment of partition __consumer_offsets-30 completed successfullyReassignment of partition __consumer_offsets-8 completed successfullyReassignment of partition __consumer_offsets-21 completed successfullyReassignment of partition __consumer_offsets-4 completed successfullyReassignment of partition __consumer_offsets-27 completed successfullyReassignment of partition __consumer_offsets-7 completed successfullyReassignment of partition __consumer_offsets-9 completed successfullyReassignment of partition __consumer_offsets-46 completed successfullyReassignment of partition __consumer_offsets-25 completed successfullyReassignment of partition __consumer_offsets-35 completed successfullyReassignment of partition __consumer_offsets-41 completed successfullyReassignment of partition __consumer_offsets-33 completed successfullyReassignment of partition __consumer_offsets-23 completed successfullyReassignment of partition __consumer_offsets-49 completed successfullyReassignment of partition __consumer_offsets-47 completed successfullyReassignment of partition __consumer_offsets-16 completed successfullyReassignment of partition __consumer_offsets-28 completed successfullyReassignment of partition __consumer_offsets-31 completed successfullyReassignment of partition __consumer_offsets-36 completed successfullyReassignment of partition __consumer_offsets-42 completed successfullyReassignment of partition __consumer_offsets-3 completed successfullyReassignment of partition __consumer_offsets-18 completed successfullyReassignment of partition __consumer_offsets-37 completed successfullyReassignment of partition __consumer_offsets-15 completed successfullyReassignment of partition __consumer_offsets-24 completed successfullyReassignment of partition __consumer_offsets-38 completed successfullyReassignment of partition __consumer_offsets-17 completed successfullyReassignment of partition __consumer_offsets-48 completed successfullyReassignment of partition __consumer_offsets-19 completed successfullyReassignment of partition __consumer_offsets-11 completed successfullyReassignment of partition __consumer_offsets-13 completed successfullyReassignment of partition __consumer_offsets-2 completed successfullyReassignment of partition __consumer_offsets-43 completed successfullyReassignment of partition __consumer_offsets-6 completed successfullyReassignment of partition __consumer_offsets-14 completed successfullyReassignment of partition __consumer_offsets-20 completed successfullyReassignment of partition __consumer_offsets-0 completed successfullyReassignment of partition __consumer_offsets-44 completed successfullyReassignment of partition __consumer_offsets-39 completed successfullyReassignment of partition __consumer_offsets-12 completed successfullyReassignment of partition __consumer_offsets-45 completed successfullyReassignment of partition __consumer_offsets-1 completed successfullyReassignment of partition __consumer_offsets-5 completed successfullyReassignment of partition __consumer_offsets-26 completed successfullyReassignment of partition __consumer_offsets-29 completed successfullyReassignment of partition __consumer_offsets-34 completed successfullyReassignment of partition __consumer_offsets-10 completed successfullyReassignment of partition __consumer_offsets-32 completed successfullyReassignment of partition __consumer_offsets-40 completed successfully
再describe确认一下
[root@bj1-mysql-dba-prod-01 kafka_2.11-2.1.1]# bin/kafka-topics.sh --zookeeper localhost:2182 --topic __consumer_offsets --describeTopic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer Topic: __consumer_offsets Partition: 0 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 1 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 2 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 3 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 4 Leader: 174 Replicas: 13,174,8 Isr: 174,13,8 Topic: __consumer_offsets Partition: 5 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 6 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 7 Leader: 174 Replicas: 13,174,8 Isr: 174,13,8 Topic: __consumer_offsets Partition: 8 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 9 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 10 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 11 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 12 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 13 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 14 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 15 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 16 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 17 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 18 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 19 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 20 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 21 Leader: 13 Replicas: 13,174,8 Isr: 13,174,8 Topic: __consumer_offsets Partition: 22 Leader: 174 Replicas: 13,174,8 Isr: 174,13,8 Topic: __consumer_offsets Partition: 23 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 24 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 25 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 26 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 27 Leader: 13 Replicas: 13,174,8 Isr: 13,174,8 Topic: __consumer_offsets Partition: 28 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 29 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 30 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 31 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 32 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 33 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 34 Leader: 174 Replicas: 13,174,8 Isr: 174,13,8 Topic: __consumer_offsets Partition: 35 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 36 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 37 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 38 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 39 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 40 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 41 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 42 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 43 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 44 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 45 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 46 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13 Topic: __consumer_offsets Partition: 47 Leader: 8 Replicas: 13,174,8 Isr: 8,174,13 Topic: __consumer_offsets Partition: 48 Leader: 13 Replicas: 13,174,8 Isr: 13,8,174 Topic: __consumer_offsets Partition: 49 Leader: 174 Replicas: 13,174,8 Isr: 174,8,13
副本是3, 都是Isr, 这样就OK了
bin/kafka-topics.sh --zookeeper localhost:2182 --describe| grep "Topic: "| grep -v "__consumer_offsets"|awk -F':' '{print $2}' |awk '{print $1}' {"version":1,"partitions":[{"topic":"pd.t1","partition":0,"replicas":[13,174,8]},{"topic":"pd.t2","partition":0,"replicas":[13,174,8]},{"topic":"pd.t3","partition":0,"replicas":[13,174,8]},{"topic":"pd.t4","partition":0,"replicas":[13,174,8]},{"topic":"bk.t5","partition":0,"replicas":[13,174,8]},{"topic":"bk.t6","partition":0,"replicas":[13,174,8]},{"topic":"bk.t7","partition":0,"replicas":[13,174,8]},{"topic":"bk.t8","partition":0,"replicas":[13,174,8]},{"topic":"bk.t9","partition":0,"replicas":[13,174,8]},{"topic":"bk.t10","partition":0,"replicas":[13,174,8]}]} 返回必须为空bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep "ReplicationFactor:[[:space:]]*1"输出Isr需要>1bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|while read line;do echo "$(awk -F'Topic:' '{print $NF}' <<< "${line}"|awk -F 'Partition:' '{print $1}'|sed 's/[ \t]*//g' ) Isr: $(awk -F'Isr:' '{print $NF}' <<< "${line}"|tr "," "\n"|sort -n|tr "\n" ","|sed 's/,$/\n/g;'|sed 's/[ \t]*//g' )";done|awk '{printf "%-64s%-4s%-23s\n",$1,$2,$3}'使用如下语句输出只有一个Isr的Topic. (if (length(a) == 1)bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|awk '{split($NF,a,",");slen=asort(a,tA);}{if (length(a) == 1){ {printf $2" : ";for (i = 1; i <= length(a); i++){if (i==length(a)&&i==1){printf tA[i];}else{printf tA[i]",";}}}{printf "\n";}}}'|awk '{printf "%-64s%-23s\n",$1,$3}'以下两个返回都需要为空bin/kafka-topics.sh --zookeeper localhost:2182 --describe --under-replicated-partitionsbin/kafka-topics.sh --zookeeper localhost:2182 --describe --unavailable-partitions 增加副本数以后, prefferred replica是不变的, 在我这个例子里, 由于之前所有topic所有分区都只有一个副本112, 所以增加副本以后prefferred replica仍然是112. 虽然auto.leader.rebalance.enable默认为true, 但是由于kafka计算leader不均衡的方法是:
Kafka计算不均衡程度的逻辑实际上非常简单——该broker上的leader不是preferred replica的分区数/ broker上总的分区数。
胡夕. Apache Kafka实战 (Kindle 位置 4563-4564). 电子工业出版社. Kindle 版本.
所以没有卵用
我们需要进行分区重分配, 目的是更改prefferred replica
以往进行重分配的场景是kafka集群增加了新的broker, 比如源集群k1 k2 k3, 新增k4 k5. 原有Topic不会自动迁移到新节点上, 为了让k4 k5也能干活, 进行重分配, 将一部分topic迁移到k4 k5
重分配后应该也会产生一些leader选举, 对客户端也会造成影响
我们需要指明一个需要进行分区重分配的topic列表(建议少量多次)
# cat /tmp/topics-to-move.json{"topics": [{"topic":"ReconChannelMatch"}, {"topic":"ReconImportBill"}, {"topic":"__consumer_offsets"}], "version":1} # bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file /tmp/topics-to-move.json --broker-list "112,167,226" --generateCurrent partition replica assignment{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":22,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"ReconChannelMatch","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"ReconImportBill","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[112,167,226],"log_dirs":["any","any","any"]}]}Proposed partition reassignment configuration{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":49,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"ReconImportBill","partition":0,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"ReconChannelMatch","partition":0,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[226,167,112],"log_dirs":["any","any","any"]}]}[root@bj3-canal-canal-prod-01 kafka]# cat /tmp/topics-to-move.json{"topics": [{"topic":"ReconChannelMatch"}, {"topic":"ReconImportBill"}, {"topic":"__consumer_offsets"}], "version":1} 保存Proposed partition reassignment configuration后面的部分到
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/move_prefferred.json --execute
执行后可以使用--verify参数来查看分区重分配执行情况
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/move_prefferred.json --verify
以下步骤每个broker依次执行, 全部执行完成再进行下一个broker的操作
过程略, 只解压软件包, 复制旧版的配置文件server.properties, 增加如下参数配置
inter.broker.protocol.version=2.1default.replication.factor=3offsets.topic.replication.factor=3transaction.state.log.replication.factor=3transaction.state.log.min.isr=2min.insync.replicas=2
开两个窗口,分别启动一个producer和consumer
bin/kafka-console-producer.sh --broker-list 172.16.x.x:9092,172.16.x.x:9092,172.16.x.x:9092 --topic test_upgradebin/kafka-console-consumer.sh --bootstrap-server 172.16.x.x:9092,172.16.x.x:9092,172.16.x.x:9092 --topic test_upgrade
测试一下producer写数据consumer是否可以消费到, 我们在下文中称这个步骤为T1
# jps -m2224 WrapperSimpleApp CloudResetPwdUpdateAgent2432 ZooKeeperMain -server localhost:21829204 Jps -m21351 QuorumPeerMain /usr/local/zookeeper/bin/../conf/zoo_test.cfg32542 QuorumPeerMain /usr/local/zookeeper/bin/../conf/zoo.cfg28335 ZooKeeperMain -server localhost:218227279 Kafka config/server.properties# kill 27279
观察日志, 确认shutdown complete
tail -f /usr/local/kafka_2.11-2.1.1/logs/kafkaServer.out INFO [ReplicaFetcher replicaId=8, leaderId=174, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
执行T1测试, 如果生产或消费失败, 则启动旧版本broker, 排查好问题后再次升级
确认一下命令无输出
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --unavailable-partitions
启动
cd new_kafka_path./bin/kafka-server-start.sh -daemon config/server.properties确认无输出bin/kafka-topics.sh --zookeeper localhost:2182 --describe --under-replicated-partitions
执行T1测试, 如果生产或消费失败, 则启动旧版本broker, 排查好问题后再次升级
修改配置参数
inter.broker.protocol.version=2.4
逐一重启broker
转载地址：http://okvub.baihongyu.com/