MSK kafka topic 副本数更新

0x00 TLDR;

在仅有两个Broker的MSK环境中，使用单副本的Topic面临Broker升级时的消息丢失风险。
为避免该风险，保障服务高可用，必须将Topic的副本因子调整为2。下文将详细阐述这一升级操作的实施步骤。

0x01、创建两个单rf的topic

kafka-topics.sh  --create --bootstrap-server ${BROKER} --topic test-2 --replication-factor 1 --partitions 2
kafka-topics.sh  --create --bootstrap-server ${BROKER} --topic test-3 --replication-factor 1 --partitions 2

0x02、创建参数 topics.json

  {
     "topics": [
       { "topic": "test-2" },
       { "topic": "test-3" }
     ],
     "version": 1
   }

0x03、生成建议内容

kafka-reassign-partitions.sh --bootstrap-server ${BROKER} --topics-to-move-json-file topics.json --broker-list "1,2" --generate

执行结果：

Current partition replica assignment
{"version":1,"partitions":[{"topic":"test-2","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"test-2","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test-2","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"test-2","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":1,"replicas":[2],"log_dirs":["any"]}]}

0x04、保存并更新proposed json

cat reassignment.json

{"version":1,"partitions":[{"topic":"test-2","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"test-2","partition":1,"replicas":[1,2],"log_dirs":["any","any"]},
{"topic":"test-3","partition":0,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"test-3","partition":1,"replicas":[1,2],"log_dirs":["any","any"]}]}

留意：每一个partition 都需要更新replicas、log_dirs 配置

0x05、提交更新

kafka-reassign-partitions.sh --bootstrap-server ${BROKER} --reassignment-json-file reassignment.json --execute

Current partition replica assignment

{"version":1,"partitions":[{"topic":"test-2","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"test-2","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"test-3","partition":1,"replicas":[2],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for test-2-0,test-2-1,test-3-0,test-3-1

0x06、确认topic RF配置更新进度

kafka-reassign-partitions.sh --bootstrap-server ${BROKER} --verify --reassignment-json-file reassignment.json