kafka consumer group 无法消费数据
打开debug日志查看具体原因(此处是logback配置)
<logger name="com.alibaba.otter" level="DEBUG"/>
<root level="DEBUG">
...
</root>
看到关键提示
2020-05-08 10:11:15.596 DEBUG --- [Thread-3] org.apache.kafka.clients.consumer.internals.AbstractCoordinator [onSuccess] [741] : [Consumer clientId=consumer--4, groupId=group_async] Received FindCoordinator response ClientResponse(receivedTimeMs=1588903875595, latencyMs=4, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=3, clientId=consumer-group_async-4, correlationId=11), responseBody=FindCoordinatorResponseData(throttleTimeMs=0, errorCode=15, errorMessage='The coordinator is not available.', nodeId=-1, host='', port=-1))
根据这个提示得知,是消费组协调leader不存在导致
查看kafka __consumer_offset topic
发现确实有Leader:none的
可能原因是 __consumer_offset topic的默认分区是50，但是备份只有1份; kafka集群部署了3台，也就是3个brokers
消费者组连接kafka,并会请求某一台来查找Coordinator(协调者)，如果连接的机器上没有备份就不会有Leader,就会出现
找不到Coordinator(协调者)
解决方式
1、停止kafka, 修改kafka配置 config/server.properties添加
修改成broker的数量
offsets.topic.replication.factor=3
2、删除zookeeper配置信息
./bin/zkCli.sh -server 127.0.0.1:2181
删除/config/topics/__consumer_offsets
delete /config/topics/__consumer_offsets
删除/brokers/topics/__consumer_offsets
deleteall /brokers/topics/__consumer_offsets
3、重启kafka
参考
https://stackoverflow.com/questions/51952398/kafka-connect-distributed-mode-the-group-coordinator-is-not-available
https://www.cnblogs.com/jun1019/p/6634545.html
https://blog.csdn.net/yf_li123/article/details/101552437
https://www.orchome.com/1296
https://github.com/edenhill/librdkafka/issues/2086
