kafka消费相同消费组问题

报错如下

failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records

排查

1、有消费者宕机下线。消费者并不一定需要真正下线,例如遇到长时间的GC、网络延迟导致消费者长时间未向 GroupCoordinator 发送心跳等情况时, GroupCoordinator 会认为消费者已经下线。
修改参数

props.put("max.poll.interval.ms",50**60*1000);
props.put("max.poll.records",500);  
//设置消费者心跳间隔
props.put("heartbeat.interval.ms",3000);
//session超时时间
props.put("session.timeout.ms", 10000);

2、kafkaConsumer.assign() 点对点消费方式和 subscribe()订阅消费方式 ,使用了相同的消费组,也就是他们 group id 相同时, group coordinator无法识别具有相同消费组group id的consumer,直接抛异常CommitFailedException
如果是这种情况,提示的解决方法max.poll.interval.ms,max.poll.record都没用,无法解决,只能修改消费组id

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
平台声明:文章内容(如有图片或视频亦包括在内)由作者上传并发布,文章内容仅代表作者本人观点,简书系信息发布平台,仅提供信息存储服务。

推荐阅读更多精彩内容