环境准备
Kafka环境搭建和生产者样例代码与《Kafka应用Demo:按主题订阅消费消息》相同。
消费者代码样例
public class KafkaConsumerService {private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConsumerService.class);private static final String NEO_TOPIC = "elon-topic";Properties properties = new Properties();private KafkaConsumer consumer = null;public KafkaConsumerService() {TopicPartition partition0 = new TopicPartition(NEO_TOPIC, 0);TopicPartition partition1 = new TopicPartition(NEO_TOPIC, 1);properties.put("bootstrap.servers","192.168.5.128:9092"); // 指定 Brokerproperties.put("group.id", "neo2"); // 指定消费组群 IDproperties.put("max.poll.records", "1");properties.put("enable.auto.commit", "false");properties.put("key.deserializer", StringDeserializer.class); // 将 key 的字节数组转成 Java 对象properties.put("value.deserializer", StringDeserializer.class); // 将 value 的字节数组转成 Java 对象consumer = new KafkaConsumer<String, String>(properties);List<TopicPartition> partitionList = new ArrayList<>();partitionList.add(partition1);partitionList.add(partition0);consumer.assign(partitionList);new Thread(this::receiveMessage).start();}public void receiveMessage() {try {while (true) {synchronized (this) {ConsumerRecords<String,String> records = consumer.poll(Duration.ofMillis(Long.MAX_VALUE));LOGGER.info("Fetch record num:{}", records.count());for (ConsumerRecord<String,String> record: records) {String info = String.format("[Topic: %s][Partition:%d][Offset:%d][Key:%s][Message:%s]",record.topic(), record.partition(), record.offset(), record.key(), record.value());LOGGER.info("Received:" + info);Thread.sleep(10000);}consumer.commitSync();}}} catch (Exception e){} finally {consumer.close();}}
}
样例代码中的consumer.assign(partitionList)绑定了主题下的0号分区和1号分区接收消息。指派分区的方式和按主题订阅的方式不能混用,也就是说一个消费者实例只能选择一种方式订阅。
分析
如果我们同时启动两个conumer实例,指派订阅相同主题和相同分区的消息。可以看到这两个实例收到了相同的消息,哪怕这两个消费者配置了相同的分组,这一点是与按主题订阅消息不同的。
根据官方指导文档的说法,如果使用assign绑定分区订阅消息,不同的消费者实例是相互独立的(编者注:相当于广播消息)。为了避免offset提交导致冲突,应该为不同消费者实例配置不同的分组。