Commiting offset.
See original GitHub issueHello, i want to commit an offset for a consumer, but i’m getting an error:
KafkaJSNonRetriableError: Must provide transactional id for transactional producer
Also i would like there to be a method for consumer, smth like commit(offset: number)
My code looks smth like:
class MyController {
async created() {
this.kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:9092']
});
this.consumer = this.kafka.consumer({ groupId: 'my-group'});
this.producer = this.kafka.producer({
idempotent: true,
retry: {
maxInFlightRequests: 1,
}
});
await this.consumer.connect();
await this.producer.connect();
await this.consumer.subscribe({
topic: 'my-topic',
fromBeginning: true,
});
await this.consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log({
partition,
offset: message.offset,
value: message.value.toString(),
});
let transaction: Transaction | undefined;
try {
transaction = await this.producer.transaction();
await transaction.sendOffsets({
consumerGroupId: 'my-group',
topics: [{
topic: 'my-topic',
partitions: [{
partition: 0,
offset: message.offset.toString(),
}]
}],
});
await transaction.commit();
} catch (e) {
if (!transaction) {
console.log(e);
} else {
await transaction.abort();
}
}
},
autoCommit: false,
});
}
}
Issue Analytics
- State:
- Created 4 years ago
- Comments:11 (2 by maintainers)
Top Results From Across the Web
Kafka - Why do we need offset commits? - LogicBig
Committing an offset for a partition is the action of saying that the offset has been processed so that Kafka cluster won't send...
Read more >Kafka - When to commit? - Quarkus
This strategy commits the offset every time a message is acknowledged. This strategy tends to commit often, and so decrease the throughput.
Read more >Kafka commit offset - Stack Overflow
It depends on how you commit: On commitSync assuming you commit each offset separately like in your example, if offset 1 is not...
Read more >Kafka Consumer | Confluent Documentation
The offset commit policy is crucial to providing the message delivery guarantees needed by your application. By default, the consumer is configured to...
Read more >apache-kafka Tutorial => How to Commit Offsets
A committed offset indicates, that all messages up to this offset got already processed. Thus, as offsets are consecutive numbers, committing offset X ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Reviewing the documentation, there is actually nothing to explain the above! I might have some time this week to fix that.
Hello!
There is still no information in documentation about that. Just to remind.