Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support committing offsets in transaction

See original GitHub issue

Overview

Kafka transactions provide support for the “consume-transform-produce loop” by allowing transactional producers to commit offsets for their recently consumed messages. This allows a client to ensure that offsets are committed for consumed messages only if the transaction is committed.

The spec recommends we enable support by adding a method “sendOffsets” (sendOffsetsToTransaction) to the transactional producer. This method will accept a data structure mapping topic-partitions to offsets, then perform the requests AddOffsetsToTxnRequest and subsequently TxnOffsetCommit.

Currently the consumer will automatically commit offsets when reading messages. We will need to update the API to allow users to disable this option, so that they may send the offsets themselves via the producer’s new sendOffsetsToTransaction method.

Would love to hear your thoughts on how best to update the consumer to accommodate this need @tulios @Nevon

Tasks

Add method sendOffsetsToTransaction
Update consumer to allow disabling auto committing
Keep offsetManager in sync with sent offsets

Issue Analytics

State:
Created 5 years ago
Comments:8 (7 by maintainers)

Top GitHub Comments

2reactions

tulioscommented, Dec 4, 2018

@ianwsperber that was also my proposal, go for it!

0reactions

Nevoncommented, Feb 8, 2019

Closing this as per #232

Top Results From Across the Web

Commit transactional offset across 2 different clusters

It is possible, you can "manually" send offsets to your own topic on the same cluster, where the produced message is sent.

How Kafka's Consumer Auto Commit Configuration Can Lead ...

To keep track of which messages have already been processed, your consumer needs to commit the offsets of the messages that were processed....

How to commit message offsets in Kafka for reliable data ...

I am using Kafka for this application. So, I want to know, how can I commit the offset of every transaction after consuming...

Kafka Producer transactions incrementing offsets twice for a ...

Consumer should commit offset only when transaction is complete. We are also facing the same issue, where transactions incrementing offsets twice for a...

Offset Management - Apache Software Foundation

This is meant as a way to attach arbitrary metadata that should be committed with this offset commit. It could be the name...