Is this correct example for Kafka Transactions?

Description

How I use Kafka Transactions in my code:

using IProducer<string, string> producer = new ProducerBuilder<string, string>(_producerConfig).Build();
producer.InitTransactions(_defaultTransactionTimeout);
_consumerConfig.AutoOffsetReset = AutoOffsetReset.Earliest;
using var consumer = new ConsumerBuilder<string, string>(_consumerConfig).Build();
consumer.Subscribe(_consumeTopic);

while (!stoppingToken.IsCancellationRequested)
{
var consumeResult = consumer.Consume(stoppingToken);

producer.BeginTransaction();

producer.Produce(_produceTopic, new Message<string, string>   {...  });

producer.SendOffsetsToTransaction(new List<TopicPartitionOffset> { consumeResult.TopicPartitionOffset}, consumer.ConsumerGroupMetadata, _defaultTransactionTimeout);

producer.CommitTransaction(_defaultTransactionTimeout);
}

Is this correct way to use Kafka Transactions? It works most of the time but sometimes it fails with error below:

%1|1609846051.178|TXNERR|rdkafka#producer-2| [thrd:main]: Fatal transaction error: Failed to add offsets to transaction: Broker: Producer attempted an operation with an old epoch (INVALID_PRODUCER_EPOCH)
%3|1609846051.178|ERROR|rdkafka#producer-2| [thrd:main]: Fatal error: Broker: Producer attempted an operation with an old epoch: Failed to add offsets to transaction: Broker: Producer attempted an operation with an old epoch

How to reproduce

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [* ] Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with “debug” : “…” as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[ksdvishnukumarcse]

I got it resolved by increasing the producer TransactionTimout setting

[ksdvishnukumarcse]

@mhowlett and @koshdim ,

Even i too face the INVALID_PRODUCER_EPOCH error. Am running the my application in Windows 10 machine. I have a single Producer and TransactionId is Random Guid, enabled Idempotence as well. No other instance of the application or same TransactionId is used. When i try to Commit in case of successful message processing or Abort in case of failure while message processing i am getting the error.

I have disabled AutoCommit and AutoOffsetStore as well in the consumer settings

Below is the Debug log for the same

      producerClient Transaction commit started...
%7|1655388819.918|TXNAPI|rdkafka#producer-1| [thrd:app]: Transactional API called: commit_transaction (begin) (in txn state InTransaction, idemp state Assigned)
%7|1655388819.918|TXNSTATE|rdkafka#producer-1| [thrd:main]: Transaction state change InTransaction -> BeginCommit
%7|1655388819.921|TXNCOMMIT|rdkafka#producer-1| [thrd:app]: Flushing 0 outstanding message(s) prior to commit
%7|1655388819.928|TXNCOMMIT|rdkafka#producer-1| [thrd:app]: Transaction commit message flush complete
%7|1655388819.928|TXNAPI|rdkafka#producer-1| [thrd:app]: Transactional API called: commit_transaction (in txn state BeginCommit, idemp state Assigned)
%7|1655388819.929|TXNSTATE|rdkafka#producer-1| [thrd:main]: Transaction state change BeginCommit -> CommittingTransaction
%7|1655388819.930|ENDTXN|rdkafka#producer-1| [thrd:main]: EndTxn returned INVALID_PRODUCER_EPOCH in state CommittingTransaction (may_retry=true)
%1|1655388819.930|TXNERR|rdkafka#producer-1| [thrd:main]: Fatal transaction error: Failed to end transaction: Local: This instance has been fenced by a newer instance (_FENCED)
%0|1655388819.932|FATAL|rdkafka#producer-1| [thrd:main]: Fatal error: Local: This instance has been fenced by a newer instance: Failed to end transaction: Local: This instance has been fenced by a newer instance
%7|1655388819.932|TXNSTATE|rdkafka#producer-1| [thrd:main]: Transaction state change CommittingTransaction -> FatalError
info: MyLogger[0]
      Producer Error Handler: Failed to end transaction: Local: This instance has been fenced by a newer instance
info: MyLogger[0]
      producerClient Transaction abort started...
%7|1655388834.104|TXNAPI|rdkafka#producer-1| [thrd:app]: Transactional API called: abort_transaction (begin) (in txn state FatalError, idemp state Assigned)

Could please provide input for the same.