Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Offsets are committed after handler's failure

See original GitHub issue

When I configure KafkaFlow with typed handlers, and the handler fails with an exception, the exception message is printed in logs, but the offsets are still committed.

Is it a bug? Or what would be the way of making the consumer to stop and do not proceed further?

Issue Analytics

State:
Created a year ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

AlexeyRagacommented, Jul 6, 2022

Having retries is nice, but there are situations where messages cannot be handled because there is simply a bug in the handler, or if there is a long term outage / breaking API in 3rd party deps, etc. Things like “it won’t be fixed until you deploy a newer version”…

I think that I just expected the default behaviour to be the safest one (stop the world, crash, don’t commit offsets) and everything else would be opt-in. Which is not the case, as I now see.

I’ll have a look at how to write a middleware and what can be done in it, thanks for the advice!

0reactions

AlexeyRagacommented, Jul 7, 2022

@filipeesch thanks for all your help! Will investigate this option.

Top Results From Across the Web

Commiting offset after consumer error handling in spring- ...

The failed record can be disposed of via a RecoveryCallback after retries have been exhausted and its offset committed then; the container ...

Rebalancing during offset commit results in event ...

A normal rebalance does not cause an error when committing offsets; the rebalance is suspended until the next poll; you should only get...

Kafka Consumer

Offset commit failures are merely annoying if the following commits succeed since they won't actually result in duplicate reads. However, if the last...

Transaction not getting committed when error handler is ...

If the transaction rolls back the offset is not committed and, by default, seek operations are performed on the consumer so the failed...

DefaultErrorHandler (Spring for Apache Kafka 3.0.10 API)

An error handler that, for record listeners, seeks to the current offset for each topic in the remaining records. Used to rewind partitions...