Too many OFFSET_OUT_OF_RANGE errors

I have a 5-broker Kafka cluster with ~100 topics, and I’m using KafkaJS to subscribe to 2 topics (1 with 400 partitions and 1 with 10), and I’m always stuck at "The requested offset is not within the range of offsets maintained by the server" errors.

These errors occur in an endless loop and I’m not able to consume any messages. I’ve also tried resetting the consumer offsets between runs, and also the fromBeginning option.

I know this is very little information, but can you help me understand what circumstances can cause this error? I can provide more info about my setup and code.

[tulios]

@paambaati there are a couple of fixes in master for this issue, you can use master disabling the v0.11 APIs (because of the LZ4 problem):

new Kafka({ ... allowExperimentalV011: false })

The MaxListenersExceededWarning is a silly bug, it isn’t leaking any memory, it is just that node.js has this value hardcoded to 11 and you probably have a higher concurrency. I will create a new issue and fix this right away. It shouldn’t cause any issues besides the annoying warning.

If you are producing in parallel you can also consider using the producer.sendBatch:

await producer.sendBatch({
  topicMessages: <TopicMessages[]>,
  acks: <Number>,
  timeout: <Number>,
  compression: <CompressionTypes>,
})

Feel free to re-open this issue if you need. Thanks again for the report, this is great for new users.

[tulios]

For reference, here is the issue about MaxListenersExceededWarning #153