question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Commiting offset.

See original GitHub issue

Hello, i want to commit an offset for a consumer, but i’m getting an error: KafkaJSNonRetriableError: Must provide transactional id for transactional producer

Also i would like there to be a method for consumer, smth like commit(offset: number)

My code looks smth like:

class MyController {
  async created() {
    this.kafka = new Kafka({
      clientId: 'my-app',
      brokers: ['localhost:9092']
    });
    this.consumer = this.kafka.consumer({ groupId: 'my-group'});
    this.producer = this.kafka.producer({
      idempotent: true,
      retry: {
        maxInFlightRequests: 1,
      }
    });
    await this.consumer.connect();
    await this.producer.connect();
    await this.consumer.subscribe({
      topic: 'my-topic',
      fromBeginning: true,
    });
    await this.consumer.run({
      eachMessage: async ({ topic, partition, message }) => {
        console.log({
          partition,
          offset: message.offset,
          value: message.value.toString(),
        });

        let transaction: Transaction | undefined;

        try {
          transaction = await this.producer.transaction();
          await transaction.sendOffsets({
            consumerGroupId: 'my-group',
            topics: [{
              topic: 'my-topic',
              partitions: [{
                partition: 0,
                offset: message.offset.toString(),
              }]
            }],
          });
          await transaction.commit();
        } catch (e) {
          if (!transaction) {
            console.log(e);
          } else {
            await transaction.abort();
          }
        }
      },
      autoCommit: false,
    });
  }
}

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:11 (2 by maintainers)

github_iconTop GitHub Comments

8reactions
JaapRoodcommented, Jun 13, 2019

Reviewing the documentation, there is actually nothing to explain the above! I might have some time this week to fix that.

3reactions
AlexKonstantinov1991commented, Jul 19, 2019

Reviewing the documentation, there is actually nothing to explain the above! I might have some time this week to fix that.

Hello!

There is still no information in documentation about that. Just to remind.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Kafka - Why do we need offset commits? - LogicBig
Committing an offset for a partition is the action of saying that the offset has been processed so that Kafka cluster won't send...
Read more >
Kafka - When to commit? - Quarkus
This strategy commits the offset every time a message is acknowledged. This strategy tends to commit often, and so decrease the throughput.
Read more >
Kafka commit offset - Stack Overflow
It depends on how you commit: On commitSync assuming you commit each offset separately like in your example, if offset 1 is not...
Read more >
Kafka Consumer | Confluent Documentation
The offset commit policy is crucial to providing the message delivery guarantees needed by your application. By default, the consumer is configured to...
Read more >
apache-kafka Tutorial => How to Commit Offsets
A committed offset indicates, that all messages up to this offset got already processed. Thus, as offsets are consecutive numbers, committing offset X ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found