Manual offset and partition assignment?

This might be a feature request, and it might not be possible.

My use case needs ‘simple consumer’ like behavior. I don’t want Kafka to manage consumer group rebalancing, and I won’t be doing any offset commits. I do want consumers to be able to provide topic, partition, and offset from which they will begin consuming, so that clients can manage their own offsets. When the protocol eventually supports consuming based on event timestamps, I’d like to support that as well.

Is this possible? I see KafkaConsumer.prototype.assign is a function, but I don’t see anything using it, and my attempts to do so leave me in Erroneous state from librdkafka. I think this is either because the offsets aren’t actually passed to librdkafka’s assign, or because of some internal subscribed state that is not being updated.

Is doing something like consumer.assign([ { topic: 'test', partition: 0, offset: 1102583 } ]) possible?

Also, if can I disable auto consumer group rebalancing? I’m looking for a way to override the rebalance_cb, but I don’t see that.

[webmakersteve]

Got too antsy 😛

Currently, the rebalance_cb is on by default and can’t be turned off. However, I’m in the process of writing it so you can opt-out.

However, opting out of the rebalance CB will probably just make librdkafka manage the state internally.

c.f. https://github.com/edenhill/librdkafka/blob/master/src-cpp/rdkafkacpp.h#L541

[ottomata]

It works! Thank you!

On Tue, Aug 23, 2016 at 1:43 PM, Stephen Parente [e-mail] wrote:

Can you try the code in this branch to see if it fixes your problem:

https://github.com/Blizzard/node-rdkafka/tree/assignment-fix

Should also now allow offsets to be passed into the object.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Blizzard/node-rdkafka/issues/18#issuecomment-241814465, or mute the thread https://github.com/notifications/unsubscribe-auth/ABjPxnTX8_JYpoFMd9z9XbeeDHYkOoU1ks5qizE-gaJpZM4JrH28 .