Kafka consumer property 'resetOffsets' has no effect

While creating a consumer (and setting group id), the property spring.cloud.stream.kafka.bindings.<channelName>.consumer.resetOffsets (as described here: http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.RELEASE/reference/htmlsingle/#_kafka_consumer_properties) seems to have no effect, and consumer consumes from last offset. Sample properties:

spring:
  cloud:
    stream:
      bindings:
        input:
          destination: topic
          group: sample_group
      kafka:
        bindings:
          input:
            consumer:
              resetOffsets: true
              startOffset: earliest

The KafkaConsumerProperties#resetOffsets extension is set to true, but it doesn’t seem to trigger any specific resetting of offsets.

[mbogoevici]

Hi @cmuelhau,

We have another open issue for this: #63 . This is an feature loss that has been introduced by the adoption of the new Kafka client with rebalancing support (we should update documentation to reflect that). Implementation-wise this is a problem when auto-rebalance with multiple instances comes into play, as multiple arriving instances may reset multiple times without the proper coordination. We are currently deciding whether to officially drop this feature or support it only when auto-rebalancing is disabled (partition allocation is static based on index/instanceCount like in 1.0).

In the mean time, a solution for this would be to use a new group name when rebalance is required, which would effectively force the consumer to reset.

Really looking for the best suggestion here, as we’d really want to be able to support this, but we’re also aware that we should do so in a manner that works properly.

Thank you for the interest!

[mbogoevici]

No problem. There is another specific use case where we could in principle support this, using named groups: rebalancing can be suppressed in the clients using autoRebalanceEnable=false - see http://docs.spring.io/autorepo/docs/spring-cloud-stream-docs/Brooklyn.BUILD-SNAPSHOT/reference/htmlsingle/#_kafka_consumer_properties. In those cases the behaviour on start (or restart) is deterministic enough so that we can consider supporting resetting offsets. We’ve thought about this for a while - happy to get some feedback here.