Consider moving commit method from CommitableOffset (and some others) to KafkaConsumer

During the discussion of https://github.com/fd4s/fs2-kafka/pull/409 I suggested that commit method from the CommitableOffset could be moved to the KafkaConsumer.

CommittableOffset feels like pure data, but it contains def commit: F[Unit], which is not data, but a behavior. Maybe it would be better if CommittableOffset will not have the commit method? Maybe it would be more natural to have it on KafkaConsumer (like def commit(committableOffset: CommitableOffset): F[Unit])?

The presence of the commit method in the CommittableOffset adds some difficulties for creating instances like Eq for CommittableOffset.

All Committable... types affected by this problem too.

As a complement to the KafkaConsumer#commit method, we could implement commit on the CommittableOffset as an extension method. It will require KafkaConsumer as an input argument, but overall I think it will help to maintain the current API as much as possible.

[bplommer]

We could consider adding the new methods to KafkaConsumer before 2.0 and deprecating the ones on CommittableX.

[LMnet]

I’m trying to implement a version where CommittableOffsetBatch could contain offsets from the different consumers.

Overall all looks fine. CommittableOffset simplified to this:

sealed abstract class CommittableOffset[F[_]] {
  def topicPartition: TopicPartition
  def offsetAndMetadata: OffsetAndMetadata
  def committer: KafkaCommit[F]
}

And CommittableOffsetBatch to this:

sealed abstract class CommittableOffsetBatch[F[_]] {
  def offsets: Map[KafkaCommit[F], Map[TopicPartition, OffsetAndMetadata]]
}

But I have some difficulties with the commit method implementation. Currently, the commit method is constructing inside KafkaConsumerActor and added to the CommittableOffset as a field. In the new API CommittableOffset doesn’t have the commit method. Instead, it has committer with the dedicated commit method. The problem is that KafkaCommit (which is in the essence a KafkaConsumer) is not available inside KafkaConsumerActor. So, CommittableOffset can’t be constructed inside KafkaConsumerActor anymore. To fix this I see a few ways with different drawbacks:

1. Construct in the KafkaConsumerActor some new entity, which is like CommittableOffset, but without committer field. When these entities reach KafkaConsumer it enriched them and turned into CommittableOffset with the committer field. It’s doable, but some decent amount of code should be rewritten and every batch should be traversed to create a CommittableOffset. I’m not sure how bad is that in terms of performance.
2. Pass a link to the KafkaConsumer in KafkaConsumerActor. It could be done only with the message passing because KafkaConsumerActor is created earlier than the KafkaConsumer. It means that we will need some new actor message like SetKafkaConsumer which will set a mutable field inside KafkaConsumerActor. It’s doable too but feels a bit dirty because KafkaConsumerActor will need a mutable optional field which in practice will be always filled with the value.
3. Create a special Committer entity for the committer field in the CommittableOffset. This entity will be created before KafkaConsumerActor and KafkaConsumer inside the KafkaConsumer constructor. It could be passed to the KafkaConsumerActor without dirty message passing. And It could create a link between a KafkaConsumerActor and KafkaConsumer. With this option, we will have an extra Committer entity in the public API, which is not good. Also, its purpose will be exclusively technical, to overcome our internal difficulties with the sharing data between KafkaConsumer and KafkaConsumerActor.

@bplommer @vlovgr wdyt? Maybe you could offer some other options to solve this issue?