Rebalance doesn't work reliably

I faced an issue with a rebalance on production: sometimes it just doesn’t happen. Looks like old partition streams are not always closing after rebalance.

I managed to minimize this issue into the test case:

withKafka { (config, topic) =>
  val numPartitions = 3
  createCustomTopic(topic, partitions = numPartitions)
  val produced = (0 until 10000).map(n => s"key-$n" -> s"value->$n")
  publishToKafka(topic, produced)

  def run(instance: Int, allAssignments: SignallingRef[IO, Map[Int, Set[Int]]]): IO[Unit] = {
    consumerStream[IO]
      .using(consumerSettings[IO](config).withGroupId("test"))
      .evalTap(_.subscribeTo(topic))
      .flatMap(_.partitionsMapStream)
      .flatMap { assignment =>
        Stream.eval(allAssignments.update { current =>
          current.updated(instance, assignment.keySet.map(_.partition()))
        }) >> Stream.emits(assignment.map { case (_, partitionStream) =>
          partitionStream.evalMap(_ => IO.sleep(10.millis))
        }.toList).parJoinUnbounded
      }.compile.drain
  }

  def checkAssignments(allAssignments: SignallingRef[IO, Map[Int, Set[Int]]])(instances: Set[Int]) = {
    allAssignments.discrete.filter { state =>
      instances.forall { instance =>
        state.get(instance).exists(_.nonEmpty)
      } && state.values.toList.flatMap(_.toList).sorted == List(0, 1, 2)
    }.take(1).compile.drain
  }

  (for {
    allAssignments <- SignallingRef[IO, Map[Int, Set[Int]]](Map.empty)
    fiber0 <- run(0, allAssignments).start
    _ <- checkAssignments(allAssignments)(Set(0))
    fiber1 <- run(1, allAssignments).start
    _ <- checkAssignments(allAssignments)(Set(0, 1))
    fiber2 <- run(2, allAssignments).start
    _ <- checkAssignments(allAssignments)(Set(0, 1, 2))
    _ <- fiber2.cancel
    _ <- checkAssignments(allAssignments)(Set(0, 1))
    _ <- fiber1.cancel
    _ <- checkAssignments(allAssignments)(Set(0))
    _ <- fiber0.cancel
  } yield succeed).unsafeRunSync()
}

I started 3 instances one by one and after that shut down them one by one.

From logs, I’m 100% sure, that on the java side all works fine because the java kafka consumer receives correct rebalance events.

I will try to fix this issue because it’s a blocker for me now. Also, It could be a blocker for any users, who try to rely on a rebalance in fs2-kafka.

[LMnet]

@bplommer I think I found a solution. I will prepare it and update the pull request. Don’t spend your time on the investigation.

[bplommer]

Fixed by #533