Problem with quarkus kafka smallrye consumer multithreading

Describe the bug

By setting “partitions” parameter to 3. I have 3 different consumer ids, but the messages are processed sequentially.

• quarkus version 2.X
• smallrye

Properties that i set

mp:
  messaging:
    incoming:
      event:
        connector: smallrye-kafka
        auto:
          offset:
            reset: earliest
        topic: MY_TOPIC
        group:
          id: my-group
        partitions: 3

Expected behavior

the messages should be processed concurrently for each partition

Actual behavior

the messages are processed sequentialy though we have different partitions

Output of java -version

jdk11

Quarkus version or git rev

2.4.1

Build tool (ie. output of mvnw --version or gradlew --version)

3.6.3

[cescoffier]

Another solution is something like:

List<Executor> executors = ...

@Incoming("kafka")
@Outgoing("foo")
public Multi<Message<T>> dispatch(Multi<Message<T>> multi) {
   return multi.onItem().transformToUniAndMerge(msg -> {
       IncomingKafkaMetadata metadata = msg.getMetadata(IncomingKafkaMetadata.class).orElseThrow();
       int p = metadata.getPartition();
       Executor executor = executors.get(p);
       return Uni.createFrom().item(msg).emitOn(executor);
  }); 
}

Note that even with this, the emission will still be serialized. They would be multi-threaded, but serialized. You may want to do something like:

@Incoming("kafka")
@Outgoing("foo")
public Multi<Message<T>> dispatch(Multi<Message<T>> multi) {
   return multi.onItem().transformToUniAndMerge(msg -> {
       IncomingKafkaMetadata metadata = msg.getMetadata(IncomingKafkaMetadata.class).orElseThrow();
       int p = metadata.getPartition();
       Executor executor = executors.get(p);
       return Uni.createFrom().item(msg).emitOn(executor)
           .onItem().invoke(msg -> {
                    process(msg); msg.ack(); })
  }); 
}

so, the processing is running on the different threads.

You can try with the low-level kafka client, but it’s a rabbit hole.

[cescoffier]

No, the documentation is right, it just does not do the dispatching as you describe it, and will still serialize the records following the reactive streams semantic.