Fault tolerance annotations not playing well with ThreadLocalPool

Describe the bug If you annotate a class with org.eclipse.microprofile.faulttolerance.Asynchronous then lots of threads are created and the ThreadlocalPool can result on max number of connections restrictions in the database being reached.

One could try to return a Future, but then there are other complains from the internal validations. It is also funny that it requires exactly a Future and doesn’t take a CompletableFuture, shouldn’t it work with any future?.

That annotation is required for org.eclipse.microprofile.faulttolerance.Timeout to work (which is also inconvenient because it requires you to return a CompletionStage, when perhaps it could also handle Uni/Multi ?).

Example:

  @Incoming("Something")
  @Counted(name = "received", description = "How many messages were received.")
  @Timeout(500)
  @Retry(maxDuration = 2000, maxRetries = 6, retryOn = {TimeoutException.class})
  @Asynchronous
  public CompletionStage<SomeResult> process(
      Message<SomeInput> record) {

I know one could alternatively use reactive handlers to deal with the timeout and the retries, but we started this way. If this is not the right way, could some documentation be added with the correct example?

Expected behavior That the thread context would somehow be reused so all threads spawn by the Asynchronous use the same connection pool instead of creating one for each one of them.

Actual behavior Each call to the method is executed in a different thread (as expected), creating multiple database connection pools and potentially reaching max limit in certain database (postgres in my case).

To Reproduce

On version 1.11.x or 1.12.x create a kafka listener with:

  @Incoming("EnvironmentsWithGeo")
  @Counted(name = "received", description = "How many environments were received.")
  @IncomingLogged
  @Timeout(500)
  @Retry(maxDuration = 2000, maxRetries = 6})
  @Asynchronous
  public CompletionStage<Void> process(
      Message<SomeInput> record) {
    someReactiveDao.doSomething(record.getPayload()).toCompletableFuture();
  }

As soon as you put load on it, lots of threads are started and it reaches the max number of db connections on postgres.

Configuration

# Add your application.properties here, if applicable.
quarkus:
  datasource:
    db-kind: postgresql
    metrics:
      enabled: true
    username: <redacted>
    password: <redacted>
    jdbc:
      url: jdbc:postgresql://localhost:5432/<redacted>
      max-size: 1
    reactive:
      url: postgresql://localhost:5432/<redacted>
      max-size: 20
kafka:
  bootstrap:
    servers: localhost:9092

mp:
  messaging:
    incoming:
      <redacted>:
        enable:
          auto:
            commit: true
        connector: smallrye-kafka
        topic: <redacted>
        pattern: true
        failure-strategy: ignore
        group:
          id: <redacted>
        value:
          deserializer: <redacted>

Screenshots (If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver:
• Output of java -version:
• GraalVM version (if different from Java):
• Quarkus version or git rev: 1.11.x or 1.12.x, the behaviour is not observed in 1.10.5.FINAL
• Build tool (ie. output of mvnw --version or gradlew --version): Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)

Additional context N/A

[gsmet]

It’s almost as if the connection pools were thread-local in the reactive datasources

I think that’s exactly how it works for the time being. /cc @Sanne

[Ladicek]

Yes, SmallRye Fault Tolerance no longer requires thread offloads to guard asynchronous actions. You can either use @NonBlocking, or use the non-compatible mode that only looks at the return type (https://smallrye.io/docs/smallrye-fault-tolerance/5.2.1/usage/extra.html#_non_compatible_mode).

As for why the actions are invoked on multiple different thread pools, I can’t tell. Using @Timeout may result in observing the timeout failure on a new thread, but that thread should still come from the main Quarkus worker pool.