[Hono-MQTT-Adapter] no credit available for sending request

Hello,

Version Hono: 1.10.0 Kubernetes Version: v1.15.6

Description We deploy 1.10.0 version of Hono on Kubernetes cluster.

$ kubectl get pods
NAME                                                READY   STATUS    RESTARTS   AGE
hono-test-adapter-mqtt-vertx-6fb876676-g9wqx        1/1     Running   0          55m
hono-test-service-auth-9769ffc68-8zv5r              1/1     Running   0          3h57m
hono-test-service-command-router-6f48c7c55d-8clg8   1/1     Running   0          3h57m
hono-test-service-device-registry-6f494b675-88hrc   1/1     Running   0          96m

we use enmasse as message broker, below is enmasse’s address:

$ kubectl get address -n enmasse-test
NAME                                  ADDRESS                     READY   PHASE    AGE
hono-test.command-internal            command_internal            true    Active   11d
hono-test.command-response.loadtest   command_response/loadtest   true    Active   11d
hono-test.command-router.loadtest     cmd_router/loadtest         true    Active   3h49m
hono-test.command.loadtest            command/loadtest            true    Active   2d2h
hono-test.event.loadtest              event/loadtest              true    Active   11d
hono-test.telemetry.loadtest          telemetry/loadtest          true    Active   11d

Then we want to do loadtest against Hono via MQTT protocol(gateway mode). So we use mqtt-jmeter to do some test.

1. At first I register 1 gateway device and corresponding credentials.
2. Then I register 1000 edge device.
3. Configure Jmeter script to start test

Gateway Device:

$ curl https://******/v1/devices/loadtest/com.test:gateway001
{
    "enabled": true,
    "status": {
        "updated": "2021-10-09T17:17:21Z"
    }
}

One Edge Device:

$ curl https://******/v1/devices/loadtest/com.test:device_1
{
    "enabled": true,
    "via": [
        "com.test:gateway001"
    ],
    "status": {
        "created": "2021-10-11T07:33:44Z"
    }
}

But If the mqtt client (the number of jmeter threads) exceeds a certain number (perhaps 200), the MQTT adapter will report the following error:

10:21:37.690 [vert.x-eventloop-thread-1] DEBUG o.e.h.c.amqp.RequestResponseClient - cannot send request to peer, no credit left for link [link target: credentials/loadtest]
10:21:37.690 [vert.x-eventloop-thread-1] DEBUG o.e.h.a.m.i.VertxBasedMqttProtocolAdapter - device authentication or early stage checks failed
org.eclipse.hono.client.ServerErrorException: no credit available for sending request
        at org.eclipse.hono.client.amqp.RequestResponseClient.lambda$sendRequest$8(RequestResponseClient.java:623)
        at org.eclipse.hono.util.HonoProtonHelper.executeOnContext(HonoProtonHelper.java:230)
        at org.eclipse.hono.client.impl.HonoConnectionImpl.executeOnContext(HonoConnectionImpl.java:235)
        at org.eclipse.hono.client.amqp.RequestResponseClient.sendRequest(RequestResponseClient.java:618)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:581)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:530)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:484)
        at org.eclipse.hono.client.registry.amqp.ProtonBasedCredentialsClient.lambda$get$3(ProtonBasedCredentialsClient.java:191)
        at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38)
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
        at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62)
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
        at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49)
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:41)
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23)
        at org.eclipse.hono.client.impl.CachingClientFactory.getOrCreateClient(CachingClientFactory.java:163)
        at org.eclipse.hono.client.impl.CachingClientFactory.lambda$getOrCreateClient$3(CachingClientFactory.java:211)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:952)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:919)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
        at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:942)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
10:21:37.690 [vert.x-eventloop-thread-1] DEBUG o.e.h.a.m.i.VertxBasedMqttProtocolAdapter - rejecting connection request from client [clientId: device_415], cause:
org.eclipse.hono.client.ServerErrorException: no credit available for sending request
        at org.eclipse.hono.client.amqp.RequestResponseClient.lambda$sendRequest$8(RequestResponseClient.java:623)
        at org.eclipse.hono.util.HonoProtonHelper.executeOnContext(HonoProtonHelper.java:230)
        at org.eclipse.hono.client.impl.HonoConnectionImpl.executeOnContext(HonoConnectionImpl.java:235)
        at org.eclipse.hono.client.amqp.RequestResponseClient.sendRequest(RequestResponseClient.java:618)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:581)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:530)
        at org.eclipse.hono.client.amqp.RequestResponseClient.createAndSendRequest(RequestResponseClient.java:484)
        at org.eclipse.hono.client.registry.amqp.ProtonBasedCredentialsClient.lambda$get$3(ProtonBasedCredentialsClient.java:191)
        at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38)
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
        at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62)
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
        at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49)
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:41)
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23)
        at org.eclipse.hono.client.impl.CachingClientFactory.getOrCreateClient(CachingClientFactory.java:163)
        at org.eclipse.hono.client.impl.CachingClientFactory.lambda$getOrCreateClient$3(CachingClientFactory.java:211)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:952)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:919)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
        at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22)
        at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:942)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
10:21:37.690 [vert.x-eventloop-thread-1] DEBUG o.e.h.c.impl.CachingClientFactory - reusing cached client [credentials-loadtest]
10:21:37.690 [vert.x-eventloop-thread-1] DEBUG o.e.h.c.amqp.RequestResponseClient - cannot send request to peer, no credit left for link [link target: credentials/loadtest]
10:21:37.691 [vert.x-eventloop-thread-1] DEBUG o.e.h.a.m.i.VertxBasedMqttProtocolAdapter - device authentication or early stage checks failed
org.eclipse.hono.client.ServerErrorException: no credit available for sending request

What did I do I insert below parameters into device-registry as environment variables:

  HONO_CREDENTIALS_SVC_RECEIVERLINKCREDIT: "100000"
  HONO_REGISTRY_SVC_RECEIVERLINKCREDIT: "100000"
  HONO_TENANT_SVC_RECEIVERLINKCREDIT: "100000"

But still doesn’t work.

Expected result How can we configure the number of credits for credentials client so that we can support more than 200 MQTT client in gateway mode?

[sophokles73]

@calohmn

The error is coming from the credentials API client in the MQTT adapter, therefore the corresponding configuration changes need to be done in the MQTT adapter config

FMPOV the number of credits flown from the server (Credentials service) to the client (MQTT adapter) needs to be configured on the server side, i.e. using the HONO_CREDENTIALS_SVC_RECEIVERLINKCREDIT environment variable as described by @gongzhao2 above.

@gongzhao2 can you describe, how you have set the environment variable in order for the device registry to pick it up during start up? Maybe the registry simply didn’t recognize your updated value.

BTW I think that setting this value to 100000 is not really feasible. Because the device registry will probably not have the resources to process them at once anyway. I would increase the value gradually to e.g. 1000 credits and see if this has an impact.

[gongzhao2]

Thanks for the help. We want to test whether Hono can support 10k/s messages (gateway mode).

Infra ENV: hono-mqtt-adapter: 2v4G, 1 pod hono-service-device-registry: 2v4G, 2 pod hono-service-auth: default, 1 pod hono-command-router: default, 1 pod Consumers: 5 consumers which is using hono-cli-1.10.0-exec.jar to simulated

Possible errors in MQTT-Adapter

• org.eclipse.hono.client.ServerErrorException: no credit available for sending request
• org.eclipse.hono.client.NoConsumerException: no credit available
• org.eclipse.hono.client.ServerErrorException: request timed out after 500ms

Current conclusion Hono can support 10k/s messages when the consumer’s capacity is sufficient, the main bottleneck is mqtt-adapter and device-registry, so we’d better give more resources to these 2 components.

Just for others’ reference.