Qpid gets into state where only 1 credit is returned to adapter each time

The Hono demo setup from the getting started guide can be brought into a state where the HTTP adapter only accepts 1 request at a time. Multiple parallel requests produce errors, caused by “no credits available” exceptions in the HTTP adapter.

To reproduce:

• Start the Hono demo setup from the getting started guide
• Start a consumer (hono-cli)
• Send some messages in parallel to the HTTP adapter -> all should be accepted
• Stop the consumer (Ctrl+C)
• Send one message -> This message is still accepted by the HTTP Adapter but rejected from Qpid
• Start the consumer again
• Send some messages in parallel to the HTTP adapter -> only one at a time will be accepted, because the adapter only ever gets 1 credit at a time from Qpid

Further analysis: When stopping the consumer, only a close frame is sent by the hono-cli receiver to Qpid:

16:08:24.473 [vert.x-eventloop-thread-0] INFO  o.e.hono.client.impl.HonoClientImpl - closing connection to container [Hono.Example.Router] at [localhost:15671] ...
[1724021650:0] -> Close{error=null}
[1724021650:0] <- Close{error=null}
16:08:24.476 [vert.x-eventloop-thread-0] INFO  o.e.hono.client.impl.HonoClientImpl - closed connection to container [Hono.Example.Router] at [localhost:15671]

The HTTP adapter doesn’t get informed about the now missing consumer. That means it still has credits. (I don’t know whether it would be different if the hono-cli-consumer would explicitly send a detach frame to close the AMQP link upon exiting the consumer.)

When sending the next message (that will get rejected by Qpid because there’s no consumer), the HTTP adapter log output looks like this:

16:05:51.758 [vert.x-eventloop-thread-0] DEBUG o.e.hono.client.impl.HonoClientImpl - reusing existing message sender [target: telemetry/DEFAULT_TENANT, credit: 250]
16:05:51.758 [vert.x-eventloop-thread-0] DEBUG o.e.hono.client.impl.HonoClientImpl - reusing existing client [target: registration/DEFAULT_TENANT]
16:05:51.759 [vert.x-eventloop-thread-0] TRACE o.e.h.s.c.SpringBasedExpiringValueCache - cache hit [key: TriTuple[one: assert, two: 4711, three: null]]
16:05:51.759 [vert.x-eventloop-thread-0] DEBUG o.e.hono.client.impl.HonoClientImpl - reusing existing client [target: tenant]
16:05:51.759 [vert.x-eventloop-thread-0] TRACE o.e.h.s.c.SpringBasedExpiringValueCache - cache hit [key: TriTuple[one: get, two: DEFAULT_TENANT, three: null]]
[301242748:0] -> Transfer{handle=0, deliveryId=3, deliveryTag=\x00\x00\x00\x04, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false} (275) "\x00Sr\xc1V\x06\xa3\x09tenant_id\xa1\x0eDEFAULT_TENANT\xa3\x09device_id\xa1\x044711\xa3\x08resource\xa1\x1dtelemetry/DEFAULT_TENANT/4711\x00Ss\xd0\x00\x00\x00V\x00\x00\x00\x0a\xa1\x15TelemetrySenderImpl-3@\xa1\x18telemetry/DEFAULT_TENANT@@@\xa3\x10application/json@@\x83\x00\x00\x01h\xbeh\x96\x10\x00St\xc1E\x06\xa1\x0corig_adapter\xa1\x09hono-http\xa1\x09device_id\xa1\x044711\xa1\x0corig_address\xa1\x0a/telemetry\x00Su\xa0\x0b{"temp": 5}"
16:05:51.763 [vert.x-eventloop-thread-0] TRACE o.e.h.c.impl.TelemetrySenderImpl - sent message [ID: TelemetrySenderImpl-3], remaining credit: 249, queued messages: 0
16:05:51.764 [vert.x-eventloop-thread-0] TRACE o.e.h.a.h.v.VertxBasedHttpProtocolAdapter - successfully processed [TELEMETRY] message for device [tenantId: DEFAULT_TENANT, deviceId: 4711]
[301242748:0] <- Flow{nextIncomingId=5, incomingWindow=400, nextOutgoingId=0, outgoingWindow=2147483647, handle=0, deliveryCount=4, linkCredit=249, available=null, drain=true, echo=false, properties=null}
[301242748:0] <- Disposition{role=RECEIVER, first=3, last=null, settled=true, state=Released{}, batchable=false}
[301242748:0] -> Flow{nextIncomingId=0, incomingWindow=65535, nextOutgoingId=5, outgoingWindow=2147483647, handle=0, deliveryCount=253, linkCredit=0, available=null, drain=true, echo=false, properties=null}
16:05:51.775 [vert.x-eventloop-thread-0] DEBUG o.e.h.c.impl.TelemetrySenderImpl - message [message ID: TelemetrySenderImpl-3] not accepted by peer, remote state: Released

Note that Qpid has sent a flow frame with link-credit=249, drain=true, also visible in the Qpid log:

[0x7fdb040011b0]:0 <- @transfer(20) [handle=0, delivery-id=3, delivery-tag=b"\x00\x00\x00\x04", message-format=0, settled=false] (275) "\x00Sr\xc1V\x06\xa3\x09tenant_id\xa1\x0eDEFAULT_TENANT\xa3\x09device_id\xa1\x044711\xa3\x08resource\xa1\x1dtelemetry/DEFAULT_TENANT/4711\x00Ss\xd0\x00\x00\x00V\x00\x00\x00\x0a\xa1\x15TelemetrySenderImpl-3@\xa1\x18telemetry/DEFAULT_TENANT@@@\xa3\x10application/json@@\x83\x00\x00\x01h\xbeh\x96\x10\x00St\xc1E\x06\xa1\x0corig_adapter\xa1\x09hono-http\xa1\x09device_id\xa1\x044711\xa1\x0corig_address\xa1\x0a/telemetry\x00Su\xa0\x0b{"temp": 5}"
[0x7fdb040011b0]:0 -> @flow(19) [next-incoming-id=5, incoming-window=400, next-outgoing-id=0, outgoing-window=2147483647, handle=0, delivery-count=4, link-credit=249, drain=true]
[0x7fdb040011b0]:0 -> @disposition(21) [role=true, first=3, settled=true, state=@released(38) []]
[0x7fdb040011b0]:0 <- @flow(19) [next-incoming-id=0, incoming-window=65535, next-outgoing-id=5, outgoing-window=2147483647, handle=0, delivery-count=253, link-credit=0, drain=true]

After restarting the consumer, Qpid only ever sends one credit to the HTTP adapter. HTTP-Adapter log:

16:08:05.904 [vert.x-eventloop-thread-0] DEBUG o.e.hono.client.impl.HonoClientImpl - reusing existing message sender [target: telemetry/DEFAULT_TENANT, credit: 1]
...
16:08:05.934 [vert.x-eventloop-thread-0] TRACE o.e.h.c.impl.TelemetrySenderImpl - sent message [ID: TelemetrySenderImpl-4], remaining credit: 0, queued messages: 0
16:08:05.938 [vert.x-eventloop-thread-0] TRACE o.e.h.a.h.v.VertxBasedHttpProtocolAdapter - successfully processed [TELEMETRY] message for device [tenantId: DEFAULT_TENANT, deviceId: 4711]
[2120579023:0] -> Disposition{role=RECEIVER, first=1, last=1, settled=true, state=Accepted{}, batchable=false}
16:08:05.940 [vert.x-eventloop-thread-0] TRACE o.e.h.client.impl.AbstractHonoClient - handling message [remotely settled: false, queued messages: 0, remaining credit: 199]
[2120579023:0] -> Flow{nextIncomingId=3, incomingWindow=65535, nextOutgoingId=3, outgoingWindow=2147483647, handle=0, deliveryCount=2, linkCredit=200, available=null, drain=false, echo=false, properties=null}
[2120579023:0] <- Flow{nextIncomingId=3, incomingWindow=131071, nextOutgoingId=3, outgoingWindow=2147483647, handle=1, deliveryCount=2, linkCredit=100, available=null, drain=false, echo=false, properties=null}
[301242748:0] <- Flow{nextIncomingId=6, incomingWindow=400, nextOutgoingId=0, outgoingWindow=2147483647, handle=0, deliveryCount=254, linkCredit=1, available=null, drain=false, echo=false, properties=null}
[301242748:0] <- Disposition{role=RECEIVER, first=4, last=null, settled=true, state=Accepted{}, batchable=false}
16:08:05.967 [vert.x-eventloop-thread-0] TRACE o.e.h.c.impl.TelemetrySenderImpl - message [message ID: TelemetrySenderImpl-4] accepted by peer

Qpid log:

[0x7fdb040011b0]:0 <- @transfer(20) [handle=0, delivery-id=4, delivery-tag=b"\x00\x00\x00\x05", message-format=0, settled=false] (275) "\x00Sr\xc1V\x06\xa3\x09tenant_id\xa1\x0eDEFAULT_TENANT\xa3\x09device_id\xa1\x044711\xa3\x08resource\xa1\x1dtelemetry/DEFAULT_TENANT/4711\x00Ss\xd0\x00\x00\x00V\x00\x00\x00\x0a\xa1\x15TelemetrySenderImpl-4@\xa1\x18telemetry/DEFAULT_TENANT@@@\xa3\x10application/json@@\x83\x00\x00\x01h\xbej\xa2+\x00St\xc1E\x06\xa1\x0corig_adapter\xa1\x09hono-http\xa1\x09device_id\xa1\x044711\xa1\x0corig_address\xa1\x0a/telemetry\x00Su\xa0\x0b{"temp": 5}"
[0xe4fe70]:0 -> @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"\x03\x00\x00\x00\x00\x00\x00\x00", message-format=0] (281) "\x00Sr\xd1\x00\x00\x00Y\x00\x00\x00\x06\xa3\x09tenant_id\xa1\x0eDEFAULT_TENANT\xa3\x09device_id\xa1\x044711\xa3\x08resource\xa1\x1dtelemetry/DEFAULT_TENANT/4711\x00Ss\xd0\x00\x00\x00V\x00\x00\x00\x0a\xa1\x15TelemetrySenderImpl-4@\xa1\x18telemetry/DEFAULT_TENANT@@@\xa3\x10application/json@@\x83\x00\x00\x01h\xbej\xa2+\x00St\xc1E\x06\xa1\x0corig_adapter\xa1\x09hono-http\xa1\x09device_id\xa1\x044711\xa1\x0corig_address\xa1\x0a/telemetry\x00Su\xa0\x0b{"temp": 5}"
[0xe4fe70]:0 <- @disposition(21) [role=true, first=0, last=0, settled=true, state=@accepted(36) []]
[0xe4fe70]:0 <- @flow(19) [next-incoming-id=1, incoming-window=65535, next-outgoing-id=1, outgoing-window=2147483647, handle=0, delivery-count=1, link-credit=200]
[0x7fdb040011b0]:0 -> @flow(19) [next-incoming-id=6, incoming-window=400, next-outgoing-id=0, outgoing-window=2147483647, handle=0, delivery-count=254, link-credit=1, drain=false]
[0x7fdb040011b0]:0 -> @disposition(21) [role=true, first=4, settled=true, state=@accepted(36) []]

For comparison: Before stopping the consumer, Qpid has always sent 250 credits:

[0x7fdb040011b0]:0 -> @flow(19) [next-incoming-id=2, incoming-window=400, next-outgoing-id=0, outgoing-window=2147483647, handle=0, delivery-count=1, link-credit=250, drain=false]
[0x7fdb040011b0]:0 -> @disposition(21) [role=true, first=0, settled=true, state=@accepted(36) []]

[calohmn]

I would opt for closing this issue and adding a new one with the task of updating Qpid to the version containing a fix.

[calohmn]

Closing this issue then - followup is #1087.