Received a `Maximum active streams violated for this endpoint.` from client side

armeria version: 1.8.0 verified client-side isn’t changing MAX_CONCURRENT_STREAMS

 java.util.concurrent.CompletionException: com.linecorp.armeria.client.UnprocessedRequestException: io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
	at com.linecorp.armeria.internal.common.HttpObjectAggregator.fail(HttpObjectAggregator.java:145)
	at com.linecorp.armeria.internal.common.HttpObjectAggregator.onError(HttpObjectAggregator.java:61)
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onError(DeferredStreamMessage.java:347)
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:276)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:284)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:276)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:448)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:391)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.doRequest(DefaultStreamMessage.java:259)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.request(DefaultStreamMessage.java:242)
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:218)
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onSubscribe(DeferredStreamMessage.java:335)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.subscribe(DefaultStreamMessage.java:147)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.subscribe(DefaultStreamMessage.java:130)
	at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:67)
	at com.linecorp.armeria.common.stream.DeferredStreamMessage.safeOnSubscribeToUpstream(DeferredStreamMessage.java:282)
	at com.linecorp.armeria.common.stream.DeferredStreamMessage.delegate(DeferredStreamMessage.java:121)
	at com.linecorp.armeria.common.DeferredHttpResponse.delegate(DeferredHttpResponse.java:48)
	at com.linecorp.armeria.common.DeferredHttpResponse.lambda$delegateWhenComplete$0(DeferredHttpResponse.java:59)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleBackoff$9(RetryingClient.java:398)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
	at java.base/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:946)
	at java.base/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2266)
	at java.base/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:143)
	at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponse$6(RetryingClient.java:352)
	at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
	at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
	at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
	at com.linecorp.armeria.client.retry.RetryingClient.handleResponse(RetryingClient.java:312)
	at com.linecorp.armeria.client.retry.RetryingClient.doExecute0(RetryingClient.java:297)
	at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleBackoff$8(RetryingClient.java:392)
	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:546)
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.linecorp.armeria.client.UnprocessedRequestException: io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
	at com.linecorp.armeria.client.UnprocessedRequestException.of(UnprocessedRequestException.java:45)
	at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:134)
	at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:54)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
	at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:205)
	at com.linecorp.armeria.client.HttpRequestSubscriber.onSubscribe(HttpRequestSubscriber.java:181)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$DownstreamSubscription.invokeOnSubscribe0(DefaultStreamMessageDuplicator.java:672)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$DownstreamSubscription.invokeOnSubscribe(DefaultStreamMessageDuplicator.java:663)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$StreamMessageProcessor.doSubscribe(DefaultStreamMessageDuplicator.java:292)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$StreamMessageProcessor.subscribe(DefaultStreamMessageDuplicator.java:272)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$ChildStreamMessage.subscribe0(DefaultStreamMessageDuplicator.java:537)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$ChildStreamMessage.subscribe(DefaultStreamMessageDuplicator.java:525)
	at com.linecorp.armeria.common.stream.DefaultStreamMessageDuplicator$ChildStreamMessage.subscribe(DefaultStreamMessageDuplicator.java:515)
	at com.linecorp.armeria.common.stream.StreamMessageWrapper.subscribe(StreamMessageWrapper.java:80)
	at com.linecorp.armeria.client.HttpSessionHandler.invoke(HttpSessionHandler.java:203)
	at com.linecorp.armeria.client.HttpClientDelegate.doExecute(HttpClientDelegate.java:276)
	at com.linecorp.armeria.client.HttpClientDelegate.acquireConnectionAndExecute(HttpClientDelegate.java:162)
	at com.linecorp.armeria.client.HttpClientDelegate.finishResolve(HttpClientDelegate.java:123)
	at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:106)
	at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:51)
	at com.linecorp.armeria.client.metric.AbstractMetricCollectingClient.execute(AbstractMetricCollectingClient.java:62)
	at com.linecorp.armeria.internal.client.ClientUtil.pushAndExecute(ClientUtil.java:150)
	at com.linecorp.armeria.internal.client.ClientUtil.executeWithFallback(ClientUtil.java:138)
	at com.linecorp.armeria.client.retry.RetryingClient.doExecute0(RetryingClient.java:286)
	... 11 common frames omitted
Caused by: io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
	at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:152)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:896)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders0(DefaultHttp2ConnectionEncoder.java:201)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:151)
	at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:45)
	at com.linecorp.armeria.client.ClientHttp2ObjectEncoder.doWriteHeaders(ClientHttp2ObjectEncoder.java:73)
	at com.linecorp.armeria.client.ClientHttpObjectEncoder.writeHeaders(ClientHttpObjectEncoder.java:41)
	at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:204)
	... 30 common frames omitted

[jrhee17]

Thanks for the detailed analysis!

I think I understand why the exception occurs for the first case consistently.

The callback in thenRun() is run from the eventloop where the response is closed. At this point, unfinishedResponses has been decremented, but netty’s numActiveStreams hasn’t been decremented yet.

So I believe the flow is as follows:

1. The first request succeeds and unfinishedResponses == 0, but numActiveStreams == 1.
2. 10 new requests are queued (since MAX_CONCURRENT_STREAMS = 10)
3. The last request fails, since unfinishedResponses == 9, but numActiveStreams == 10.
4. All subsequent requests try to use the same connection, but fail due to MAX_CONCURRENT_STREAMS violation.

I’ve also verified that the test code passes for the branch #3908

[ks-yim]

Demonstration…: https://github.com/ks-yim/max-stream-violated/blob/main/src/test/java/MaxStreamViolatedTest.java

It turns out that I was wrong about the cause in the above comment. It’s not related to decorator or 2 different clients with the same host.

The exception occurs when requests are made in a callback of another request like…:

 // exception
client.get("/resource").aggregate().thenRun(() -> concurrentReqToSameHost());

// but exception is not thrown if `concurrentReqToSameHost()` is called in async callback;
client.get("/resource").aggregate().thenRunAsync(() -> concurrentReqToSameHost(), ctx.eventLoop());

More on this in the demo code.