Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Disturbing error logs from blaze-client 0.17 under heavy load

See original GitHub issue

Problem occurs with blaze client (version 0.17.0-M3). I’m using default PooledHttp1Client() and after performing load testing with apache benchmark(and other tools like siege or gatling)(few thousands GET requests, concurrency level 15) I get stacktrace listed below. Despite error message everything seems to work fine(All the messages I’m getting from client are correct)

Background:

  • I’m using akka-http as server and http4s as a client for getting request
  • I’ve set up mock server for testing performance, so client is getting always the same message
  • sometimes error occurs only after few hundred messages, sometimes after few thousands.

Discussion on gitter suggested that it’s probably debug message that somehow got to stacktrace

[info] java.lang.Exception: Cannot send inbound command on disconnected stage
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:241) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:17) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:17) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:90) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:87) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128) [na:1.8.0_131]
[info] 	at sun.nio.ch.Invoker$2.run(Invoker.java:218) [na:1.8.0_131]
[info] 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
[info] 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
[info]  [2017-07-11T08:43:30.867Z] [ERROR]; ; ReadBufferingStage received unhandled error command
[info] java.lang.Exception: Cannot send inbound command on disconnected stage
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:241) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:17) ~[http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:17) ~[http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:90) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:87) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128) [na:1.8.0_131]
[info] 	at sun.nio.ch.Invoker$2.run(Invoker.java:218) [na:1.8.0_131]
[info] 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
[info] 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
[info]  [2017-07-11T08:43:30.868Z] [ERROR]; ; NIO2 channel closed with an unexpected error
[info] java.lang.Exception: Cannot send inbound command on disconnected stage
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:241) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:17) ~[http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:17) ~[http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.inboundCommand(Stages.scala:251) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:14) [http4s-blaze-client_2.11-0.17.0-M3.jar:0.17.0-M3]
[info] 	at org.http4s.blaze.pipeline.Head$class.sendInboundCommand(Stages.scala:238) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:90) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:87) [blaze-core_2.11-0.12.5.jar:0.12.5]
[info] 	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128) [na:1.8.0_131]
[info] 	at sun.nio.ch.Invoker$2.run(Invoker.java:218) [na:1.8.0_131]
[info] 	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
[info] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
[info] 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:7
  • Comments:26 (9 by maintainers)

github_iconTop GitHub Comments

3reactions
BobWall23commented, Jan 3, 2019

Seeing these on a server running http4s 0.20.0-M4.

java.lang.Exception: Cannot send inbound command on disconnected stage
                at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:264)
                at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:252)
                at org.http4s.blazecore.IdleTimeoutStage.sendInboundCommand(IdleTimeoutStage.scala:12)
                at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:273)
                at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:271)
                at org.http4s.blazecore.IdleTimeoutStage.inboundCommand(IdleTimeoutStage.scala:12)
                at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:256)
                at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:252)
                at org.http4s.blaze.channel.ChannelHead.sendInboundCommand(ChannelHead.scala:9)
                at org.http4s.blaze.channel.nio1.NIO1HeadStage.org$http4s$blaze$channel$nio1$NIO1HeadStage$$doClose$1(NIO1HeadStage.scala:311)
                at org.http4s.blaze.channel.nio1.NIO1HeadStage$$anon$3.run(NIO1HeadStage.scala:326)
                at org.http4s.blaze.util.TaskQueue.go$1(TaskQueue.scala:82)
                at org.http4s.blaze.util.TaskQueue.executeTasks(TaskQueue.scala:99)
                at org.http4s.blaze.channel.nio1.SelectorLoop.org$http4s$blaze$channel$nio1$SelectorLoop$$runLoop(SelectorLoop.scala:165)
                at org.http4s.blaze.channel.nio1.SelectorLoop$$anon$1.run(SelectorLoop.scala:67)
                at java.lang.Thread.run(Thread.java:748)
2reactions
lucaviolanticommented, Apr 26, 2018

Same here on 0.18.9:

2018-04-26 01:49:08,480 [ERROR] org.http4s.blaze.pipeline.Stage -
java.lang.Exception: Cannot send inbound command on disconnected stage
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:241)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.dt_2_sendInboundCommand_50(ByteBufferHead.scala)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:94)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:91)
	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-04-26 01:49:08,481 [ERROR] org.http4s.blaze.pipeline.Stage - ReadBufferingStage received unhandled error command
java.lang.Exception: Cannot send inbound command on disconnected stage
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:241)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.dt_2_sendInboundCommand_50(ByteBufferHead.scala)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:94)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:91)
	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-04-26 01:49:08,481 [ERROR] org.http4s.blaze.pipeline.Stage - NIO2 channel closed with an unexpected error
java.lang.Exception: Cannot send inbound command on disconnected stage
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:241)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ClientTimeoutStage.sendInboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ClientTimeoutStage.inboundCommand(ClientTimeoutStage.scala:14)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.client.blaze.ReadBufferStage.sendInboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.inboundCommand(Stages.scala:251)
	at org.http4s.blaze.pipeline.Head.inboundCommand$(Stages.scala:249)
	at org.http4s.client.blaze.ReadBufferStage.inboundCommand(ReadBufferStage.scala:13)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand(Stages.scala:238)
	at org.http4s.blaze.pipeline.Head.sendInboundCommand$(Stages.scala:235)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.dt_2_sendInboundCommand_50(ByteBufferHead.scala)
	at org.http4s.blaze.channel.nio2.ByteBufferHead.sendInboundCommand(ByteBufferHead.scala:17)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:94)
	at org.http4s.blaze.channel.nio2.ByteBufferHead$$anon$3.failed(ByteBufferHead.scala:91)
	at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
	at sun.nio.ch.Invoker$2.run(Invoker.java:218)
	at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Read more comments on GitHub >

github_iconTop Results From Across the Web

Disturbing error logs from blaze-client 0.17 under heavy load
Problem occurs with blaze client (version 0.17.0-M3). I'm using default PooledHttp1Client() and after performing load testing with apache ...
Read more >
http4s/http4s - Gitter
I'm trying to add tracing to an application and I'm not sure if I'm taking the right approach. I want to create a...
Read more >
Http4s Blaze Client Builder wait queue full failure
Our service is built using Scala, Http4s and Cats-Effect and is using http4s-blaze-client library for making outbound HTTP calls. Currently in ...
Read more >
MadCapIdea | Sacha's Blog - WordPress.com
This is kind of a hard post to write, as what I will have to do ... Keeping all the traffic in one...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

alpn-api vulnerability

Next page

HTTP/2 response bodies (strict) bigger than 16KiB are not being sent with Ember