Abnormal NIO1HeadStage termination and Error in requestLoop() on client timeout

Version

We are using http4s version 0.20.0-M1 with blaze version 0.14.0-M9

Description

We are seeing the following pair of errors in the logs. The service completes the request successfully, so no errors are exposed to the client, but we believe these errors to be occurring on a per-client basis. So, when a client makes a request, they can make multiple requests to the service without triggering the exceptions, but once there is a pause in the rate of requests we see the errors in the logs.

13:41:36.946 [blaze-selector-0-2] ERROR o.h.blaze.channel.nio1.NIO1HeadStage - Abnormal NIO1HeadStage termination
java.util.concurrent.TimeoutException: Timeout of 30 seconds triggered. Killing pipeline.
	at org.http4s.blaze.pipeline.stages.TimeoutStageBase$$anon$1.run(TimeoutStageBase.scala:29)
	at org.http4s.blaze.util.Execution$$anon$3.execute(Execution.scala:66)
	at org.http4s.blaze.util.TickWheelExecutor$Node.run(TickWheelExecutor.scala:261)
	at org.http4s.blaze.util.TickWheelExecutor$Bucket.checkNext$1(TickWheelExecutor.scala:193)
	at org.http4s.blaze.util.TickWheelExecutor$Bucket.prune(TickWheelExecutor.scala:200)
	at org.http4s.blaze.util.TickWheelExecutor.go$3(TickWheelExecutor.scala:153)
	at org.http4s.blaze.util.TickWheelExecutor.org$http4s$blaze$util$TickWheelExecutor$$cycle(TickWheelExecutor.scala:156)
	at org.http4s.blaze.util.TickWheelExecutor$$anon$1.run(TickWheelExecutor.scala:55)
13:41:36.946 [blaze-selector-0-2] ERROR o.h.s.blaze.Http1ServerStage$$anon$1 - Fatal Error: Error in requestLoop()
java.util.concurrent.TimeoutException: Timeout of 30 seconds triggered. Killing pipeline.
	at org.http4s.blaze.pipeline.stages.TimeoutStageBase$$anon$1.run(TimeoutStageBase.scala:29)
	at org.http4s.blaze.util.Execution$$anon$3.execute(Execution.scala:66)
	at org.http4s.blaze.util.TickWheelExecutor$Node.run(TickWheelExecutor.scala:261)
	at org.http4s.blaze.util.TickWheelExecutor$Bucket.checkNext$1(TickWheelExecutor.scala:193)
	at org.http4s.blaze.util.TickWheelExecutor$Bucket.prune(TickWheelExecutor.scala:200)
	at org.http4s.blaze.util.TickWheelExecutor.go$3(TickWheelExecutor.scala:153)
	at org.http4s.blaze.util.TickWheelExecutor.org$http4s$blaze$util$TickWheelExecutor$$cycle(TickWheelExecutor.scala:156)
	at org.http4s.blaze.util.TickWheelExecutor$$anon$1.run(TickWheelExecutor.scala:55)

Please let me know if there are additional details that would be helpful! Thank you!

[rossabaker]

Aha! It happens on Chrome and Firefox.

[rossabaker]

So it actually is a real timeout. It’s because the idleTimeout is not just the timeout for an active request, but for a persistent connection. This is the server too loudly chasing off a persistent connection that the browser kept.

http4s/http4s#2244 makes it quieter and less scary with the existing semantics.

http4s/blaze#675 opens a discussion about whether we’re conflating settings and what smarter things we might do.