Blaze Client keeps half-closed connection after server sends FIN-ACK through an idle connection

This is a problem because both sides can’t release resources allocated to the connection, but also, because blaze will later try and fail to use that connection.

I prepared code that reproduces this behaviour using BlazeServer’s idleTimeout as the mechanism closing the connection on server’s side.

The scenario:

• make a request
• wait until server closes the connection (due to 5s idle timeout)
• make another request

For comparison there’s the same scenario but using AsyncHttpClient. The behaviour of AsyncHttpClient is in my opinion the expected behaviour (It’s also consistent with for example Firefox’s behaviour).

  val server = BlazeServerBuilder[IO](munitExecutionContext)
    .withIdleTimeout(5.second)
    .bindLocal(12345)
    .withHttpApp(Router("/" -> HttpRoutes.of[IO] { case GET -> Root => IO.sleep(1.seconds) >> Ok("") }).orNotFound)
    .resource

  test("Blaze Client keeps half-closed connection after server sends FIN-ACK through an idle connection") {
    (
      server,
      BlazeClientBuilder[IO](munitExecutionContext).resource
    ).tupled.use { case (_, client) =>for {
      _ <- client.expect[String](Uri.fromString(s"http://127.0.0.1:12345").yolo).attemptTap(result => IO(println(result)))
      _ <- IO.sleep(6.seconds)
      _ <- client.expect[String](Uri.fromString(s"http://127.0.0.1:12345").yolo).attemptTap(result => IO(println(result)))
    } yield ()
    }.unsafeRunSync()
  }

  test("AsyncHttpClient closes connection after server sends FIN-ACK through an idle connection") {
    (
      server,
      AsyncHttpClient.resource[IO]()
    ).tupled.use { case (_, client) => for {
      _ <- client.expect[String](Uri.fromString(s"http://127.0.0.1:12345").yolo).attemptTap(result => IO(println(result)))
      _ <- IO.sleep(6.seconds)
      _ <- client.expect[String](Uri.fromString(s"http://127.0.0.1:12345").yolo).attemptTap(result => IO(println(result)))
    } yield ()
    }.unsafeRunSync()
  }

(Full runnable code: https://github.com/RafalSumislawski/http4s/tree/blaze-half-closed-connection-issue)

The code itself just reproduces the behaviour, without checking if it’s correct. I use wireshark to analyse to communication between the client and server.

Blaze client:

In packet 1519 server closes the connection. The client ACKs it, but it doesn’t send a FIN, so the connection remains half-closed (CLOSE_WAIT from server’s perspective). Then packet 1521 is clients attempt to use that connection to send another request, to which the server responds with RST finally killing the connection. The client then establishes a new connection and successfully resends the request. Nonetheless precious resources have been wasted on the attempt to send request though closed connection. The retry mechanism was useful in this scenario, but it just covers the problem, not fixes it. And I see some other scenarios where it causes problems.

AsyncHttpClient:

After receiving FIN-ACK, the AsyncHttpClient immediately responds with FIN-ACK releasing resource on both ends. When a new request needs to be sent, I understands that it needs to open a new connection.

I realise this may be an issue inside blaze, not http4s, nonetheless I think the perspective of http4s-client user, and the comparison with other implementation is useful for clear definition of the problem and the expected behaviour.

[RafalSumislawski]

I’ll try to find the source of the problem.

[RafalSumislawski]

I’ve opened a PR fixing this for blaze. I will open a separate issue for ember, so that we don’t keep issues half-open (like ember keeps connections half-open 😉 )

Interesting, so Ember doesn’t even attempt to open a new connection? That’s probably a blind spot in the pooling we should fix.

I actually think that this retry behavior of blaze is more harmful than helpful. Sure it helps in this specific case. But in general, sending the same HTTP request twice is not safe. http4s doesn’t have enough knowledge to say if the request is idempotent or not, or to say if the server started processing the request blaze sent in the first attempt. (see the issue no 2 I described in #4796). Furthermore, blazes retry it unbounded in number of attempts and time. A big no-no for reliability.