HTTP Client does not receive body and gets stuck or silently dies

Ktor Version and Engine Used (client or server and name) 1.3.2, client: apache (but same behavior on CIO), server: netty, enabled calls logging Http client:

fun httpClient() =
    HttpClient(Apache) {
        install(JsonFeature) {
            serializer = JacksonSerializer()
        }

        install(ResponseObserver) {
            onResponse {
                httpClientLogger.trace { "Response received" }
            }
        }

        install(Logging) {
            logger = Logger.TRACE
            level = LogLevel.ALL
        }
    }

Describe the bug I have following code:

        runCatching {
            client.get<ConversationInformation> {
                url(endpoint)
                header("Authorization", "Bearer $token")
            }
        }.onFailure {
            logger.error(it) { "It was not possible to fetch conversation information!" }
        }.onSuccess {
            logger.debug { "Successfully got conversation information." }
            logger.trace { it }
        }.getOrNull()

And when executed, SOMETIMES it never logs either onFailure nor onSuccess block. The client GET request in some cases either silently fails or gets stuck/suspended for indefinite time, not sure how is that possible, but following logs are produced by the call logging mentioned previously.

21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - REQUEST: https://<internal URL>
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - METHOD: HttpMethod(value=GET)
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - COMMON HEADERS
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - -> Authorization: Bearer <some bearer>
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - -> Accept: application/json
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - -> Accept-Charset: UTF-8
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - CONTENT HEADERS
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - BODY Content-Type: null
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - BODY START
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging -
21/04/2020 12:27:54 [TRACE] com.wire.HttpCallsLogging - BODY END
21/04/2020 12:27:55 [TRACE] com.wire.ObserverLogger - Response received
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - BODY Content-Type: application/json
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - BODY START
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - RESPONSE: 200 OK
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - METHOD: HttpMethod(value=GET)
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - FROM: https://<internal URL>
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - COMMON HEADERS
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Date: Tue, 21 Apr 2020 12:27:54 GMT
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Content-Type: application/json
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Vary: Accept-Encoding
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Content-Length: 125
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Via: 1.1 google
21/04/2020 12:27:55 [TRACE] com.wire.HttpCallsLogging - -> Alt-Svc: clear

It seems like the body was never received.

When the request is successful, log looks like this:

21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - REQUEST: https://<internal URL>
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - METHOD: HttpMethod(value=GET)
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - COMMON HEADERS
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Authorization: Bearer <some bearer>
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Accept: application/json
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Accept-Charset: UTF-8
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - CONTENT HEADERS
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - BODY Content-Type: null
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - BODY START
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging -
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - BODY END
21/04/2020 12:43:24 [TRACE] com.wire.ObserverLogger - Response received
21/04/2020 12:43:24 [TRACE] com.wire.ObserverLogger - Sending to registry
21/04/2020 12:43:24 [TRACE] com.wire.ObserverLogger - Registered
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - BODY Content-Type: application/json
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - BODY START
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - RESPONSE: 200 OK
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - METHOD: HttpMethod(value=GET)
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - FROM: https://<internal URL>
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - COMMON HEADERS
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Date: Tue, 21 Apr 2020 12:43:24 GMT
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Content-Type: application/json
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Vary: Accept-Encoding
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Content-Length: 125
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Via: 1.1 google
21/04/2020 12:43:24 [TRACE] com.wire.HttpCallsLogging - -> Alt-Svc: clear
21/04/2020 12:43:24 [DEBUG] c.w.b.p.services.ConversationService - Successfully got conversation information.
21/04/2020 12:43:24 [TRACE] c.w.b.p.services.ConversationService - ConversationInformationd(dto=SomeDataHere)
21/04/2020 12:43:25 [TRACE] com.wire.HttpCallsLogging - {"id":"3f7fad1d-5066-487d-8836-9e40928bb12f","name":"a","members":[{"id":"420e4b22-fac2-47e8-b07f-63958d7fcccd","status":0}]}
21/04/2020 12:43:25 [TRACE] com.wire.HttpCallsLogging - BODY END

To Reproduce It is happening for like 1/5 of all requests, whole service code available here

Expected behavior IF the request is not successful, the client should throw exception or eventually time out and then throw exception and not to silently die. Not sure what is happening though

EDIT: tried with

            val response = client.get<HttpStatement> {
                url(endpoint)
                header("Authorization", "Bearer $token")
            }.execute()
            logger.trace { "Executed" }

as well and if it gets stuck, Executed is never printed. Also, seems similar to #679

EDIT2: I found out that the flow is following:

1. http client executes GET
2. target server respond with 200
3. http client gets stuck
4. target server throws SocketTimeoutException after a while

I this case I would expect exception to be thrown.

EDIT3: this is definitely connected to

        install(ResponseObserver) {
            onResponse {
                httpClientLogger.trace { "Response received" }
            }
        }

as without observer, everything is nice and shiny

EDIT4: Happens with Jetty and OkHttp engines as well. Mostly it happens after multiple requests, like first one or two works, but then it stops working. This is happening in our staging Kubernets.

[brandonwu]

For anybody else encountering this: ResponseObserver splits the response body ByteReadChannel into two channels so that the hook can read from it without also consuming it from the response returned at the end of the pipeline. But you also need to consume all the bytes in the body channel sent to the hook with

it.content.discard()

or else bad things will happen (timeouts, blank bodies) because the other channel is stalled. The documentation for ResponseObserver should probably mention this but it’s mostly blank.

[Stexxe]

Cannot reproduce it using Ktor 1.6.1.