Apache HTTP Client often hangs forever (deadlocks) in async block

Ktor Version

1.0.0

Ktor Engine Used(client or server and name)

Apache HTTP Client

JVM Version, Operating System and Relevant Context

Oracle JDK 1.8.0_161-b12 (same issue on OpenJDK 8) OS: Windows 10, 64bit

Feedback

When running the Apache HTTP client (through the ktor client API), the client sometimes just “freezes” and hangs indefinitly, sending the application into a deadlock. This effect is very noticeable when calling the client from an async block.

Please consider the following self-contained JUnit test:

import io.ktor.client.HttpClient
import io.ktor.client.engine.apache.Apache
import io.ktor.client.engine.cio.CIO
import io.ktor.client.request.get
import kotlinx.coroutines.*
import org.junit.Test

class CoroutineTest {

    @Test
    fun testWithGlobalScope(){
        println("WITH GLOBAL SCOPE")
        runBlocking {
            runTest(GlobalScope)
        }
    }

    @Test
    fun testWithCoroutineScope(){
        println("WITH COROUTINE SCOPE")
        runBlocking {
            runTest(this)
        }
    }

    @Test
    fun testWithIndependentScope(){
        println("WITH BLOCK SCOPE")
        runBlocking {
            coroutineScope{
                runTest(this)
            }
        }
    }

    private suspend fun runTest(scope: CoroutineScope): Int{
        // retrieve number of CPU cores
        var sum = 0
        val tasks = mutableListOf<Deferred<Int>>()
        for(i in 0 until 50){
            println("Launching async task #${i+1}")
            tasks += scope.async {
                val client = HttpClient(Apache)
                try{
                    log("HTTP client open. Performing GET")
                    val string = client.get<String>("https://www.google.com")
                    log("GET result received: ${string.replace("\r?\n", " ").substring(0, 20)}...")
                }finally{
                    log("Closing HTTP client.")
                    client.close()
                    log("HTTP client closed.")
                }
                return@async 1
            }
        }
        for(i in 0 until tasks.size){
            sum += tasks[i].await()
            log("Task #${i} completed.")
        }
        return sum
    }

    private fun log(message: String){
        println("[${Thread.currentThread().name}] ${message}")
    }
}

In particular, testWithGlobalScope (Apache HTTP, called from GlobalScope.async(...)) tends to freeze almost every try at some point (although I observed that the target web URL does make a difference; perhaps it’s a race condition dependent on the response time?). The other two variants work fine for me.

As a side note, I have observed that if you are in a test run where the HTTP client causes a freeze, it will always occur after the n-th request, where n is the number of CPU cores on your machine (I guess that kotlin determines the size of its executor pools based on this number, when the pools run out of usable threads the remaining coroutines can’t execute anymore).

Interestingly, if we replace HttpClient(Apache) by HttpClient(CIO), then things work like a charm. I would argue that this strongly points towards the Apache HTTP Client as the culprit of the deadlocks.

[MartinHaeusler]

Oh okay… well, I’m glad to see this issue fixed, thanks for the explanation 😃 👍

[e5l]

Thanks for the report, I’ll investigate and report the details.