Ktor Client - Downloading large files with ByteReadChannel

Hello guys,

I’m developing a Kotlin Multiplatform app for Android and iOS. The app should be able to download large files via HTTP by writing the data directly into a local file. The following code works perfectly for small files (I tried it with files smaller than 1 MB).

fileWriter.initialize()

val httpRequest = HttpRequestBuilder().apply {
    url("…")
    header("Authorization", "Bearer …")
}

httpClient.get(httpRequest).execute { response: HttpResponse ->
    val bytes = response.receive<ByteReadChannel>()

    val byteBufferSize = 1024 * 100
    val byteBuffer = ByteArray(byteBufferSize)

    do {
        val currentRead = bytes.readAvailable(byteBuffer, 0, byteBufferSize)

        if (currentRead > 0) {
            println("Write ${currentRead} bytes...")

            // write the data into the file
            fileWriter.writeToFile(if (currentRead < byteBufferSize) {
                byteBuffer.slice(0 until currentRead)
            } else {
                byteBuffer
            })
        }
    } while (currentRead >= 0)

    fileWriter.finish()
}

I tried to download a 2 GB file, but the app is stuck inhttpClient.get(httpRequest) until the RAM usage is too high and the OS decides to kill my app. I would expect that response.receive<ByteReadChannel>() streams the content of the request and I can read it step by step with bytes.readAvailable(…), but the code actually never reaches bytes.readAvailable(…).

Why is httpClient.get(httpRequest) downloading the 2 GB intead of streaming the data? How would you approach this and how can you download a large file without filling the RAM with the Ktor client?

Thank you! 😃

Edit: I tested the described scenario only on iOS so far. Ktor version: 1.3.1 Kotlin version: 1.3.61

[e5l]

Streaming support and download is fixed in master. The fix will be available with the next minor release.

[dmitrievanthony]

As far as I remember, native clients don’t implement streaming so far, they download the whole response at first and only after that return it to you. As a result, if your response is big it will be fully loaded into the memory and may cause OOM.

Hi @cy6erGn0m, @e5l, please correct me if I’m wrong.