[ktor-server-netty] Unable to send responses larger than ~2GB

We are unable to send responses larger than 2GB which include the Content-Length header. The issue would appear to be here: https://github.com/ktorio/ktor/blob/e2e622c90f9beba9248249714ec58c0d33e7cfd7/ktor-server/ktor-server-netty/src/io/ktor/server/netty/cio/NettyResponsePipeline.kt#L191

This is also mentioned in the stacktrace we get. We are running version ktor 0.9.2 with ktor-server-netty.

[DanThrane]

Sure. The response body is streamed directly from an external system, hence our need for writing directly to the ByteWriteChannel. The code works, even for files larger than Int.MAX_VALUE, as long as we don’t send the Content-Length. For files smaller than Int.MAX_VALUE we can send the Content-Length.

get("foo") {
    val size: Long = // ...
    val contentType: ContentType = // ...
    call.respondDirectWrite(size, contentType, HttpStatusCode.OK) {
        // Write something
    }
}

suspend fun ApplicationCall.respondDirectWrite(
    size: Long? = null,
    contentType: ContentType? = null,
    status: HttpStatusCode? = null,
    writer: suspend ByteWriteChannel.() -> Unit
) {
    val message = DirectWriteContent(writer, size, contentType, status)
    return respond(message)
}

class DirectWriteContent(
    private val writer: suspend ByteWriteChannel.() -> Unit,
    override val contentLength: Long? = null,
    override val contentType: ContentType? = null,
    override val status: HttpStatusCode? = null
) : OutgoingContent.WriteChannelContent() {
    override suspend fun writeTo(channel: ByteWriteChannel) {
        writer(channel)
    }
}

[e5l]

Awesome. Please reopen this if it happens again.