application/json default charset is not UTF-8 when parsing request in Ktorio Ktor

Explanation of the problem

The current system is experiencing issues with decoding requests that have a Content-Type of “application/json”. These requests are not being decoded as UTF-8, while requests with a Content-Type of “application/json; charset=utf-8” are being decoded correctly.

To address this issue, the system must be updated to ensure that all requests with a Content-Type of “application/json” are decoded as UTF-8, regardless of the presence or absence of the “charset=utf-8” attribute.

The following code snippet illustrates the current behavior of the system:

if (request.headers["Content-Type"] == "application/json" && 
    !request.headers["Content-Type"].includes("charset=utf-8")) {
    // decode request as non-UTF-8
} else {
    // decode request as UTF-8
}

To fix this issue, the following code snippet can be used:

if (request.headers["Content-Type"] == "application/json") {
    // decode request as UTF-8
}

Troubleshooting with the Lightrun Developer Observability Platform

Getting a sense of what’s actually happening inside a live application is a frustrating experience, one that relies mostly on querying and observing whatever logs were written during development.
Lightrun is a Developer Observability Platform, allowing developers to add telemetry to live applications in real-time, on-demand, and right from the IDE.

• Instantly add logs to, set metrics in, and take snapshots of live applications
• Insights delivered straight to your IDE or CLI
• Works where you do: dev, QA, staging, CI/CD, and production

Start for free today

Problem solution for application/json default charset is not UTF-8 when parsing request in Ktorio Ktor

fun ContentType.defaultCharset(): Charset = when (this) {
    ContentType.Application.Json -> Charsets.UTF_8
    else -> Charsets.ISO_8859_1
}

This block of code shows how the function checks the Content-Type and assigns the default charset accordingly. The function then uses the suitable charset to decode the HTTP request body and returns the result as a string.

val contentType = request.contentType()
val suitableCharset = contentType.charset() ?: contentType.defaultCharset()
return receiveStream().bufferedReader(charset = suitableCharset).readText()

It is worth noting that the proposed solution is only a workaround, the developer notes that it is still an issue and it may need further attention.

The use of the “receiveTextWithCorrectEncoding()” function can help to mitigate the impact of not specifying a charset in the HTTP header, but it is important to keep in mind that it is not a definitive solution to the problem. It is always best to specify the charset in the HTTP header to ensure that the data is interpreted correctly.

Other popular problems with Ktor

install(Multipart) {
    // configure the feature here
}

The feature can then be used to handle the multipart/form-data requests in the application’s routing functions:

post("/upload") {
    val multipart = call.receiveMultipart()
    multipart.forEachPart { part ->
        when (part) {
            is PartData.FileItem -> {
                val ext = File(part.originalFileName).extension
                part.streamProvider().use { its ->
                    // save file
                }
            }
            is PartData.FormItem -> {
                // handle form data
            }
        }
    }
}

This solution makes it easy to handle the multipart/form-data requests in a way that is consistent with the rest of the application’s routing functions.

Problem: Handling of long-polling requests

Another common problem developers encounter when working with Ktor is handling long-polling requests. These types of requests are used to wait for an event or a response for a certain amount of time. However, Ktor does not provide a built-in solution for handling long-polling requests and developers need to implement their own solution.

Solution:

Use the “ktor-features-timeout” module, which provides a timeout feature for Ktor. This feature allows developers to set a timeout for a request and handle it if it is not fulfilled within the specified time frame. The following code can be added to the application’s main function to install the timeout feature:

install(Timeout) {
    requestTimeoutMillis = 5_000
    timeoutResponse = HttpResponse(HttpStatusCode.RequestTimeout) {
        // response to return when a timeout occurs
    }
}

The feature can then be used in the routing function to handle the timeout if it occurs:

get("/long-poll") {
    try {
        // wait for an event or a response
    } catch (e: TimeoutCancellationException) {
        // handle the timeout
    }
}

This solution makes it easy to handle the long-polling requests in a way that is consistent with the rest of the application’s routing functions.

Problem: Handling of CORS requests

CORS (Cross-Origin Resource Sharing) is a mechanism that allows web pages to make requests to different origins than the one that served the web page. It is a common problem that developers face when working with Ktor, as it does not provide a built-in solution for handling CORS requests.

Solution:

Use the “ktor-features-cors” module, which provides a CORS feature for Ktor. This feature allows developers to configure the CORS headers that are sent in the response. The following code can be added to the application’s main function to install the CORS feature:

install(CORS) {
    anyHost()
    allowCredentials = true
    listOf("Content-Type", "Authorization").forEach { header("Access-Control-Allow-Headers", it) }
    listOf("GET", "POST", "PUT", "DELETE").forEach { method(it) }
}

This configuration allows any host to access the application, allows credentials to be sent, allows a specific set of headers and methods for CORS requests.

The CORS feature can then be used in the routing function to handle the CORS requests, for example:

options("/data") {
    // handle CORS preflight request
}

A brief introduction to Ktor

Ktor is a framework for building asynchronous web applications in the Kotlin programming language. It is built on top of the Kotlin coroutines library, which allows for efficient and non-blocking I/O operations. Ktor provides a wide range of features for handling HTTP requests and responses, such as routing, content negotiation, and form submission. It also provides a modular design, where developers can choose to include or exclude certain features based on the needs of their application, making it very flexible and customizable.

Ktor is highly optimized for performance and low memory usage, making it suitable for building high-performance web applications. It also provides support for various web standards, such as WebSockets, HTTP/2, and CORS, and a variety of other features such as support for file uploads and downloads, and handling of multipart/form-data requests. Additionally, it provides an easy-to-use API for handling JSON and other types of data. It also supports a variety of different platforms and deployment options, such as running on the JVM, Android, and JavaScript.

Most popular use cases for Ktor

get("/users/{id}") {
    val userId = call.parameters["id"]?.toInt() ?: throw IllegalArgumentException("Invalid user ID")
    val user = users.find { it.id == userId }
    if (user != null) call.respond(user) else call.respond(HttpStatusCode.NotFound)
}

webSocket("/ws") {
    for (frame in incoming) {
        when (frame) {
            is Frame.Text -> {
                // handle text frame
            }
            is Frame.Binary -> {
                // handle binary frame
            }
        }
    }
}