• 25-Jan-2023
Lightrun Team
Author Lightrun Team
Share
This article is about fixing applicationjson default charset is not UTF-8 when parsing request in Ktorio Ktor

application/json default charset is not UTF-8 when parsing request in Ktorio Ktor

Lightrun Team
Lightrun Team
25-Jan-2023

Explanation of the problem

The current system is experiencing issues with decoding requests that have a Content-Type of “application/json”. These requests are not being decoded as UTF-8, while requests with a Content-Type of “application/json; charset=utf-8” are being decoded correctly.

To address this issue, the system must be updated to ensure that all requests with a Content-Type of “application/json” are decoded as UTF-8, regardless of the presence or absence of the “charset=utf-8” attribute.

The following code snippet illustrates the current behavior of the system:

if (request.headers["Content-Type"] == "application/json" && 
    !request.headers["Content-Type"].includes("charset=utf-8")) {
    // decode request as non-UTF-8
} else {
    // decode request as UTF-8
}

To fix this issue, the following code snippet can be used:

if (request.headers["Content-Type"] == "application/json") {
    // decode request as UTF-8
}

Troubleshooting with the Lightrun Developer Observability Platform

Getting a sense of what’s actually happening inside a live application is a frustrating experience, one that relies mostly on querying and observing whatever logs were written during development.
Lightrun is a Developer Observability Platform, allowing developers to add telemetry to live applications in real-time, on-demand, and right from the IDE.

  • Instantly add logs to, set metrics in, and take snapshots of live applications
  • Insights delivered straight to your IDE or CLI
  • Works where you do: dev, QA, staging, CI/CD, and production

Start for free today

Problem solution for application/json default charset is not UTF-8 when parsing request in Ktorio Ktor

The proposed solution to this issue is the use of the “receiveTextWithCorrectEncoding()” function. This function is designed to handle the case where the Content-Type is not specified in the HTTP header. It first checks if the Content-Type is specified, and if it is not, it uses ISO_8859_1 as the default charset. However, if the Content-Type is specified as “application/json”, it uses UTF-8 as the default charset instead.
fun ContentType.defaultCharset(): Charset = when (this) {
    ContentType.Application.Json -> Charsets.UTF_8
    else -> Charsets.ISO_8859_1
}

This block of code shows how the function checks the Content-Type and assigns the default charset accordingly. The function then uses the suitable charset to decode the HTTP request body and returns the result as a string.

val contentType = request.contentType()
val suitableCharset = contentType.charset() ?: contentType.defaultCharset()
return receiveStream().bufferedReader(charset = suitableCharset).readText()

It is worth noting that the proposed solution is only a workaround, the developer notes that it is still an issue and it may need further attention.

The use of the “receiveTextWithCorrectEncoding()” function can help to mitigate the impact of not specifying a charset in the HTTP header, but it is important to keep in mind that it is not a definitive solution to the problem. It is always best to specify the charset in the HTTP header to ensure that the data is interpreted correctly.

Other popular problems with Ktor

Problem: Handling of multipart/form-data requests

One of the most common problems developers face when working with Ktor is handling multipart/form-data requests. These types of requests are used to upload files or submit form data, and they can be difficult to handle correctly. The problem is that Ktor does not provide a built-in solution for handling multipart/form-data requests, and developers need to implement their own solution.

Solution:

Use the “ktor-features-multipart” module, which provides a multipart feature for Ktor. This feature allows developers to easily handle multipart/form-data requests by adding the following code to the application’s main function:

install(Multipart) {
    // configure the feature here
}

The feature can then be used to handle the multipart/form-data requests in the application’s routing functions:

post("/upload") {
    val multipart = call.receiveMultipart()
    multipart.forEachPart { part ->
        when (part) {
            is PartData.FileItem -> {
                val ext = File(part.originalFileName).extension
                part.streamProvider().use { its ->
                    // save file
                }
            }
            is PartData.FormItem -> {
                // handle form data
            }
        }
    }
}

This solution makes it easy to handle the multipart/form-data requests in a way that is consistent with the rest of the application’s routing functions.

Problem: Handling of long-polling requests

Another common problem developers encounter when working with Ktor is handling long-polling requests. These types of requests are used to wait for an event or a response for a certain amount of time. However, Ktor does not provide a built-in solution for handling long-polling requests and developers need to implement their own solution.

Solution:

Use the “ktor-features-timeout” module, which provides a timeout feature for Ktor. This feature allows developers to set a timeout for a request and handle it if it is not fulfilled within the specified time frame. The following code can be added to the application’s main function to install the timeout feature:

install(Timeout) {
    requestTimeoutMillis = 5_000
    timeoutResponse = HttpResponse(HttpStatusCode.RequestTimeout) {
        // response to return when a timeout occurs
    }
}

The feature can then be used in the routing function to handle the timeout if it occurs:

get("/long-poll") {
    try {
        // wait for an event or a response
    } catch (e: TimeoutCancellationException) {
        // handle the timeout
    }
}

This solution makes it easy to handle the long-polling requests in a way that is consistent with the rest of the application’s routing functions.

Problem: Handling of CORS requests

CORS (Cross-Origin Resource Sharing) is a mechanism that allows web pages to make requests to different origins than the one that served the web page. It is a common problem that developers face when working with Ktor, as it does not provide a built-in solution for handling CORS requests.

Solution:

Use the “ktor-features-cors” module, which provides a CORS feature for Ktor. This feature allows developers to configure the CORS headers that are sent in the response. The following code can be added to the application’s main function to install the CORS feature:

install(CORS) {
    anyHost()
    allowCredentials = true
    listOf("Content-Type", "Authorization").forEach { header("Access-Control-Allow-Headers", it) }
    listOf("GET", "POST", "PUT", "DELETE").forEach { method(it) }
}

This configuration allows any host to access the application, allows credentials to be sent, allows a specific set of headers and methods for CORS requests.

The CORS feature can then be used in the routing function to handle the CORS requests, for example:

options("/data") {
    // handle CORS preflight request
}

This solution makes it easy to handle the CORS requests in a way that is consistent with the rest of the application’s routing functions and ensuring that the web page can securely communicate with the different origins.

A brief introduction to Ktor

Ktor is a framework for building asynchronous web applications in the Kotlin programming language. It is built on top of the Kotlin coroutines library, which allows for efficient and non-blocking I/O operations. Ktor provides a wide range of features for handling HTTP requests and responses, such as routing, content negotiation, and form submission. It also provides a modular design, where developers can choose to include or exclude certain features based on the needs of their application, making it very flexible and customizable.

Ktor is highly optimized for performance and low memory usage, making it suitable for building high-performance web applications. It also provides support for various web standards, such as WebSockets, HTTP/2, and CORS, and a variety of other features such as support for file uploads and downloads, and handling of multipart/form-data requests. Additionally, it provides an easy-to-use API for handling JSON and other types of data. It also supports a variety of different platforms and deployment options, such as running on the JVM, Android, and JavaScript.

Most popular use cases for Ktor

  1. Building RESTful web services: Ktor can be used to build RESTful web services, which are web services that follow the principles of Representational State Transfer (REST). Ktor provides a routing feature that allows developers to define the routes of the web service and handle the corresponding HTTP requests and responses. The following code block shows an example of how to define a GET route in Ktor:
get("/users/{id}") {
    val userId = call.parameters["id"]?.toInt() ?: throw IllegalArgumentException("Invalid user ID")
    val user = users.find { it.id == userId }
    if (user != null) call.respond(user) else call.respond(HttpStatusCode.NotFound)
}
  1. Building web applications with real-time functionality: Ktor can be used to build web applications that have real-time functionality, such as chat applications or online games. It provides support for WebSockets, which is a communication protocol that allows for two-way communication between a client and a server. The following code block shows an example of how to handle a WebSocket connection in Ktor:
webSocket("/ws") {
    for (frame in incoming) {
        when (frame) {
            is Frame.Text -> {
                // handle text frame
            }
            is Frame.Binary -> {
                // handle binary frame
            }
        }
    }
}
  1. Building mobile and desktop applications using Kotlin Multiplatform: Ktor can be used to build mobile and desktop applications using Kotlin Multiplatform. It allows to share the codebase between multiple platforms such as Android, iOS, Linux, Windows, MacOS and more. With Ktor, developers can define the server-side logic of the application once and reuse it across different platforms, making it a more efficient way to develop cross-platform applications.
Share

It’s Really not that Complicated.

You can actually understand what’s going on inside your live applications. It’s a registration form away.

Get Lightrun

Lets Talk!

Looking for more information about Lightrun and debugging?
We’d love to hear from you!
Drop us a line and we’ll get back to you shortly.

By submitting this form, I agree to Lightrun’s Privacy Policy and Terms of Use.