[SECURITY] I Keep Finding Netty HTTP Request/Response Splitting Vulnerabilities in OSS

Hi Netty Team,

I’ve recently been doing security research directly focused on HTTP Response Splitting vulnerabilities due to a misuse of Netty by disabling header validation by passing false to any of these constructors:

• DefaultFullHttpResponse
• DefaultHttpResponse (2)
• DefaultHttpHeaders

I’ve written a CodeQL query as a part of the GitHub Security Lab Bug Bounty Program which detects this vulnerability in OSS. That query can be found here:

• https://github.com/Semmle/ql/blob/master/java/ql/src/Security/CWE/CWE-113/NettyResponseSplitting.ql
• https://lgtm.com/rules/1510696449842/

The GitHub Security Lab Team was kind enough to award me $1,500 for this submission: https://github.com/github/securitylab/issues/22

---

HTTP Response Splitting is a CRLF injection vulnerability. In this particular case, it’s because the libraries building on top of netty are disabling the header validation. This means that if user supplied data flows to the setter locations for these headersm an attacker can perform the following attacks.

HTTP Response Splitting

This is more commonly associated with CWE-113. This vulnerability allows an attacker to control the full response of the request.

For example, taking this example from a real vulnerability I found in Ratpack (see below) assume that the following code exists.

// A real vulnerable example. Ratpack has been vulnerable to HTTP Response Splitting
RatpackServer startedServer =  RatpackServer.start(server -> {
    server.handlers(chain -> chain.all(ctx -> {
        // User supplied query parameter
        String header = ctx.getRequest().getQueryParams().get("header");
        // User supplied data used to populate a header value.
        ctx.header("the-header", header)
            .render("OK!");
    }));
});

By sending the following in the header parameter (assume newlines are \r\n):

Content-Type: text/html
X-XSS-Protection: 0

<script>alert(document.domain)</script>

The attacker is able to fully control the HTML response from the server. This vulnerability enables the following attacks:

• Cross-User Defacement
• Cache Poisoning
• Cross-Site Scripting
• Page Hijacking

HTTP Request Splitting

This vulnerability is different from HTTP Response Splitting because in this case, the user supplied data is being used to create another outgoing HTTP Request. One example of this would be a microarchitecture based infrastructure where the incoming request is relayed back out to an internal service.

When CRLF injection attacks exists here, the impact can also be quite nasty.

Let’s take the following server example (this vulnerability has not & does not exist in Ratpack, this is just an example).

// Not a real vulnerable example. Ratpack has never been vulnerable to HTTP Request Splitting
RatpackServer startedServer = RatpackServer.start(server -> {
    server.handlers(chain -> chain.all(ctx -> {
        // User supplied query parameter
        String header = ctx.getRequest().getQueryParams().get("header");
        HttpClient client = ctx.get(HttpClient.class);
        URI localhost = URI.create("http://localhost:12345");
        client
            .get(localhost, spec -> {
                spec.getHeaders().add("the-header", header);
            })
            .onError(Throwable::printStackTrace)
            .then(response -> ctx.render(response.getBody().getText()));
    }));
});

If, in this example, RatPack had header validation disabled for building outgoing requests an attacker would be able to control the outgoing HTTP request and inject their own additional requests.

For example, by sending the following in the header parameter (assume newlines are \r\n):

Note: Connection: Keep-Alive is used to keep the connection open so another request can be stuffed in.

Connection: Keep-Alive


POST /hello/super-secret HTTP/1.1
Host: 127.0.0.1
Content-Length: 31

{\"new\":\"json\",\"content\":\"here\"}

Here, the attacker is able to stuff an additional request into the GET request allowing the attacker to manipulate internal endpoints that they would normally be unable to control. This is also known as Server Side Request Forgery.

This is a real example of a vulnerability that I’ve actually found in a project (patch and CVE pending).

---

So far, I’ve been finding this vulnerability in quite a few users of Netty and have had the following CVE numbers issued.

Ratpack/Ratpack

CVE	https://nvd.nist.gov/vuln/detail/CVE-2019-17513
CVSSv3	7.5
GitHub Advisory	https://github.com/ratpack/ratpack/security/advisories/GHSA-mvqp-q37c-wf9j

line/armeria

CVE	https://nvd.nist.gov/vuln/detail/CVE-2019-16771
CVSSv3	6.5
GitHub Advisory	https://github.com/line/armeria/security/advisories/GHSA-35fr-h7jr-hh86

wso2/transport-http

CVE	https://nvd.nist.gov/vuln/detail/CVE-2019-10797
CVSSv3	6.5
Snyk Advisory	https://snyk.io/vuln/SNYK-JAVA-ORGWSO2TRANSPORTHTTP-548944

HotelsDotCom/styx

CVE	https://nvd.nist.gov/vuln/detail/CVE-2020-6858
CVSSv3	Not yet available
GitHub Advisory	https://github.com/HotelsDotCom/styx/security/advisories/GHSA-6v7p-v754-j89v

On top of these, I’ve got 2 more CVE’s pending (both in fairly popular projects). One of those vulns is HTTP Request Splitting (instead of HTTP Response Splitting).

---

The question that I pose to the Netty team is this, how can the netty team improve this issue. I’m not certain why many of these projects are disabling header validation, but the reality is that they are. Maybe it’s performance related? If so, maybe there’s something that can be done there?

Perhaps it’s time to consider deprecating the ability to disable this validation? Are there other solutions here?

[JLLeitschuh]

Hey Netty Team,

Has there been any discussion around this security issue? Part of me is tempted to consider that the ability to disable the request splitting checks a vulnerability in-itself, potentially worthy of a seperate CVE? IE. Should Netty have a CVE for even allowing you to disable these checks and have a version after-which users can’t disable these checks?

[Bennett-Lynch]

I think one of the primary reasons that some users/applications may choose to disable validateHeaders is the extremely binary nature of it. It’s either on or off, and it brings a plethora of validation that may be scary to introduce to a production environment. (Request desync’ing should be even more scary, of course, but that may not be an obvious risk to some users.)

For example, maybe I don’t want to enforce ASCII-only validation, but I do want to enforce some other level of header validation. Today, my only option is to outright disable validateHeaders, and then manually perform my own validation for the things I care about.

If, instead, validation was more configurable… something along the lines of a builder or CodecOption (similar in vein to ChannelOption), we might see less users opting to take the nuclear route of disabling everything.

That said, short of a total interface overhaul, I think Netty should make it impossible for you to unwittingly make yourself vulnerable to request desync’ing. Any validation that is covered under validateHeaders today that may open yourself up to this vulnerability should be changed to be the default behavior and not influenced by validateHeaders.

It would be ideal if we could identify the narrowest-possible subset of the existing validation that encompasses this vulnerability and enforce only that.