Netty enforce HTTP proxy to support HTTP CONNECT method

Expected behavior

In case of HTTP protocol, the client request is sent as is, without sending the initial message with HTTP CONNECT method.

Background

CONNECT This specification reserves the method name CONNECT for use with a proxy that can dynamically switch to being a tunnel (e.g. SSL tunneling 44).

Source: https://tools.ietf.org/html/rfc2616#section-9.9

CONNECT The CONNECT method requests that the recipient establish a tunnel to the destination origin server identified by the request-target and, if successful, thereafter restrict its behavior to blind forwarding of packets, in both directions, until the tunnel is closed. Tunnels are commonly used to create an end-to-end virtual connection, through one or more proxies, which can then be secured using TLS (Transport Layer Security, RFC5246). Source: https://tools.ietf.org/html/rfc7231#section-4.3.6

Request

No every HTTP proxy supports being a transparent tunnelling proxy. Therefore, Netty as a library shall either react differently on HTTP and HTTPS protocols, or alternatively provide an option to control this behaviour.

This issue is related to:

• https://github.com/netty/netty/issues/3634
• https://github.com/reactor/reactor-netty/issues/159

Actual behaviour

No matter of the concrete HTTP protocol, HTTP or HTTPS, Netty’s HttpProxyHandler always sends HTTP CONNECT request as initial message.

Steps to reproduce

Use pure Netty (or Spring WebClient based on Netty) as plain HTTP client behind a plain HTTP proxy.

Minimal yet complete reproducer code (or URL to code)

N/A

Netty version

Up to 4.1.51.FINAL

JVM version (e.g. java -version)

N/A

OS version (e.g. uname -a)

N/A

[manolvalchev]

It turns out Netty’s ProxyHandler is written with the wrong assumption that every proxy (in general) offers tunneling option with a dedicated protocol-specific handshake based on a single request/response mechanism. In case no initial message is sent, and no response is received - it simply times out. Latter seems to be a bug as the impl allow skipping the initial message. Childs have no control over this behaviour.

As per specification, for HTTP protocol, there are two cases:

1. Protocol scheme is “http” - the client has two options - with or without tunneling - the most common option is without tunneling, notable examples are cURL, Jetty, Java UrlConnection, Apache HTTP Client, etc.
2. Protocol scheme is “https” - the client must use tunneling

If there is a NoTunnelingHttpProxyHandler or even NoTunnelingProxyHandler with just the connect() method implemented, and if that handler is used, for example, in the reactor-netty’s HttpClient pipeline as ProxyHandler, then the scenario would succeed as the only thing that the no tunneling proxy handler would be to connect to the proxy instead of the target host, specified in the HttpClient configuration, either as pure host or as URI. Proxy-Authorization and other HTTP headers could be handled the same way as in the current implementation of HttpProxyHandler.

I see the following options:

• introducing a switch flag stating whether the proxy handler’s behaviour shall be tunneling or not - each child can decide by its implementation - it could also be defined by the higher level user in case it’s exposed via constructor parameter, for example, in reactor-netty’s ProxyProvider class.
• pass somehow the protocol scheme to the protocol specific handler and let it decide whether to request tunneling or not - again, for the sake of backward compatibility, maybe a flag would be still required
• provide a dedicated impl in Netty for the non-tunneling HTTP proxy handler - ProxyHandler would need to change to an interface as now it’s an abstract class and a new impl would not be suitably used, e.g. ProxyProvider class.

[Kitanotori]

Just found a workaround. For Spring Boot users, the workaround is to use jetty-reactive-httpclient as WebClient’s client connector.

Example: https://www.baeldung.com/jetty-reactivestreams-http-client

Hopefully netty/reactor-netty can fix the issue so that there won’t be need for such workaround. Apparently I’m not the only one struggling with this issue.