TLS connection fails through HTTPS proxy after CONNECT tunnel is established

I set proxy by this code:

class HttpProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = 'https://127.0.0.1:8787'

It’s error is:

scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy 127.0.0.1:8787

Then I test the proxy by requests: resp = requests.get('https://......', proxies={'https': 'https://127.0.0.1:8787'}) It’ work!

So, what is it happen?

[redapple]

Alright, I think I figured this one out. What happens in that Scrapy’s TunnelingTCP4ClientEndpoint does not consume all of the proxy’s response to the CONNECT request (it only checks that the status code is 200), and because TunnelingTCP4ClientEndpoint.processProxyResponse() is called with chunks of the response, the remaining bytes on the transport are fed into the TLS layer, as if sent by the server as a response to the ClientHello, but these are plain ASCII bytes and therefore OpenSSL says “No way!”

Step 0: send CONNECT

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443

Step 1: receive first chunk:

'HTTP/1.1 200 OK\r\nKeep-Alive'

Step 2: Scrapy says: “Cool! the proxy is ready, let’s initiate the TLS connection.” a ClientHello is sent over the TCP connection…

Step 3: there are more bytes from the proxy where the HTTP 200 came from…

': timeout=38\r\nContent-Length: 0\r\n\r\n'

Step 4: OpenSSL is not happy with these bytes (there are not a ServerHello) and aborts the connection

Simple fix: add a small buffer when reading the initial response from the proxy and detect \r\n\r\n before starting the TLS negotiation.

Advanced fix: use some HTTP parsing state machine (Twisted’s?) to do this properly.

[aiportal]

Thanks very much. 👍