SSL3_WRITE_PENDING error from urllib3.contrib.pyopenssl#sendall

I’m writing a program that makes somewhat large (e.g. tens to hundreds of kilobytes) POST requests over HTTPS using requests. Such POSTs fail due to SSL3_WRITE_PENDING errors from OpenSSL. In short, SSL3_WRITE_PENDING occurs when an incomplete write operation (i.e. a write that resulted in a WantWriteError) is retried with arguments not exactly equal to those passed in the initial call (see here and here).

Stracktrace from my call into requests down through urllib3:

File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 511, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python2.7/httplib.py", line 1053, in request
self._send_request(method, url, body, headers)
File "/usr/local/lib/python2.7/httplib.py", line 1093, in _send_request
self.endheaders(body)
File "/usr/local/lib/python2.7/httplib.py", line 1049, in endheaders
self._send_output(message_body)
File "/usr/local/lib/python2.7/httplib.py", line 893, in _send_output
self.send(msg)
File "/usr/local/lib/python2.7/httplib.py", line 869, in send
self.sock.sendall(data)
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 220, in sendall
sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 206, in _send_until_done
return self.connection.send(data)
File "/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1271, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1187, in _raise_ssl_error
_raise_current_error()
File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
raise exception_type(errors)
Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

A bit of digging revealed #412 and #626 as issues that relate to the relevant code in urllib3. WrappedSocket.sendall proxies to WrappedSocket._send_until_done, which invokes OpenSSL.SSL.send in a loop. The conditions for loop termination are a) the write succeeds or b) a timeout expires while waiting for the socket to become writable in the case of a WantWriteError.

I think that this issue boils down to:

1. WrappedSocket.sendall converts the data to be sent to a memoryview.
2. WrappedSocket._send_until_done invokes OpenSSL.SSL.send in a loop according to the conditions above.
3. OpenSSL.SSL.send calls the memoryview.tobytes() and passes the bytestring result in the OpenSSL write call.
4. In the WantWriteError case, 2 and 3 are repeated and a new bytestring is passed to the OpenSSL write, resulting in a SSL3_WRITE_PENDING error.

I was able to get around this issue by patching urllib3/contrib/pyopenssl.py to enforce that a single bytestring is used for all calls to OpenSSL.SSL.send: https://gist.github.com/evnm/af92092c6a7776e08339

Please advise on whether this is a good way to fix the issue. I tried adding a test case to test/with_dummyserver/test_https.py, but haven’t figured out how to tickle the specific issue I’ve run into.

Relevant versions in use:

• Python 2.7.10
• requests 2.8.1
• pyOpenSSL 0.15.1
• urllib3 1.12

[kengruven]

I’ll see if I can isolate it, and provide a simple repro.

Things we (think we) know so far:

• We’re using requests here indirectly, to connect to ElasticSearch via the elasticsearch Python library, with connection_class=elasticsearch.connection.RequestsHttpConnection
• It still connects, and can run queries just fine
• Sometimes, adding data works. The problem only seems to occur when adding a lot of data to ES, which we do via elasticsearch.helpers.bulk
• We call bulk() with the defaults, which are up to 500 documents at once (we probably hit this sometimes) and 100MB (we never hit this)
• It started happening when we upgraded requests from 2.8.1 to 2.9.1 (we also used 2.3.0 for a while before upgrading to 2.8.1), but a few other things changed around the same time

[cristianocca]

The issue seems to come from requests_aws4auth:

https://github.com/sam-washington/requests-aws4auth/issues/24