"OverflowError: string longer than 2147483647 bytes" when trying requests.put
See original GitHub issueHi,
I’m trying to upload a file that weight about 3GB and I’m getting the following error: “OverflowError: string longer than 2147483647 bytes”
If I understand correctly it seems like there’s a 2GB limit? didnt manage to find any reference to such limiation or how to bypass it (if possible).
The code i’m using is:
datafile = 'someHugeFile'
with open(datafile, 'rb') as myfile:
args = myfile.read()
resp = requests.put(url, data=args, verify=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
return request('put', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 327, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python2.7/httplib.py", line 995, in request
self._send_request(method, url, body, headers)
File "/usr/local/lib/python2.7/httplib.py", line 1029, in _send_request
self.endheaders(body)
File "/usr/local/lib/python2.7/httplib.py", line 991, in endheaders
self._send_output(message_body)
File "/usr/local/lib/python2.7/httplib.py", line 844, in _send_output
self.send(msg)
File "/usr/local/lib/python2.7/httplib.py", line 820, in send
self.sock.sendall(data)
File "/usr/local/lib/python2.7/ssl.py", line 234, in sendall
v = self.send(data[count:])
File "/usr/local/lib/python2.7/ssl.py", line 203, in send
v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
For smaller files this code works fine for me.
Issue Analytics
- State:
- Created 8 years ago
- Comments:33 (15 by maintainers)
Top Results From Across the Web
how to upload chunks of a string longer than 2147483647 ...
Your question has been asked on the requests bug tracker; their suggestion is to use streaming upload. If that doesn't work, you might...
Read more >ERROR - string longer than 2147483647 bytes with ...
I have a DAG that downloads a file, parses it to json, and then uses a GoogleCloudStorageHook to upload back to GCS. The...
Read more >Bug #1671755 “OverflowError 2147483647 bytes”
Hi all, We have a lot data to backup. After uploading backup Duplicity return error: Attempt 1 failed. OverflowError: string longer than ......
Read more >Permafailing [Sym] OverflowError: string longer than ...
Permafailing [Sym] OverflowError: string longer than 2147483647 bytes ... [task 2021-12-22T14:07:07.035Z] INFO:upload-symbols:Attempt 1 of 7.
Read more >"OverflowError: string longer than 2147483647 bytes" when ...
"OverflowError: string longer than 2147483647 bytes" when trying requests.put. requests. 12 August 2015 Posted by EB123. Hi,. I'm trying to upload a file ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I’m having the same basic issue as @gjedeer and see the same behavior as @cmbasnett (that wrapping in BytesIO is not a solution). I’m trying to use a file object to upload something larger than 2GB using a TLS encrypted post. Specifically I’m trying to use a presigned url to upload a file to S3. It appears that the underlying ssl library in python doesn’t like files over 2GB. Is there an accepted workaround to this? Stack trace:
Basic code:
@SpoonMeiser Wrapping the file contents in a
BytesIO
does not fix the problem on Python 3.6, still raises the exact same error.