ReadTimeoutError - how to change the timeout and avoid this error

Hi, I had this script running fine in python2 every day loading the same amount of data, but today got and exception. The script is downloading and extracting a 1.7GB gzip file from S3 (10.4GB uncompressed).

[06:20:05] [INFO] [dku.utils]  -   File "<string>", line 135, in <module>
[06:20:05] [INFO] [dku.utils]  -   File "/usr/lib64/python2.7/gzip.py", line 455, in readline
[06:20:05] [INFO] [dku.utils]  -     c = self.read(readsize)
[06:20:05] [INFO] [dku.utils]  -   File "/usr/lib64/python2.7/gzip.py", line 261, in read
[06:20:05] [INFO] [dku.utils]  -     self._read(readsize)
[06:20:05] [INFO] [dku.utils]  -   File "/usr/lib64/python2.7/gzip.py", line 301, in _read
[06:20:05] [INFO] [dku.utils]  -     buf = self.fileobj.read(size)
[06:20:05] [INFO] [dku.utils]  -   File "/data/data1/dss/acode-envs/python/py2/env/lib/python2.7/site-packages/smart_open/s3.py", line 262, in read
[06:20:05] [INFO] [dku.utils]  -     self._fill_buffer(size)
[06:20:05] [INFO] [dku.utils]  -   File "/data/data1/dss/acode-envs/python/py2/env/lib/python2.7/site-packages/smart_open/s3.py", line 320, in _fill_buffer
[06:20:05] [INFO] [dku.utils]  -     bytes_read = self._buffer.fill(self._raw_reader)
[06:20:05] [INFO] [dku.utils]  -   File "/data/data1/dss/acode-envs/python/py2/env/lib/python2.7/site-packages/smart_open/bytebuffer.py", line 146, in fill
[06:20:05] [INFO] [dku.utils]  -     new_bytes = source.read(size)
[06:20:05] [INFO] [dku.utils]  -   File "/data/data1/dss/acode-envs/python/py2/env/lib/python2.7/site-packages/smart_open/s3.py", line 188, in read
[06:20:05] [INFO] [dku.utils]  -     binary = self._body.read(size)
[06:20:05] [INFO] [dku.utils]  -   File "/data/data1/dss/acode-envs/python/py2/env/lib/python2.7/site-packages/botocore/response.py", line 81, in read
[06:20:05] [INFO] [dku.utils]  -     raise ReadTimeoutError(endpoint_url=e.url, error=e)
[06:20:05] [INFO] [dku.utils]  - ReadTimeoutError: Read timeout on endpoint URL: "None"

This is not the same issue as https://github.com/RaRe-Technologies/smart_open/issues/312, because I can’t repeat this error, and there is no problem with encoding, the data was loaded fine in the next retry.

            with smart_open.smart_open( uri ) as fin:
                cnt=0
                for line in fin:
                    if cnt != 0 or not removeHeader:
                        f.write(line)
                    if cnt == 0:
                        header = line.strip()
                    cnt+=1

[koshy1123]

We’ve gotten around this by using resource_kwargs param within transport_params like so:

smart_open.open(
f's3://{bucket}/{key}', 
transport_params={
'session': so_session,
'resource_kwargs': {'config': so_config}})

[funes79]

Thanks @koshy1123 as I said unfortunately I can’t repeat the error, but thanks for the help, will use this solution.