s3: introduce timeout config option

Bug Report

Description

I am trying to push files with approximately 1GB per file to an internal s3 storage. For other (smaller) files this works perfectly fine, however with the 1GB files I am experiencing a timeout, while the logs from the storage do not show any error and keep uploading even after the timeout.

Reproduce

adding 5 1GB files to the repository and then using dvc push -v -j 1

Expected

The files should be pushed to the remote.

Environment information

Microsoft Windows 10 Enterprise Ver. 10.0.18363 Build 18363

Output of dvc doctor:

$ dvc doctor

DVC version: 2.4.1 (pip)
---------------------------------
Platform: Python 3.7.4 on Windows-10-10.0.18362-SP0
Supports: http, https, s3, ssh
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: s3
Workspace directory: NTFS on C:\
Repo: dvc, git

Additional Information (if any):

2021-06-28 17:29:15,903 DEBUG: Uploading '.dvc\cache\a2\2fe6fed4cf9e93cd9938b6cb3dc911' to 's3://mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911'
2021-06-28 17:34:59,468 ERROR: failed to upload '.dvc\cache\a2\2fe6fed4cf9e93cd9938b6cb3dc911' to 's3://mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911' - 'NoneType' object is not subscriptable
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "c:\util\anaconda3\lib\http\client.py", line 1336, in getresponse
    response.begin()
  File "c:\util\anaconda3\lib\http\client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "c:\util\anaconda3\lib\http\client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "c:\util\anaconda3\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "c:\util\anaconda3\lib\ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "c:\util\anaconda3\lib\ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\botocore\httpsession.py", line 323, in send
    chunked=self._chunked(request.headers),
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "c:\util\anaconda3\lib\site-packages\urllib3\util\retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "c:\util\anaconda3\lib\site-packages\urllib3\packages\six.py", line 735, in reraise
    raise value
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 337, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='myhost', port=443): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 156, in wrapper
    func(*args, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 214, in _upload
    Config=self._transfer_config,
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\inject.py", line 280, in object_upload_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\transfer.py", line 279, in upload_file
    future.result()
  File "c:\util\anaconda3\lib\site-packages\s3transfer\futures.py", line 106, in result
    return self._coordinator.result()
  File "c:\util\anaconda3\lib\site-packages\s3transfer\futures.py", line 265, in result
    raise self._exception
  File "c:\util\anaconda3\lib\site-packages\s3transfer\tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "c:\util\anaconda3\lib\site-packages\s3transfer\tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "c:\util\anaconda3\lib\site-packages\s3transfer\upload.py", line 722, in _main
    Body=body, **extra_args)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
    success_response, exception):
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 256, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "c:\util\anaconda3\lib\site-packages\botocore\httpsession.py", line 349, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://myhost/mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911?uploadId=2~pIdriVwtHnYe2HUsrEJrSn0wQy7HSW8&partNumber=1"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\dvc\remote\base.py", line 36, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\base.py", line 235, in upload
    no_progress_bar=no_progress_bar,
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 160, in wrapper
    raise translate_boto_error(exc)
  File "c:\util\anaconda3\lib\site-packages\s3fs\errors.py", line 142, in translate_boto_error
    code = error.response["Error"].get("Code")
TypeError: 'NoneType' object is not subscriptable

Any hints would be greatly appreciated.

[JenniferHem]

@efiop so far 1GB is the largest filesize I am trying to upload, I believe it will not get larger than this.

I agree that 400 is much more than should be needed as a timeout, so I guess it would not be the useful default to set, however it might be helpful for users to be able to define a timeout for cases where bad Internet + a VPN is making life hard. I will try to set different timeouts, however I will not manage to do this within this week.

[JenniferHem]

@efiop Hi sorry I have to internally figure things out, that is why I am taking a bit of time. So we have a GUI which uses boto3 for the upload. However, currently I am seeing no Error in the GUI but using the CLI via boto3 in a script I again see the timeout. So I am investigating on our side again. I will report back here with more information