urllib3.exceptions.ProtocolError not catched, even with retry=adaptive

[Not a contribution]

From time to time, when downloading large files, the download is aborted with:

Traceback (most recent call last):
     ...
     File ".../python3.7/site-packages/boto3/s3/inject.py", line 172, in download_file
       extra_args=ExtraArgs, callback=Callback)
     File ".../python3.7/site-packages/boto3/s3/transfer.py", line 307, in download_file
       future.result()
     File ".../python3.7/site-packages/s3transfer/futures.py", line 106, in result
       return self._coordinator.result()
     File ".../python3.7/site-packages/s3transfer/futures.py", line 265, in result
       raise self._exception
     File ".../python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
       return self._execute_main(kwargs)
     File ".../python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
       return_value = self._main(**kwargs)
     File ".../python3.7/site-packages/s3transfer/download.py", line 521, in _main
       for chunk in chunks:
     File ".../python3.7/site-packages/s3transfer/download.py", line 649, in __next__
       chunk = self._body.read(self._chunksize)
     File ".../python3.7/site-packages/s3transfer/utils.py", line 570, in read
       value = self._stream.read(*args, **kwargs)
     File ".../python3.7/site-packages/botocore/response.py", line 77, in read
       chunk = self._raw_stream.read(amt)
     File ".../python3.7/site-packages/urllib3/response.py", line 540, in read
       raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
     File ".../python3.7/contextlib.py", line 130, in __exit__
       self.gen.throw(type, value, traceback)
     File ".../python3.7/site-packages/urllib3/response.py", line 454, in _error_catcher
       raise ProtocolError("Connection broken: %r" % e, e)
   urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(54, 'Connection reset by peer')", ConnectionResetError(54, 'Connection reset by peer'))

even though adaptive retry mode is enabled.

I can see the debug messages: standard - Not retrying request., but no message like standard - Max attempts of 4 reached. which I usually see when downloading files and for example the internet connection drops.

It seems that this code path (an exception in self._raw_stream.read(amt) in __next__ in for chunk in chunks: in s3transfer.download.GetObjectTask._main) doesn’t do any exception handling, as opposed to an exception occurring in s3transfer.download.GetObjectTask._main earlier on during response = client.get_object, which handles exceptions down in the stack in botocore.endpoint.Endpoint._send_request.

Is this on purpose, or can/should this exception also be retried?

[nateprewitt]

Hi @jwmx, I’ve added boto/botocore#2573 with a preliminary proposal to start handling this with a BotoCoreError subclassed error. You should be able to track progress on this there.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.