Pip not retrying downloads

Description

For whatever reason we’re seeing timeouts when downloading from our private PyPi repository every so often, still running down the root cause on that, but pip doesn’t seem to be retrying like I thought it should be. It’s dying with:

pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='jfrog-prod-use1-shared-virginia-main.s3.amazonaws.com', port=443): Read timed out.

Expected behavior

Pip should retry the download operation a few times, 5 by default according to the documentation.

pip version

21.3.1

Python version

3.7

OS

Ubuntu

How to Reproduce

I’m seeing this in various build logs across our build farms, don’t have an easily reproducible test case at the moment unfortunately.

Output

2022-05-25T12:25:57.6306129Z   Downloading https://internal-stuff.url.systems/packages/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl (98 kB)
2022-05-25T12:25:58.3305288Z Collecting tenacity
2022-05-25T12:25:58.5194483Z   Downloading https://internal-stuff.url.systems/packages/packages/f2/a5/f86bc8d67c979020438c8559cc70cfe3a1643fd160d35e09c9cca6a09189/tenacity-8.0.1-py3-none-any.whl (24 kB)
2022-05-25T12:25:59.2982655Z Collecting p4python
2022-05-25T12:25:59.7298319Z   Downloading https://internal-stuff.url.systems/p4python/2019.1.1858212/p4python-2019.1.1858212-cp37-cp37m-manylinux2010_x86_64.whl (5.3 MB)
2022-05-25T12:26:32.7190243Z ERROR: Exception:
2022-05-25T12:26:32.7191013Z Traceback (most recent call last):
2022-05-25T12:26:32.7196379Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
2022-05-25T12:26:32.7196894Z     yield
2022-05-25T12:26:32.7197438Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
2022-05-25T12:26:32.7197818Z     data = self._fp.read(amt) if not fp_closed else b""
2022-05-25T12:26:32.7198385Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
2022-05-25T12:26:32.7198725Z     data = self.__fp.read(amt)
2022-05-25T12:26:32.7198999Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/http/client.py", line 465, in read
2022-05-25T12:26:32.7199285Z     n = self.readinto(b)
2022-05-25T12:26:32.7199555Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/http/client.py", line 509, in readinto
2022-05-25T12:26:32.7199846Z     n = self.fp.readinto(b)
2022-05-25T12:26:32.7200114Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/socket.py", line 589, in readinto
2022-05-25T12:26:32.7200392Z     return self._sock.recv_into(b)
2022-05-25T12:26:32.7200695Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/ssl.py", line 1071, in recv_into
2022-05-25T12:26:32.7200978Z     return self.read(nbytes, buffer)
2022-05-25T12:26:32.7201249Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/ssl.py", line 929, in read
2022-05-25T12:26:32.7201543Z     return self._sslobj.read(len, buffer)
2022-05-25T12:26:32.7201773Z socket.timeout: The read operation timed out
2022-05-25T12:26:32.7201883Z 
2022-05-25T12:26:32.7202105Z During handling of the above exception, another exception occurred:
2022-05-25T12:26:32.7202244Z 
2022-05-25T12:26:32.7202410Z Traceback (most recent call last):
2022-05-25T12:26:32.7202973Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
2022-05-25T12:26:32.7203322Z     status = run_func(*args)
2022-05-25T12:26:32.7203930Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
2022-05-25T12:26:32.7204303Z     return func(self, options, args)
2022-05-25T12:26:32.7204820Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 339, in run
2022-05-25T12:26:32.7205201Z     reqs, check_supported_wheels=not options.target_dir
2022-05-25T12:26:32.7205765Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 93, in resolve
2022-05-25T12:26:32.7206173Z     collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
2022-05-25T12:26:32.7206757Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
2022-05-25T12:26:32.7207140Z     state = resolution.resolve(requirements, max_rounds=max_rounds)
2022-05-25T12:26:32.7207877Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 374, in resolve
2022-05-25T12:26:32.7208621Z     failure_causes = self._attempt_to_pin_criterion(name)
2022-05-25T12:26:32.7209242Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 214, in _attempt_to_pin_criterion
2022-05-25T12:26:32.7209634Z     criteria = self._get_updated_criteria(candidate)
2022-05-25T12:26:32.7210265Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 205, in _get_updated_criteria
2022-05-25T12:26:32.7210670Z     self._add_to_criteria(criteria, requirement, parent=candidate)
2022-05-25T12:26:32.7211246Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
2022-05-25T12:26:32.7211623Z     if not criterion.candidates:
2022-05-25T12:26:32.7212140Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
2022-05-25T12:26:32.7212501Z     return bool(self._sequence)
2022-05-25T12:26:32.7213053Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
2022-05-25T12:26:32.7213407Z     return any(self)
2022-05-25T12:26:32.7213974Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
2022-05-25T12:26:32.7214386Z     return (c for c in iterator if id(c) not in self._incompatible_ids)
2022-05-25T12:26:32.7215002Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
2022-05-25T12:26:32.7215358Z     candidate = func()
2022-05-25T12:26:32.7215917Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
2022-05-25T12:26:32.7216303Z     version=version,
2022-05-25T12:26:32.7216829Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 287, in __init__
2022-05-25T12:26:32.7217196Z     version=version,
2022-05-25T12:26:32.7217726Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
2022-05-25T12:26:32.7218083Z     self.dist = self._prepare()
2022-05-25T12:26:32.7218635Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
2022-05-25T12:26:32.7219006Z     dist = self._prepare_distribution()
2022-05-25T12:26:32.7219603Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
2022-05-25T12:26:32.7220050Z     return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
2022-05-25T12:26:32.7220664Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
2022-05-25T12:26:32.7221104Z     return self._prepare_linked_requirement(req, parallel_builds)
2022-05-25T12:26:32.7221703Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 528, in _prepare_linked_requirement
2022-05-25T12:26:32.7222133Z     link, req.source_dir, self._download, self.download_dir, hashes
2022-05-25T12:26:32.7222696Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 217, in unpack_url
2022-05-25T12:26:32.7223101Z     hashes=hashes,
2022-05-25T12:26:32.7223654Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 94, in get_http_url
2022-05-25T12:26:32.7224076Z     from_path, content_type = download(link, temp_dir.path)
2022-05-25T12:26:32.7224640Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/network/download.py", line 145, in __call__
2022-05-25T12:26:32.7224970Z     for chunk in chunks:
2022-05-25T12:26:32.7225481Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/cli/progress_bars.py", line 144, in iter
2022-05-25T12:26:32.7225797Z     for x in it:
2022-05-25T12:26:32.7226287Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_internal/network/utils.py", line 87, in response_chunks
2022-05-25T12:26:32.7226639Z     decode_content=False,
2022-05-25T12:26:32.7227139Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
2022-05-25T12:26:32.7227514Z     data = self.read(amt=amt, decode_content=decode_content)
2022-05-25T12:26:32.7228066Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 541, in read
2022-05-25T12:26:32.7228446Z     raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
2022-05-25T12:26:32.7228796Z   File "/home/rfmibuild/.pyenv/versions/3.7.13/lib/python3.7/contextlib.py", line 130, in __exit__
2022-05-25T12:26:32.7229089Z     self.gen.throw(type, value, traceback)
2022-05-25T12:26:32.7229620Z   File "/home/rfmibuild/myagent/_work/_temp/checkout_venv/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
2022-05-25T12:26:32.7230018Z     raise ReadTimeoutError(self._pool, None, "Read timed out.")
2022-05-25T12:26:32.7230608Z pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='blah.s3.amazonaws.com', port=443): Read timed out.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

A ReadTimeoutError means your server is likely too slow to respond, and a retry is not likely to help (because it still needs the server to respond). You should probably do something with the server you’re working with, or try --timeout.

[pfmoore]

My (personal) opinion is that we should do nothing in pip and you should use the existing --timeout option. If it were a serious issue, we would have seen more people report it. And it’s not at all clear to me that the proposed change won’t be a problem for our other users, as @uranusjr mentioned. Sorry, I appreciate that this is a non-trivial problem for you, but we have to prioritise here - pip is a purely volunteer developed project with extremely limited resources.