AIX: `pip download package` does not work since 21.2

Description

Recently got started with needing to update a number of packages - and ran into a problem that I could not download ansible-base (so I suspect it is an issue with any packages that are not pure Python).

So, rolled back pip to a much older version (20.2.4) and all was okay.

In increments I updated pip and 21.1.3 was the last version that worked as expected (as far as download is concerned, have not tried anything else).

Where I think the regression occurred

aixtools@x064:[/data/prj/python/git/pip]git diff 21.1.3 21.2.1 -- ./src/pip/_internal/utils/unpacking.py
diff --git a/src/pip/_internal/utils/unpacking.py b/src/pip/_internal/utils/unpacking.py
index 44ac47535..bffb3cd65 100644
--- a/src/pip/_internal/utils/unpacking.py
+++ b/src/pip/_internal/utils/unpacking.py
@@ -178,7 +178,7 @@ def untar_file(filename, location):
             filename,
         )
         mode = "r:*"
-    tar = tarfile.open(filename, mode)
+    tar = tarfile.open(filename, mode, encoding="utf-8")
     try:
         leading = has_leading_dir([member.name for member in tar.getmembers()])
         for member in tar.getmembers():

Expected behavior

• Note: I get the same problem as above when trying to download ansible-base==2.10.16, but the ansible I have been using for two years is based on 2.10.1 - so I tried that version again.
• I saw you are withdrawing support for py36 - so, please note, this is not a request for py36 support. I first saw this on Python3-9 which is what I wanted to update. py36 is only being used because that is known to be working - and I was looking for when the regression appeared.

(py360) aixtools@x064:[/home/aixtools/download/py360]pip download ansible-base==2.10.1
Collecting ansible-base==2.10.1
  Using cached ansible-base-2.10.1.tar.gz (6.0 MB)
Collecting jinja2
  Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting PyYAML
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting cryptography
  Using cached cryptography-36.0.1.tar.gz (572 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting packaging
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     |################################| 40 kB 44 kB/s
Collecting cffi>=1.12
  Using cached cffi-1.15.0.tar.gz (484 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1.tar.gz (18 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)
     |################################| 98 kB 286 kB/s
Saved ./ansible-base-2.10.1.tar.gz
Saved ./cryptography-36.0.1.tar.gz
Saved ./cffi-1.15.0.tar.gz
Saved ./Jinja2-3.0.3-py3-none-any.whl
Saved ./MarkupSafe-2.0.1.tar.gz
Saved ./packaging-21.3-py3-none-any.whl
Saved ./pyparsing-3.0.7-py3-none-any.whl
Saved ./pycparser-2.21-py2.py3-none-any.whl
Saved ./PyYAML-6.0.tar.gz

pip version

21.3.1, 21.2.4, 21.2

Python version

3.6, 3.9

OS

AIX

How to Reproduce

1. System Admin Installs Python3-9 (or 3-6) including virtualenv
2. Following steps as regular (no root powers)
3. virtualenv py360
4. mkdir -p downloads/py360
5. . py360/bin/activate
6. cd downloads/py360
7. pip3 download ansible-base==2.10.1

Output

- Example:

(py360) aixtools@x064:[/home/aixtools/download/py360]pip install pip --upgrade
Requirement already satisfied: pip in /home/aixtools/py360/lib/python3.6/site-packages (21.1.3)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed pip-21.3.1
(py360) aixtools@x064:[/home/aixtools/download/py360]pip download ansible-base==2.10.1
Collecting ansible-base==2.10.1
  File was already downloaded /home/aixtools/download/py360/ansible-base-2.10.1.tar.gz
ERROR: Exception:
Traceback (most recent call last):
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/commands/download.py", line 128, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 93, in resolve
    collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 349, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    version=version,
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 287, in __init__
    version=version,
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 528, in _prepare_linked_requirement
    link, req.source_dir, self._download, self.download_dir, hashes
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 223, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 247, in unpack_file
    untar_file(filename, location)
  File "/home/aixtools/py360/lib/python3.6/site-packages/pip/_internal/utils/unpacking.py", line 218, in untar_file
    with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 138-141: ordinal not in range(256)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

I pretty much agree with @pfmoore here, so I’ve gone ahead and applied the R: no action label here; and am going to close this.

AIX is not a platform that pip supports anyway (https://pip.pypa.io/en/stable/installation/#compatibility). I’ll take this as a good reminder to update our docs to note that pip working on unsupported platforms (eg: AIX) is considered incidental and “it runs != it is supported”.

[aixtools]

Configuring an AIX system to use UTF-8 as the filesystem encoding (like most Linux systems do these days) would remove the issue, but it’s a global change that would have much wider implications. Asking the ansible project to avoid Unicode filenames might be a practical workaround, too.

• IMHO: unlikely that IBM would make this change and risk breaking binary compatability

It’s unlikely pip will be modified to handle this case, it’s basically far too rare, and there’s no reasonable behaviour that would be guaranteed safe. We don’t want to get into those sort of heuristics - we follow the behaviour of Python’s open function here as that’s well-defined and well-known.

• understood: Windows is a supported system, and AIX is not. So even though earlier changes anticipated problems eventually AIX is still in a limbo area - not quite supported.

I’m inclined to close this as “no action” / “out of scope” (basically “won’t fix”, but we don’t have a label for that) because I don’t think there’s any change to pip that we’d be willing to make to handle this.

• AS Python is part of AIX 7.3 base - I hope you (pypa, Python) won’t actively oppose seeing AIX as a supported platform. I know I tried for years to meet as many of the demands as I could - but I was not IBM enough to satisify anyone, and bored with with always hearing about AIX’s limbo status - I stopped working on platform issues.
• As always, I wish you the best and thank you for what you have done ( 😉 just not for AIX)
• Also thanks for the expanded clarification.