Dependency Import Broken starting with v2022.08.31

Issue description

Starting with 2022.08.31 dependency installs have started breaking for a single package we install. This was working for all version prior to 2022.08.31, and has as a result of not pinning pipenv, broken all our pipelines.

Package: yajl-py

Expected result

pipenv install
Installing dependencies from Pipfile.lock (3553a9)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:02
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Actual result

/usr/local/Cellar/pipenv/2022.8.31/bin/pipenv install --verbose
Installing dependencies from Pipfile.lock (3553a9)...
Writing supplied requirement line to temporary file: 'six==1.16.0 --hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254'
Writing supplied requirement line to temporary file: 'yajl-py==2.1.2 --hash=sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a'
Install Phase: Standard Requirements
Preparing Installation of 'six'
Preparing Installation of 'yajl-py'
$ /Users/mhaber/.local/share/virtualenvs/recreate-6qyGiqPQ/bin/python /usr/local/Cellar/pipenv/2022.8.31/libexec/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pipenv-oek2o8b_-requirements/pipenv-z1q_iz95-hashed-reqs.txt
Using source directory: '/Users/mhaber/.local/share/virtualenvs/recreate-6qyGiqPQ/src'
Collecting six==1.16.0

  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)

Collecting yajl-py==2.1.2

  Using cached yajl-py-2.1.2.tar.gz (7.8 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'error'

error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-uclw58aa/yajl-py_ce1bcc49c1db47138c27f20bd2f506e0/setup.py", line 4, in <module>
          from yajl import __version__ as version
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-uclw58aa/yajl-py_ce1bcc49c1db47138c27f20bd2f506e0/yajl/__init__.py", line 18, in <module>
          from .yajl_parse import (
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-uclw58aa/yajl-py_ce1bcc49c1db47138c27f20bd2f506e0/yajl/yajl_parse.py", line 6, in <module>
          import six
      ModuleNotFoundError: No module named 'six'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
An error occurred while installing six==1.16.0 --hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254! Will try again.
An error occurred while installing yajl-py==2.1.2 --hash=sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a! Will try again.
Installing initially failed dependencies...
Writing supplied requirement line to temporary file: 'six==1.16.0 --hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254'
Writing supplied requirement line to temporary file: 'yajl-py==2.1.2 --hash=sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a'
Install Phase: Standard Requirements
Preparing Installation of 'six'
Preparing Installation of 'yajl-py'
$ /Users/mhaber/.local/share/virtualenvs/recreate-6qyGiqPQ/bin/python /usr/local/Cellar/pipenv/2022.8.31/libexec/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-use-pep517 --no-deps --exists-action=i -r /var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pipenv-oek2o8b_-requirements/pipenv-i94wdm_d-hashed-reqs.txt
Using source directory: '/Users/mhaber/.local/share/virtualenvs/recreate-6qyGiqPQ/src'
Collecting six==1.16.0

  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)

Collecting yajl-py==2.1.2

  Using cached yajl-py-2.1.2.tar.gz (7.8 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'error'

error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/setup.py", line 4, in <module>
          from yajl import __version__ as version
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/yajl/__init__.py", line 18, in <module>
          from .yajl_parse import (
        File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/yajl/yajl_parse.py", line 6, in <module>
          import six
      ModuleNotFoundError: No module named 'six'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   × python setup.py egg_info did not run successfully.
[pipenv.exceptions.InstallError]:   │ exit code: 1
[pipenv.exceptions.InstallError]:   ╰─> [10 lines of output]
[pipenv.exceptions.InstallError]:       Traceback (most recent call last):
[pipenv.exceptions.InstallError]:         File "<string>", line 2, in <module>
[pipenv.exceptions.InstallError]:         File "<pip-setuptools-caller>", line 34, in <module>
[pipenv.exceptions.InstallError]:         File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/setup.py", line 4, in <module>
[pipenv.exceptions.InstallError]:           from yajl import __version__ as version
[pipenv.exceptions.InstallError]:         File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/yajl/__init__.py", line 18, in <module>
[pipenv.exceptions.InstallError]:           from .yajl_parse import (
[pipenv.exceptions.InstallError]:         File "/private/var/folders/jt/ly5256g913b5j012ywxznvhw0000gn/T/pip-install-y8baa9cq/yajl-py_b1b1536d465d4579a36bc96ac0860feb/yajl/yajl_parse.py", line 6, in <module>
[pipenv.exceptions.InstallError]:           import six
[pipenv.exceptions.InstallError]:       ModuleNotFoundError: No module named 'six'
[pipenv.exceptions.InstallError]:       [end of output]
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]: error: metadata-generation-failed
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: × Encountered error while generating package metadata.
[pipenv.exceptions.InstallError]: ╰─> See above for output.
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: note: This is an issue with the package mentioned above, not pip.
[pipenv.exceptions.InstallError]: hint: See above for details.
ERROR: Couldn't install package: [Requirement(_name='six', vcs=None, req=NamedRequirement(name='six', version='==1.16.0', req=Requirement.parse('six==1.16.0'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=six, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===1.16.0, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=six==1.16.0)>), markers=None, _specifiers='==1.16.0', index='pypi', editable=False, hashes=frozenset({'sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926', 'sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=six, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===1.16.0, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=six==1.16.0)>, _ireq=None), Requirement(_name='yajl-py', vcs=None, req=NamedRequirement(name='yajl-py', version='==2.1.2', req=Requirement.parse('yajl-py==2.1.2'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=yajl-py, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2.1.2, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=yajl-py==2.1.2)>), markers=None, _specifiers='==2.1.2', index='pypi', editable=False, hashes=frozenset({'sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=yajl-py, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2.1.2, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=yajl-py==2.1.2)>, _ireq=None)]
 Package installation failed...

Steps to replicate

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
six = "*"
yajl-py = "*"

[dev-packages]

[requires]
python_version = "3.10"

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "06410356d9d26d54cd9fc3753488ad44d9defcf79ac9c71c3af24135d83553a9"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "six": {
            "hashes": [
                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
            ],
            "index": "pypi",
            "version": "==1.16.0"
        },
        "yajl-py": {
            "hashes": [
                "sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a"
            ],
            "index": "pypi",
            "version": "==2.1.2"
        }
    },
    "develop": {}
}

Pipenv Support

$ pipenv --support | pbcopy

$ pipenv --support

Pipenv version: '2022.8.15'

Pipenv location: '/Users/mhaber/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pipenv'

setuptools version: '58.1.0'

Python location: '/Users/mhaber/.pyenv/versions/3.10.4/bin/python3.10'

OS Name: 'posix'

User pip version: '22.2.2'

user Python installations found:

• 3.10.6: /usr/local/bin/python3
• 3.10.4: /Users/mhaber/.pyenv/versions/3.10.4/bin/python3
• 3.10.4: /Users/mhaber/.pyenv/versions/3.10.4/bin/python3
• 3.10.4: /Users/mhaber/.pyenv/versions/3.10.4/bin/python
• 3.9.13: /Users/mhaber/.pyenv/versions/3.9.13/bin/python3
• 3.8.11: /Users/mhaber/.pyenv/versions/3.8.11/bin/python3
• 3.8.9: /usr/bin/python3
• 3.7.11: /Users/mhaber/.pyenv/versions/3.7.11/bin/python3
• 2.7.16: /usr/bin/python
• 2.7.16: /usr/bin/python2
• 2.7.16: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '20.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 20.6.0: Tue Apr 19 21:04:45 PDT '
                     '2022; root:xnu-7195.141.29~1/RELEASE_X86_64',
 'python_full_version': '3.10.4',
 'python_version': '3.10',
 'sys_platform': 'darwin'}

System environment variables:

• NVM_INC
• TERM_PROGRAM
• PYENV_ROOT
• NVM_CD_FLAGS
• TERM
• SHELL
• TMPDIR
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• PYENV_VERSION
• NVM_DIR
• USER
• SSH_AUTH_SOCK
• PYENV_DIR
• PATH
• GITHUB_OAUTH_TOKEN
• LaunchInstanceID
• __CFBundleIdentifier
• PWD
• LANG
• PYENV_HOOK_PATH
• XPC_FLAGS
• XPC_SERVICE_NAME
• PYENV_SHELL
• HOME
• SHLVL
• LOGNAME
• NVM_BIN
• GITHUB_TOKEN
• SECURITYSESSIONID
• GH_TOKEN
• __CF_USER_TEXT_ENCODING
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/mhaber/.pyenv/versions/3.10.4/bin:/usr/local/Cellar/pyenv/2.3.2/libexec:/usr/local/Cellar/pyenv/2.3.2/plugins/python-build/bin:/Users/mhaber/.poetry/bin:/Users/mhaber/.nvm/versions/node/v16.13.1/bin:/Users/mhaber/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
• SHELL: /bin/zsh
• LANG: en_CA.UTF-8
• PWD: /Users/mhaber/recreate

---

Contents of Pipfile (‘/Users/mhaber/recreate/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
six = "*"
yajl-py = "*"

[dev-packages]

[requires]
python_version = "3.10"

Contents of Pipfile.lock (‘/Users/mhaber/recreate/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "06410356d9d26d54cd9fc3753488ad44d9defcf79ac9c71c3af24135d83553a9"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "six": {
            "hashes": [
                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
            ],
            "index": "pypi",
            "version": "==1.16.0"
        },
        "yajl-py": {
            "hashes": [
                "sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a"
            ],
            "index": "pypi",
            "version": "==2.1.2"
        }
    },
    "develop": {}
}

[mattthaber]

I was able to sync with your lockfile on 2022.8.30 and I see where the problem is – apparently the setup.py of yajl-py is importing files from itself and causing itself to import from six during the install phase. You were getting lucky prior because the pip installs happened serially and so six would get installed before yajl-py. Now due to #5301 the same pip process is doing all of the packages in the same install basically which cut the time of the install in half on the benchmark: https://lincolnloop.github.io/python-package-manager-shootout/ (prior version has already fallen off the graph).

Anyway, what I can suggest as a work around is to pipenv run pip install six before running pipenv sync in order to work around the complication of the yajl-py setup.py. ~FWIW locking that Pipfile is problematic on all versions that I have tried.~ The locking was problematic because I needed to install libyajl-dev

ahhh ok makes sense then. we will resolve it on our end to get around then.

i really appreciate the quick response and help investigating!

[matteius]

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-5319$ pipenv run pip install six
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0
matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-5319$ pipenv sync --verbose
Installing dependencies from Pipfile.lock (3553a9)...
Writing supplied requirement line to temporary file: 'yajl-py==2.1.2 --hash=sha256:086d5031aac0dc456476e79ce32615455b5e7bc0c29bcece595e65a41b6a551a'
Install Phase: Standard Requirements
Preparing Installation of 'yajl-py'
$ /home/matteius/.virtualenvs/pipenv-5319-7Y5NzgWN/bin/python /home/matteius/.local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /tmp/pipenv-43h3bi7y-requirements/pipenv-7g1mgb8s-hashed-reqs.txt
Using source directory: '/home/matteius/.virtualenvs/pipenv-5319-7Y5NzgWN/src'
Collecting yajl-py==2.1.2

  Using cached yajl-py-2.1.2.tar.gz (7.8 kB)

  Preparing metadata (setup.py): started

  Preparing metadata (setup.py): finished with status 'done'

Building wheels for collected packages: yajl-py

  Building wheel for yajl-py (setup.py): started

  Building wheel for yajl-py (setup.py): finished with status 'done'

  Created wheel for yajl-py: filename=yajl_py-2.1.2-py3-none-any.whl size=8066 sha256=0452a443a0fef9ecb62405520fd702e6e757bb9654dd78ef84bab6733b8c0c04

  Stored in directory: /home/matteius/.cache/pipenv/wheels/26/a4/14/5c3878079d30acb5365fd75770a478e7cc3edf0f4910fccc5f

Successfully built yajl-py

Installing collected packages: yajl-py

Successfully installed yajl-py-2.1.2


To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-5319$ pipenv --version
pipenv, version 2022.8.31