PipEnv fails installs with TypeError ('LegacySpecifier' not iterable)

PipEnv fails some installs with an exception, TypeError: 'LegacySpecifier' object is not iterable:

$ pip3 freeze | grep pipenv
pipenv==2020.11.15

Installation should work or show an actual error related to a package. It fails on a TypeError exception without any legible problem report related to the install:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/resolver.py", line 709, in resolve_packages
    requirements_dir=requirements_dir,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/resolver.py", line 692, in resolve
    req_dir=requirements_dir
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/utils.py", line 1403, in resolve_deps
    req_dir=req_dir,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/utils.py", line 1110, in actually_resolve_deps
    resolver.resolve_constraints()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/utils.py", line 862, in resolve_constraints
    marker = marker_from_specifier(candidate.requires_python)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 719, in marker_from_specifier
    for marker_segment in cleanup_pyspecs(spec):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 288, in cleanup_pyspecs
    for op_and_version_type, versions in _group_by_op(tuple(specs)):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 203, in _group_by_op
    specs = [_get_specs(x) for x in list(specs)]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 203, in <listcomp>
    specs = [_get_specs(x) for x in list(specs)]
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv/vendor/requirementslib/models/markers.py", line 187, in _get_specs
    for spec in set(specset):
TypeError: 'LegacySpecifier' object is not iterable

Verbose log is attached as a file.

I believe this is related to parsing markers, e.g. importlib-metadata>=3.6 ; python_version<='3.7'. However, I have not deep dived very far into the cause.

Reproduce:

rm -r dist *.egg* ; pip3 uninstall exotic ; rm -rf Pipfile* rm -rf ~/.local/share/virtualenvs/somelocalappname*
pipenv --verbose install

You could probably reproduce it by simply installing the EXOTIC product, which is a dependency of somelocalappname:

pipenv install exotic

IT WORKS WITH AN OLDER VERSION OF PIPENV: pip3 uninstall pipenv ; pip3 install pipenv==2018.11.26

---

someorgs-MacBook-Pro:somelocalappname someuser$ pipenv --support

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pipenv'

Python location: '/Library/Frameworks/Python.framework/Versions/3.7/bin/python3'

Python installations found:

• 3.7.3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
• 3.7.3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m
• 3.7.3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
• 3.7.3: /usr/local/bin/python3
• 3.7.3: /usr/local/bin/python3.7m
• 3.7.3: /usr/local/bin/python3.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '19.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST '
                     '2021; root:xnu-6153.141.16~1/RELEASE_X86_64',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_PROGRAM
• TERM
• SHELL
• TESTRAIL_KEY
• TMPDIR
• TERM_PROGRAM_VERSION
• OLDPWD
• TERM_SESSION_ID
• UID_GID
• USER
• TERM_SESSION_CLASS_ID
• SSH_AUTH_SOCK
• BASH_SILENCE_DEPRECATION_WARNING
• MAVEN_OPTS
• MAVEN_HOME
• PATH
• PWD
• JAVA_HOME
• LANG
• ARTIFACTORY_PSW
• TEST_PSW
• PYTHONHOME
• XPC_FLAGS
• TESTRAIL_USER
• XPC_SERVICE_NAME
• ARTIFACTORY_USR
• M2_HOME
• HOME
• SHLVL
• TEST_USR
• PYTHONPATH
• LOGNAME
• TESTRAIL_PSW
• TESTRAIL_USR
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenvspecific environment variables:

Debugspecific environment variables:

• PATH: /Library/Java/JavaVirtualMachines/jdk-1.8.0_232.jdk/Contents/Home/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Users/someuser/Library/Python/3.7/bin:/opt/maven/latest/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
• SHELL: /bin/zsh
• LANG: en_US.UTF-8
• PWD: /proj/survey-ws/source/somelocalappname

---

Contents of Pipfile (‘/System/Volumes/Data/proj/survey-ws/source/somelocalappname/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
astropy = ">=4.0.1.post1"
beautifulsoup4 = "~=4.9"
dawgie = "~=1.2"
exotic = "~=0.41.0"
filelock = "~=3.0"
importlib-metadata = ">=3.6"
json2html = "~=1.3"
jdcal = "~=1.4.1"
jplephem = ">=2.14"
lmfit = "~=1.0.1"
matplotlib = ">=3.2.2"
paramiko = ">=2.7.2"
pymc3 = "~=3.9"
pytz = ">=2019.3"
pytesseract = "~=0.2.9"
python-dateutil = "~=2.8.1"
requests = "~=2.23"
scipy = "~=1.4"
skyfield = "~=1.18"
statsmodels = ">=0.12.0"
tenacity = "~=5.1"
LDTk = "~=1.4.1"
TwitterAPI = "~=2.5"

[dev-packages]

[requires]
python_version = "3.7"

[jpl-jengelke]

Agreed that the version quotes provoked an issue in pipenv but it didn’t result in errors with pip. Installation fails were only experienced in pipenv. The Versioning PEPs are a bit amorphous as some indicate quotes and others don’t. Also, the Setup Tools documentation lacks detail for python_requires arguments in setup.py. Anyway, as removing the quotes will only help, the PR will integrated.

Just to clear some misunderstanding, the error manifested with installing the requirements file by itself, and the requirements file doesn’t reference EXOTIC. It does however contain markers, similar to importlib-metadata>=3.6 ; python_version<='3.7'. Installing only the requirements file into pipenv failed with the LegacySpecifier error.

There may have been a component of system configuration cruft confounding this. Now I’ve completely reset my pip and pipenv installations, including removing caches and pulling entirely new distributions. Without updating EXOTIC with the PR, the error is no longer reproducible.

Here’s how I reset the environment:

rm -r dist *.egg* build Pipfile*
pip3 uninstall <app> 
cd <app_dir> && git reset --hard HEAD 
pipenv --clear 
pip uninstall pipenv 
pip cache purge 
rm -rf <path_to_virtual_envs>/<app>-<hash>
pip install pipenv (or pip install <path_to_branch_of_pipenv>)
<now proceed with reinstall>

I think we can call this closed. I’m quite certain a year ago this was a problem with the older release. If any similar errors arise again in the context of a requirements file, I will raise a new issue to cleanly indicate that. Thanks for working on Pipenv which is ultimately a great tool.

[matteius]

@jpl-jengelke Yes that is the one, and fwiw I am trying it right now, just installed Cython first. Its still running … not sure how long it should take. I’ll edit this message hopefully soon. EDIT: Well I think I reproduced your bug, I was hopeful it would have been fixed as well.

(pipenv-4768) matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4681$ pipenv install exotic
Installing exotic...
Adding exotic to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (7ed8a8) out of date, updating to (0df0f7)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

Traceback (most recent call last):
  File "/home/matteius/pipenv/pipenv/resolver.py", line 766, in <module>
    main()
  File "/home/matteius/pipenv/pipenv/resolver.py", line 760, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/matteius/pipenv/pipenv/resolver.py", line 743, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/matteius/pipenv/pipenv/resolver.py", line 704, in resolve_packages
    results, resolver = resolve(
  File "/home/matteius/pipenv/pipenv/resolver.py", line 685, in resolve
    return resolve_deps(
  File "/home/matteius/pipenv/pipenv/utils.py", line 1376, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/matteius/pipenv/pipenv/utils.py", line 1107, in actually_resolve_deps
    resolver.resolve_constraints()
  File "/home/matteius/pipenv/pipenv/utils.py", line 900, in resolve_constraints
    marker = marker_from_specifier(requires_python)
  File "/home/matteius/pipenv/pipenv/vendor/requirementslib/models/markers.py", line 714, in marker_from_specifier
    for marker_segment in cleanup_pyspecs(spec):
  File "/home/matteius/pipenv/pipenv/vendor/requirementslib/models/markers.py", line 286, in cleanup_pyspecs
    for op_and_version_type, versions in _group_by_op(tuple(specs)):
  File "/home/matteius/pipenv/pipenv/vendor/requirementslib/models/markers.py", line 200, in _group_by_op
    specs = [_get_specs(x) for x in list(specs)]
  File "/home/matteius/pipenv/pipenv/vendor/requirementslib/models/markers.py", line 200, in <listcomp>
    specs = [_get_specs(x) for x in list(specs)]
  File "/home/matteius/pipenv/pipenv/vendor/requirementslib/models/markers.py", line 184, in _get_specs
    for spec in set(specset):
TypeError: 'LegacySpecifier' object is not iterable