Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

New Python template for pip without pipenv

See original GitHub issue

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

pipenv is really slow and sometimes it hangs for a really long time.

Any objection to removing it? I think leaving the Python env setup to the user is fine - and just recommending Pyenv + virtualenv. Perhaps writing out a simple requirements.txt file instead of a Pipfile in new templates and docs.

Example where it hangs for me - perhaps because I’m using a private PyPi instance that makes it worse, but that works fine with plain pip.

Installing dependencies from Pipfile.lock (8d4353)…

References

https://github.com/pypa/pipenv/issues/3827

Issue Analytics

State:
Created 3 years ago
Reactions:7
Comments:12 (10 by maintainers)

Top GitHub Comments

1reaction

cmclaughlincommented, Aug 21, 2020

I spent some time looking into my specific/recent problem with pipenv… I’ve determined pipenv is not compatible with AWS Code Artifact… sorry I brushed that off as a possibility earlier. While most cdktf users won’t experience this, it certainly frustrates me enough to not want to use pipenv and recommend against anyone else using - because it’s not following what the rest of the Python community is doing. Add that to my previous bad experiences with it, heaps of other people that don’t like it and the general feeling that it’s slow and unnecessary.

Here are more details:

PyPi has a “legacy” API with a “/simple” endpoint. Despite the name, it’s still used.

The PyPi reference implementation supports listing all packages via that endpoint, however as best as I can tell normal pip does not use it.

Here you can see the endpoint is still supported.

$ curl -s https://pypi.org/simple/ | head
<!DOCTYPE html>
<html>
  <head>
    <title>Simple index</title>
  </head>
  <body>
    <a href="/simple/0/">0</a>
    <a href="/simple/0-0/">0-._.-._.-._.-._.-._.-._.-0</a>
    <a href="/simple/00000a/">00000a</a>
    <a href="/simple/0-0-1/">0.0.1</a>

I’m under the impression that normal pip used to parse that HTML response to follow links to packages or perhaps it still uses the HTML response to search/list packages. But I’m pretty sure newer endpoints, such as the JSON API, are used for that now.

I am fairly certain normal pip does not use the /simple endpoint to install packages. Instead it simply gets /simple/$PACKAGE_NAME. I scoured through the pip code to confirm this, but ultimately some simple verbose output shows that installing boto3 goes straight to /simple/boto3/

$ pip --no-cache-dir -v install boto3
https://pypi.org/simple
https://test.pypi.org/simple
Using pip 20.2.2 from /Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-ephem-wheel-cache-hsm01ybq
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Initialized build tracking at /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Created build tracker: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Entered build tracker: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-install-vpuyirq2
1 location(s) to search for versions of boto3:
* https://pypi.org/simple/boto3/
Fetching project page and analyzing links: https://pypi.org/simple/boto3/
Getting page https://pypi.org/simple/boto3/
Found index url https://pypi.org/simple
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/boto3/ HTTP/1.1" 200 114817
  Found link https://files.pythonhosted.org/packages/3f/95/a24847c245befa8c50a9516cbdca309880bd21b5879e7c895e953217e947/boto3-0.0.1-py2.py3-none-any.whl#sha256=bc9b3ce78d3863e45b43a33d076c7b0561f6590205c94f0f8a23a4738e79a13f (from https://pypi.org/simple/boto3/), version: 0.0.1

Now, on to AWS CodeArtifact - it implements the PyPi spec, but it does not support listing packages via the simple API - from https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html:

“CodeArtifact supports PyPI’s Legacy APIs, except the simple API. CodeArtifact does not support PyPI’s XML-RPC or JSON APIs.”

Clearly AWS left out support for listing the /simple endpoint intentionally - presumably because they knew pip no longer uses it.

This is relevant because when you use a private PyPI with pipenv, you add it as a source to the Pipfile - here’s an example:

[[source]]
url = "https://aws:$CODEARTIFACT_AUTH_TOKEN@repo-account_id.d.codeartifact.us-east-1.amazonaws.com/pypi/company/simple/"
verify_ssl = true
name = "code-artifact"

Sadly, pipenv uses the /simple API to list packages. It took me a while to figure this out - here’s an example where I waited over two hours with no errors.

$ time pipenv install psycopg2-binary
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing psycopg2-binary…
✘
Error:  An error occurred while installing psycopg2-binary!
WARNING: I/O operation on closed file
Traceback (most recent call last):
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/core.py", line 2125, in do_install
    sp.write_err(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/vistir/spin.py", line 316, in write_err
    self.out_buff.write(decode_output(text, target_stream=self.out_buff))
ValueError: I/O operation on closed file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cmclaughlin-local/.pyenv/versions/cdktf/bin/pipenv", line 10, in <module>
    sys.exit(cli())
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/cli/command.py", line 232, in install
    retcode = do_install(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/core.py", line 2146, in do_install
    sp.write_err(vistir.compat.fs_str(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/vistir/spin.py", line 316, in write_err
    self.out_buff.write(decode_output(text, target_stream=self.out_buff))
ValueError: I/O operation on closed file

real	143m31.134s
user	0m27.613s
sys	0m4.403s

But after banging on it some more, the error is here in clear sight:

$ pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches ssm-agent (from -r /var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pipenv9kaeddt2requirements/pipenv-zxr4fo1l-constraints.txt (line 3))
No versions found
Were https://pypi.org/simple or https://aws:xxx@xxx-xxxxx.d.codeartifact.us-east-1.amazonaws.com/pypi/xxxx/simple/ reachable?

At this point, I’d be happy to dig into the pipenv source to be absolutely certain I’m on the right track - that it’s really trying to get /simple.

And of course, as the AWS docs clearly state, the endpoint simply does not exist:

$ curl https://aws:xxxx@shopstyle-xxxx.d.codeartifact.us-east-1.amazonaws.com/pypi/xxxx/simple/
Not Found

All of this might seem more like a bug report for pipenv rather than cdktf, but I really don’t care about pipenv. I’m loving cdktf so far and would rather spend my time ripping pipenv out of cdktf than filing a bug report against pipenv - there are plenty examples of issues on that repo going no where.

PS While I didn’t show an example of normal pip installing from Code Artifact, that does in fact work - and as I previusly mentioned I think normal pip and a basic requirements.txt in the root of any cdktf stack is simple and sufficient. Combined with a recommendation of pyenv and virtualenv, I think the community will have less problems in the long term.

0reactions

github-actions[bot]commented, Dec 7, 2022

I’m going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you’ve found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.