question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. ItĀ collects links to all the places you might be looking at while hunting down a tough bug.

And, if youā€™re still stuck at the end, weā€™re happy to hop on a call to see how we can help out.

New Python template for pip without pipenv

See original GitHub issue

Community Note

  • Please vote on this issue by adding a šŸ‘ reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave ā€œ+1ā€ or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

pipenv is really slow and sometimes it hangs for a really long time.

Any objection to removing it? I think leaving the Python env setup to the user is fine - and just recommending Pyenv + virtualenv. Perhaps writing out a simple requirements.txt file instead of a Pipfile in new templates and docs.

Example where it hangs for me - perhaps because Iā€™m using a private PyPi instance that makes it worse, but that works fine with plain pip.

Installing dependencies from Pipfile.lock (8d4353)ā€¦

References

https://github.com/pypa/pipenv/issues/3827

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:7
  • Comments:12 (10 by maintainers)

github_iconTop GitHub Comments

1reaction
cmclaughlincommented, Aug 21, 2020

I spent some time looking into my specific/recent problem with pipenvā€¦ Iā€™ve determined pipenv is not compatible with AWS Code Artifactā€¦ sorry I brushed that off as a possibility earlier. While most cdktf users wonā€™t experience this, it certainly frustrates me enough to not want to use pipenv and recommend against anyone else using - because itā€™s not following what the rest of the Python community is doing. Add that to my previous bad experiences with it, heaps of other people that donā€™t like it and the general feeling that itā€™s slow and unnecessary.

Here are more details:

PyPi has a ā€œlegacyā€ API with a ā€œ/simpleā€ endpoint. Despite the name, itā€™s still used.

The PyPi reference implementation supports listing all packages via that endpoint, however as best as I can tell normal pip does not use it.

Here you can see the endpoint is still supported.

$ curl -s https://pypi.org/simple/ | head
<!DOCTYPE html>
<html>
  <head>
    <title>Simple index</title>
  </head>
  <body>
    <a href="/simple/0/">0</a>
    <a href="/simple/0-0/">0-._.-._.-._.-._.-._.-._.-0</a>
    <a href="/simple/00000a/">00000a</a>
    <a href="/simple/0-0-1/">0.0.1</a>

Iā€™m under the impression that normal pip used to parse that HTML response to follow links to packages or perhaps it still uses the HTML response to search/list packages. But Iā€™m pretty sure newer endpoints, such as the JSON API, are used for that now.

I am fairly certain normal pip does not use the /simple endpoint to install packages. Instead it simply gets /simple/$PACKAGE_NAME. I scoured through the pip code to confirm this, but ultimately some simple verbose output shows that installing boto3 goes straight to /simple/boto3/

$ pip --no-cache-dir -v install boto3
https://pypi.org/simple
https://test.pypi.org/simple
Using pip 20.2.2 from /Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-ephem-wheel-cache-hsm01ybq
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Initialized build tracking at /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Created build tracker: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Entered build tracker: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-req-tracker-9i8hbanx
Created temporary directory: /private/var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pip-install-vpuyirq2
1 location(s) to search for versions of boto3:
* https://pypi.org/simple/boto3/
Fetching project page and analyzing links: https://pypi.org/simple/boto3/
Getting page https://pypi.org/simple/boto3/
Found index url https://pypi.org/simple
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/boto3/ HTTP/1.1" 200 114817
  Found link https://files.pythonhosted.org/packages/3f/95/a24847c245befa8c50a9516cbdca309880bd21b5879e7c895e953217e947/boto3-0.0.1-py2.py3-none-any.whl#sha256=bc9b3ce78d3863e45b43a33d076c7b0561f6590205c94f0f8a23a4738e79a13f (from https://pypi.org/simple/boto3/), version: 0.0.1

Now, on to AWS CodeArtifact - it implements the PyPi spec, but it does not support listing packages via the simple API - from https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html:

ā€œCodeArtifact supports PyPIā€™s Legacy APIs, except the simple API. CodeArtifact does not support PyPIā€™s XML-RPC or JSON APIs.ā€

Clearly AWS left out support for listing the /simple endpoint intentionally - presumably because they knew pip no longer uses it.

This is relevant because when you use a private PyPI with pipenv, you add it as a source to the Pipfile - hereā€™s an example:

[[source]]
url = "https://aws:$CODEARTIFACT_AUTH_TOKEN@repo-account_id.d.codeartifact.us-east-1.amazonaws.com/pypi/company/simple/"
verify_ssl = true
name = "code-artifact"

Sadly, pipenv uses the /simple API to list packages. It took me a while to figure this out - hereā€™s an example where I waited over two hours with no errors.

$ time pipenv install psycopg2-binary
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing psycopg2-binaryā€¦
āœ˜
Error:  An error occurred while installing psycopg2-binary!
WARNING: I/O operation on closed file
Traceback (most recent call last):
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/core.py", line 2125, in do_install
    sp.write_err(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/vistir/spin.py", line 316, in write_err
    self.out_buff.write(decode_output(text, target_stream=self.out_buff))
ValueError: I/O operation on closed file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cmclaughlin-local/.pyenv/versions/cdktf/bin/pipenv", line 10, in <module>
    sys.exit(cli())
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/cli/command.py", line 232, in install
    retcode = do_install(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/core.py", line 2146, in do_install
    sp.write_err(vistir.compat.fs_str(
  File "/Users/cmclaughlin-local/.pyenv/versions/3.8.2/envs/cdktf/lib/python3.8/site-packages/pipenv/vendor/vistir/spin.py", line 316, in write_err
    self.out_buff.write(decode_output(text, target_stream=self.out_buff))
ValueError: I/O operation on closed file

real	143m31.134s
user	0m27.613s
sys	0m4.403s

But after banging on it some more, the error is here in clear sight:

$ pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependenciesā€¦
Locking [packages] dependenciesā€¦
Building requirements...
Resolving dependencies...
āœ˜ Locking Failed!
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/Users/cmclaughlin-local/.pyenv/versions/3.8.1/envs/cdktf-playground/lib/python3.8/site-packages/pipenv/utils.py", line 818, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches ssm-agent (from -r /var/folders/pw/v0cjf_4967q409vfpwg0v3140000gp/T/pipenv9kaeddt2requirements/pipenv-zxr4fo1l-constraints.txt (line 3))
No versions found
Were https://pypi.org/simple or https://aws:xxx@xxx-xxxxx.d.codeartifact.us-east-1.amazonaws.com/pypi/xxxx/simple/ reachable?

At this point, Iā€™d be happy to dig into the pipenv source to be absolutely certain Iā€™m on the right track - that itā€™s really trying to get /simple.

And of course, as the AWS docs clearly state, the endpoint simply does not exist:

$ curl https://aws:xxxx@shopstyle-xxxx.d.codeartifact.us-east-1.amazonaws.com/pypi/xxxx/simple/
Not Found

All of this might seem more like a bug report for pipenv rather than cdktf, but I really donā€™t care about pipenv. Iā€™m loving cdktf so far and would rather spend my time ripping pipenv out of cdktf than filing a bug report against pipenv - there are plenty examples of issues on that repo going no where.

PS While I didnā€™t show an example of normal pip installing from Code Artifact, that does in fact work - and as I previusly mentioned I think normal pip and a basic requirements.txt in the root of any cdktf stack is simple and sufficient. Combined with a recommendation of pyenv and virtualenv, I think the community will have less problems in the long term.

0reactions
github-actions[bot]commented, Dec 7, 2022

Iā€™m going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If youā€™ve found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Basic Usage of Pipenv - Read the Docs
Pipenv is a reference implementation for using Pipfile. ... Do not keep Pipfile.lock in version control if multiple versions of Python are being...
Read more >
template Ā· PyPI
A CLI tool for generating files from Jinja2 templates and environment variables.
Read more >
pipenv Documentation - Read the Docs
Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the. Python...
Read more >
How to set up a perfect Python project - Sourcery
Pipx is a handy utility that allows quick installation of python command line tools. We'll be using it to install pipenv and cookiecutter....
Read more >
Why Python devs should use Pipenv - Opensource.com
Enter pip, which most Python users are familiar with. pip lets us install and uninstall packages. We could specify versions, run pip freeze...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found