question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Can not install any poetry based project with pip from source distribution

See original GitHub issue
  • I am on the latest Poetry version.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

  • OS version and name: NixOS 18.09

  • Poetry version: –

  • Link of a Gist with the contents of your pyproject.toml file: –

Issue

Hi,

this is not really a bug about how poetry works, but how poetry interacts with other python packaging ecosystem.

While working on a project of mine which uses pip inside (for now) I noticed that we can not install a package called backoff which in its recent version switched to poetry.

To reproduce the error you simply need to invoke the following command:

$ pip install backoff --no-binary :all:

Above command tries to install packages from source distribution and not from wheel (which is a requirement of my project).

Digging deeper I noticed that problem is that poetry depends on tomlkit which depends on poetry as a build tool.

I would bet that this is a bug of pip and looking at their issue tracker I found the issue being reported there: https://github.com/pypa/pip/issues/5739. If you look closed you will see that also flit was affected but they solved it by moving their dependencies around a bit.

I’m not sure what a possible solution might be but maybe vendoring of the dependencies might be a solution (eg. like pip does it). Or maybe there is a better way…

This kinda puts poetry (and actually all projects using it) as a blocker for me and prevents me from investigating further into poetry, which I really hope becomes more used in python ecosystem. This is also a reason why I’m posting this issue here, to try to bring attention to this kinds of bugs and to make poetry adoption easier.

In any case, thank you for you work.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:8
  • Comments:23 (8 by maintainers)

github_iconTop GitHub Comments

8reactions
abncommented, Apr 18, 2020

The upcoming release should provide a solution for this. With the next release poetry-core will become available. Applying the following patch to backoff should enable this to work with newer pip versions. There is a bug in pip==19.1.1 that does not setup the PEP517 environment correctly, hence failing --no-binary :all: builds. The following fix was verified to work with pip==20.0.2

diff --git a/pyproject.toml b/pyproject.toml
index 3469bf2..a42652d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,5 +38,5 @@ pytest-cov = "^2.6"
 pytest-asyncio = {version = "^0.10.0",python = "^3.5"}
 
 [build-system]
-requires = ["poetry>=0.12"]
-build-backend = "poetry.masonry.api"
+requires = ["poetry-core>=1.0.0a5"]
+build-backend = "poetry.core.masonry.api"

Note that this uses an alpha release of poetry-core.

3reactions
reversefoldcommented, Oct 2, 2019

It seems like, in lieu of a fix from pip, poetry needs to do something to solve this issue. The only real solution to me seems like vendoring any of poetry’s dependencies which, in turn, use poetry for their builds. This likely also applies to any transitive dependencies of poetry too, which makes the problem not only more complicated but prone to changing as more packages switch to poetry.

I have found a workaround which works for my use-case, at least for now. I’m building wheels for everything in the poetry.lock during a build. If I pass each of the packages to --no-binary except for tomlkit, poetry, and pastel I still get builds from source for everything else. My project doesn’t depend on those packages, they’re only needed for poetry during the build. I imagine this might still end up causing a problem for any build which wants or needs to use --no-binary :all: and which also depends on any of these 3 packages.

Here’s an example of what the (insanely long) command looks like in my case (built by a script):

poetry run pip wheel -w wheels --no-cache-dir --isolated --no-binary aniso8601 --no-binary argh --no-binary attrs --no-binary boto3 --no-binary botocore --no-binary CacheControl --no-binary cached-property --no-binary cachy --no-binary certifi --no-binary chardet --no-binary cleo --no-binary Click --no-binary colorama --no-binary coverage --no-binary dnspython --no-binary docopt --no-binary docutils --no-binary ec2-metadata --no-binary eventlet --no-binary fasteners --no-binary filelock --no-binary Flask --no-binary Flask-RESTful --no-binary greenlet --no-binary html5lib --no-binary ianitor --no-binary idna --no-binary importlib-metadata --no-binary itsdangerous --no-binary Jinja2 --no-binary jmespath --no-binary jsonschema --no-binary lockfile --no-binary magicreq --no-binary MarkupSafe --no-binary monotonic --no-binary more-itertools --no-binary msgpack --no-binary nose --no-binary nosecount --no-binary packaging --no-binary pathtools --no-binary pex --no-binary pkginfo --no-binary pluggy --no-binary psutil --no-binary py --no-binary pycryptodome --no-binary pylev --no-binary pyparsing --no-binary pyrsistent --no-binary python-consul --no-binary python-daemon --no-binary python-dateutil --no-binary pytz --no-binary PyYAML --no-binary requests --no-binary requests-toolbelt --no-binary reversefold.util --no-binary s3transfer --no-binary setproctitle --no-binary shellingham --no-binary six --no-binary tabulate --no-binary tenacity --no-binary toml --no-binary tox --no-binary urllib3 --no-binary virtualenv --no-binary watchdog --no-binary webencodings --no-binary Werkzeug --no-binary zipp aniso8601==8.0.0 argh==0.26.2 attrs==19.2.0 boto3==1.9.240 botocore==1.12.240 CacheControl==0.12.5 cached-property==1.5.1 cachy==0.2.0 certifi==2019.9.11 chardet==3.0.4 cleo==0.6.8 Click==7.0 colorama==0.4.1 coverage==4.5.4 dnspython==1.16.0 docopt==0.6.2 docutils==0.15.2 ec2-metadata==1.8.0 eventlet==0.25.1 fasteners==0.15 filelock==3.0.12 Flask==1.1.1 Flask-RESTful==0.3.7 greenlet==0.4.15 html5lib==1.0.1 ianitor==0.1.0 idna==2.8 importlib-metadata==0.23 itsdangerous==1.1.0 Jinja2==2.10.1 jmespath==0.9.4 jsonschema==3.0.2 lockfile==0.12.2 magicreq==0.6.0 MarkupSafe==1.1.1 monotonic==1.5 more-itertools==7.2.0 msgpack==0.6.2 nose==1.3.7 nosecount==3.7.2.59 packaging==19.2 pastel==0.1.1 pathtools==0.1.2 pex==1.6.11 pkginfo==1.5.0.1 pluggy==0.13.0 poetry==0.12.17 psutil==5.6.3 py==1.8.0 pycryptodome==3.9.0 pylev==1.3.0 pyparsing==2.4.2 pyrsistent==0.14.11 python-consul==0.7.2 python-daemon==2.1.1 python-dateutil==2.8.0 pytz==2019.2 PyYAML==5.1.2 requests==2.22.0 requests-toolbelt==0.8.0 reversefold.util==3.0.1 s3transfer==0.2.1 setproctitle==1.1.10 shellingham==1.3.1 six==1.12.0 tabulate==0.8.5 tenacity==5.1.1 toml==0.10.0 tomlkit==0.5.5 tox==3.14.0 urllib3==1.25.6 virtualenv==16.7.5 watchdog==0.9.0 webencodings==0.5.1 Werkzeug==0.16.0 zipp==0.6.0
Read more comments on GitHub >

github_iconTop Results From Across the Web

Python package built with Poetry does not install supporting ...
I'm writing a Python package (assume it's "mypkg") and using Poetry to build the distributable wheel/tar.gz files for installation by the ...
Read more >
Cannot install project using poetry (#218) · Issues - GitLab
Installing the project is a pain because of dependencies. Historically, some packages (numpy, scipy, opencv) won't install from source, ...
Read more >
tox-poetry-installer - PyPI
A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile.
Read more >
12. Creating packages — Level Up Your Python - GitHub Pages
If there is no matching wheel (only for projects with binary components, in general), then pip gets the SDist and builds/installs manually. 12.2....
Read more >
Repositories | Documentation | Poetry - Python dependency ...
This means, in order to resolve dependencies for a project, Poetry needs metadata for all platform specific distributions. And when this metadata is...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found