Requirements and pip check
See original GitHub issueHey, thanks for putting together this package template. I’ve used a lot of stuff from your blog posts and this cookiecutter.
One thing I wanted to point out, as it’s something that we ran into, is that in your nox tests session you install the package requirements from the lock file. This created an issue when we slightly misspecified the dependencies in the pyproject.toml
file. We were technically correct in the specification in that Poetry was able to smartly resolve a set of dependencies and put those in the lock file. However, the requirements list that would end up used by pip led to incompatible versions being installed.
Here is the order of operations for the cookiecutter when installing the package and running tests:
nox > Re-using existing virtual environment at .nox/tests-3-7.
nox > poetry export --ansi --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --requirement=[...]/requirements.txt
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt Cython numpy
nox > poetry build --ansi
Building package (0.0.1)
- Building sdist
- Built package-0.0.1.tar.gz
- Building wheel
- Built package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry version
nox > pip install --no-deps --force-reinstall dist/package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt coverage[toml] pytest boto3 moto
And here is how we do it now:
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/tests-3-7
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt Cython numpy
nox > poetry build --ansi
Building package (0.0.1)
- Building sdist
- Built package-0.0.1.tar.gz
- Building wheel
- Built package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry version
nox > pip install --upgrade --upgrade-strategy eager dist/package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > pip check
No broken requirements found.
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt coverage[toml] pytest boto3 moto
nox > coverage run --source package --parallel -m pytest -m not slow tests/
We use pip install --upgrade --upgrade-strategy eager
so that the nox environments are forced to replicate what it would be like for a new install of the package. And we use pip check
to verify the install before running the tests. We treat this as a comprehensive step but are not married to it.
Some final thoughts:
- Local development isn’t slowed down by this if you reuse the nox environments. Packages will update if needed and the CI pipeline (we use Gitlab) will still install things from scratch as before.
- An alternative way would be to directly validate the
pyproject.toml
file but I don’t know of anything that does this. - Using pip 20.3 makes this moot so eventually one won’t have to worry so much about this.
- Related to this would be validating the
pyproject.toml
against thepoetry.lock
file. - I wrote a class for manipulating poetry based off the stuff in your nox file. Might be nice to get that stuff into the nox-poetry package.
Once again, thanks for doing this. I learned a ton from it!
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (2 by maintainers)
Top GitHub Comments
Oh yeah, not an issue. so much has changed with pip too. I think it’s open just because I commented on it. Closing now.
Sure. These are the dependencies we specified:
poetry
will solve this aspyarrow=0.17.1
andawswrangler=1.6.3
. However,pip
will now installawswrangler=1.7.0
(which requirespyarrow=1.0.0
) due to its recent release and because we used the caret notation. Of course, it will still installpyarrow=0.17.1
due to the explicit dependency onpyarrow
. This caused apip
error message about incompatible libraries during thepip install
. However,pip
doesn’t terminate on this error.We didn’t really expect our dependency spec to be a problem but
awswrangler
doesn’t really follow version rules where minor releases don’t have breaking changes. To fix this, we had to update our dependencies to be more constraining sopip
wouldn’t install1.7.0
. It’s fine that an error occurred and that we had to adjust, but thenox
sessions weren’t able to detect this error because they relied heavily on thepoetry
lock file.Our solution ended up being the following
pip-check
session:We adopted this approach because
pip check
checks the specification of the run dependencies of just the package and won’t collide with any dev dependencies. We opted to only run this in the CI pipeline just to make sure nothing slips by again.When
pip
adopts stronger dependency resolution in later versions, this all may end up being moot. I should add, this new resolver likely would only be about the dependencies of the specific package in question. The new resolver appears to not fully require all packages to be compatible in a distribution.Since
pip check
only works on installed packages, a preferable thing would be to not waste time installing the wheel. Instead, one would just parse thepyproject.toml
file and run the check on the full list of dependencies that would be installed. I found some of the code that could be borrowed frompip
to do this but with the new resolver coming, it’s probably not worth it.Anyhow, that’s the story. Happy to share the Poetry class we made and some of the other ways we set up the nox file.