Requirements and pip check

Hey, thanks for putting together this package template. I’ve used a lot of stuff from your blog posts and this cookiecutter.

One thing I wanted to point out, as it’s something that we ran into, is that in your nox tests session you install the package requirements from the lock file. This created an issue when we slightly misspecified the dependencies in the pyproject.toml file. We were technically correct in the specification in that Poetry was able to smartly resolve a set of dependencies and put those in the lock file. However, the requirements list that would end up used by pip led to incompatible versions being installed.

Here is the order of operations for the cookiecutter when installing the package and running tests:

nox > Re-using existing virtual environment at .nox/tests-3-7.
nox > poetry export --ansi --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --requirement=[...]/requirements.txt
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt Cython numpy
nox > poetry build --ansi
Building package (0.0.1)
 - Building sdist
 - Built package-0.0.1.tar.gz

 - Building wheel
 - Built package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry version
nox > pip install --no-deps --force-reinstall dist/package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt coverage[toml] pytest boto3 moto

And here is how we do it now:

nox > Creating virtual environment (virtualenv) using python3.7 in .nox/tests-3-7
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt Cython numpy
nox > poetry build --ansi
Building package (0.0.1)
 - Building sdist
 - Built package-0.0.1.tar.gz

 - Building wheel
 - Built package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > poetry version
nox > pip install --upgrade --upgrade-strategy eager dist/package-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
nox > pip check
No broken requirements found.
nox > poetry export --ansi --dev --format=requirements.txt --output=[...]/requirements.txt
nox > pip install --constraint=[...]/requirements.txt coverage[toml] pytest boto3 moto
nox > coverage run --source package --parallel -m pytest -m not slow tests/

We use pip install --upgrade --upgrade-strategy eager so that the nox environments are forced to replicate what it would be like for a new install of the package. And we use pip check to verify the install before running the tests. We treat this as a comprehensive step but are not married to it.

Some final thoughts:

• Local development isn’t slowed down by this if you reuse the nox environments. Packages will update if needed and the CI pipeline (we use Gitlab) will still install things from scratch as before.
• An alternative way would be to directly validate the pyproject.toml file but I don’t know of anything that does this.
• Using pip 20.3 makes this moot so eventually one won’t have to worry so much about this.
• Related to this would be validating the pyproject.toml against the poetry.lock file.
• I wrote a class for manipulating poetry based off the stuff in your nox file. Might be nice to get that stuff into the nox-poetry package.

Once again, thanks for doing this. I learned a ton from it!

[alexifm]

Oh yeah, not an issue. so much has changed with pip too. I think it’s open just because I commented on it. Closing now.

[alexifm]

Sure. These are the dependencies we specified:

pyarrow = "^0.17"
awswrangler = "^1.6.2"

poetry will solve this as pyarrow=0.17.1 and awswrangler=1.6.3. However, pip will now install awswrangler=1.7.0 (which requires pyarrow=1.0.0) due to its recent release and because we used the caret notation. Of course, it will still install pyarrow=0.17.1 due to the explicit dependency on pyarrow. This caused a pip error message about incompatible libraries during the pip install. However, pip doesn’t terminate on this error.

We didn’t really expect our dependency spec to be a problem but awswrangler doesn’t really follow version rules where minor releases don’t have breaking changes. To fix this, we had to update our dependencies to be more constraining so pip wouldn’t install 1.7.0. It’s fine that an error occurred and that we had to adjust, but the nox sessions weren’t able to detect this error because they relied heavily on the poetry lock file.

Our solution ended up being the following pip-check session:

@nox.session(name="pip-check", python=package.python_versions)
def pip_check(session: Session) -> None:
    """Run pip check."""
    if not run_state.is_merge_request:
        session.skip()

    poetry = Poetry(session)
    poetry.install_package(build_dependencies=package.build_dependencies)  # <- runs `poetry build` and then `pip install` on the wheel
    session.run("pip", "check")

We adopted this approach because pip check checks the specification of the run dependencies of just the package and won’t collide with any dev dependencies. We opted to only run this in the CI pipeline just to make sure nothing slips by again.

When pip adopts stronger dependency resolution in later versions, this all may end up being moot. I should add, this new resolver likely would only be about the dependencies of the specific package in question. The new resolver appears to not fully require all packages to be compatible in a distribution.

Since pip check only works on installed packages, a preferable thing would be to not waste time installing the wheel. Instead, one would just parse the pyproject.toml file and run the check on the full list of dependencies that would be installed. I found some of the code that could be borrowed from pip to do this but with the new resolver coming, it’s probably not worth it.

Anyhow, that’s the story. Happy to share the Poetry class we made and some of the other ways we set up the nox file.