Including (but not importing!) a requirements.txt file

I’ve started using pipenv to manage local development environments for Tritium’s electric vehicle DC fast chargers, and it’s working pretty well for that purpose (huzzah!).

However, getting pylint to work nicely in dev environments is proving to be something of a challenge, since the way the charger environment creation works is to:

1. Keep a local “requirements.txt” in the repo adjacent to each deployed app
2. When those dependency declarations change (or when we want to do a dependency refresh for any other reason), generate a complete “locked-requirements.txt” for that target subsystem with pip-compile (so that we can have a single self-consistent venv-per-execution-environment, rather than a venv-per-app)
3. Use the locked requirements files to generate a wagon archive for inclusion in the deployed firmware update bundles

I don’t have any intention to migrate the charger level dependency management away from pip-compile, since the app-requirements -> env-requirements -> deployment-bundle aspects of the pipeline are all very specific to our particular use case, and any tool that was sufficiently general purpose to be able to handle this would be harder to configure than just writing the required process automation scripts directly atop pip-compile.

The bit that’s relevant to pipenv is that what I’d like to be able to express via Pipfile is “When creating a dev environment, install the charger dependencies as well as the dev environment dependencies” (as that way, all the runtime APIs will be visible to pylint and other static analysis tools like mypy).

While pipenv install --dev -r subsystem/locked-requirements.txt comes close to this, it embeds a static snapshot of the locked requirements directly into Pipfile, rather than adding a reference to the input requirements to Pipfile, and then embedding the locked requirements themselves into Pipfile.lock. While that’s OK as a workaround in the near term, in the long run it’s a recipe for the snapshot in Pipfile getting out of sync with the actual charger dependencies in the subsystem-specific locked-requirements.txt files.

As a potential design for supporting this, my first thought is that we might actually need a new section in Pipfile: [include]

Entries in include would either be:

name = input_filename # "name" is just for debugging purposes

Or else a mapping that tweaks certain aspects of the included dependencies (like adding them to dev-packages instead of packages, setting a particular source for them all, setting environment markers for the inclusion, etc).

From the point of view of lockfile generation, included dependencies would be just like regular dependencies, with one exception: the _meta section would gain a new included_files field mapping relative path names for included files to their expected hash values.

If such a feature was deemed a potentially acceptable candidate for inclusion, then I’d focus specifically on requirements.txt support initially, as I think the considerations for composing a meta-Pipfile from other projects all using Pipfile themselves are potentially going to be different from those involved in composing a combined dev environment Pipfile from lower level flat requirements.txt files. (If I prove to be wrong about that, that would be a fine outcome - I just think it’s worth postponing the question until after we’ve considered the simpler requirements.txt-only concept)

[uranusjr]

On careful thought this seems like a reasonable thing to do. Cargo (of Rust) has something similar called Workspace. A workspace Cargo.toml does not have a set of dependencies itself, but only references a collection of child projects (members). Each member’s Cargo.toml represents a project on its own, and has its own lock file.

Note that a workspace in Cargo is all-or-nothing. A workspace Cargo.toml cannot specify any dependencies, and a non-workspace Cargo.toml cannot have any members. It would be worth discussing whether Pipfile should also be like this, or if we’d allow both [include] (maybe it’s better to name this includes, btw?) and [packages] to work in the same file.

As for the command-line API, I’d say it’s better to have a seperate subcommand, pipenv include [subproject-to-include]. Also it would make sense to be able to include another Pipfile, as well as a requirements.txt file.

[matteius]

Can this be closed?