Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Including (but not importing!) a requirements.txt file

See original GitHub issue

I’ve started using pipenv to manage local development environments for Tritium’s electric vehicle DC fast chargers, and it’s working pretty well for that purpose (huzzah!).

However, getting pylint to work nicely in dev environments is proving to be something of a challenge, since the way the charger environment creation works is to:

  1. Keep a local “requirements.txt” in the repo adjacent to each deployed app
  2. When those dependency declarations change (or when we want to do a dependency refresh for any other reason), generate a complete “locked-requirements.txt” for that target subsystem with pip-compile (so that we can have a single self-consistent venv-per-execution-environment, rather than a venv-per-app)
  3. Use the locked requirements files to generate a wagon archive for inclusion in the deployed firmware update bundles

I don’t have any intention to migrate the charger level dependency management away from pip-compile, since the app-requirements -> env-requirements -> deployment-bundle aspects of the pipeline are all very specific to our particular use case, and any tool that was sufficiently general purpose to be able to handle this would be harder to configure than just writing the required process automation scripts directly atop pip-compile.

The bit that’s relevant to pipenv is that what I’d like to be able to express via Pipfile is “When creating a dev environment, install the charger dependencies as well as the dev environment dependencies” (as that way, all the runtime APIs will be visible to pylint and other static analysis tools like mypy).

While pipenv install --dev -r subsystem/locked-requirements.txt comes close to this, it embeds a static snapshot of the locked requirements directly into Pipfile, rather than adding a reference to the input requirements to Pipfile, and then embedding the locked requirements themselves into Pipfile.lock. While that’s OK as a workaround in the near term, in the long run it’s a recipe for the snapshot in Pipfile getting out of sync with the actual charger dependencies in the subsystem-specific locked-requirements.txt files.

As a potential design for supporting this, my first thought is that we might actually need a new section in Pipfile: [include]

Entries in include would either be:

name = input_filename # "name" is just for debugging purposes

Or else a mapping that tweaks certain aspects of the included dependencies (like adding them to dev-packages instead of packages, setting a particular source for them all, setting environment markers for the inclusion, etc).

From the point of view of lockfile generation, included dependencies would be just like regular dependencies, with one exception: the _meta section would gain a new included_files field mapping relative path names for included files to their expected hash values.

If such a feature was deemed a potentially acceptable candidate for inclusion, then I’d focus specifically on requirements.txt support initially, as I think the considerations for composing a meta-Pipfile from other projects all using Pipfile themselves are potentially going to be different from those involved in composing a combined dev environment Pipfile from lower level flat requirements.txt files. (If I prove to be wrong about that, that would be a fine outcome - I just think it’s worth postponing the question until after we’ve considered the simpler requirements.txt-only concept)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:1
  • Comments:15 (13 by maintainers)

github_iconTop GitHub Comments

uranusjrcommented, May 6, 2018

On careful thought this seems like a reasonable thing to do. Cargo (of Rust) has something similar called Workspace. A workspace Cargo.toml does not have a set of dependencies itself, but only references a collection of child projects (members). Each member’s Cargo.toml represents a project on its own, and has its own lock file.

Note that a workspace in Cargo is all-or-nothing. A workspace Cargo.toml cannot specify any dependencies, and a non-workspace Cargo.toml cannot have any members. It would be worth discussing whether Pipfile should also be like this, or if we’d allow both [include] (maybe it’s better to name this includes, btw?) and [packages] to work in the same file.

As for the command-line API, I’d say it’s better to have a seperate subcommand, pipenv include [subproject-to-include]. Also it would make sense to be able to include another Pipfile, as well as a requirements.txt file.

matteiuscommented, Aug 17, 2022

Can this be closed?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to use imports from requirements.txt in python
txt file, but when the python file tries to import either of the requirements, it cannot find the module. I am currently working...
Read more >
Why requirements.txt isn't enough - model.predict
Create a file and list just the direct dependencies of your app. The same way you'd do with requirements.txt in Scenario #1....
Read more >
User Guide - pip documentation v22.3.1
“Requirements files” are files containing a list of items to be installed using pip install like so: Unix/macOS. python -m pip install -r...
Read more >
Why and How to make a Requirements.txt - Robert Boscacci
In short, we generate and share requirements.txt files to make it easier for other developers to install the correct versions of the required...
Read more >
How to Generate the Requirements of your Python project ...
txt to project imports. --clean <file> Clean up requirements.txt by removing modules that are not imported in project. --no-pin Omit version of ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found