Support for "soft" or suggested constraints

Feature proposed: constraints

It would be great is Poetry supports the fantastic feature that pip has: constraints.

Constraints are extremely useful for more complex applications that have many extras - thus optional dependencies and transitive dependencies as well. It is a great tool to provide reproducible installs of Python applications, without imposing strict pinning of dependencies and allowing the users of applications to manually upgrade and downgrade dependencies of the main “application” installed, even if they are relased after the main application has been released.

Short summary of how constraints work

When installing an application in pip user can specify --constraint flag with specification of constraints to use (in the same form as requirements - local file, http URL etc.). The constraints specified this way should be “pinned” versions (i.e ==VERSION only") and they change package resolution in the way that the only the version specified for the package is considered during dependency resolution.

Constraints are not “requirements” - if the user does not install specific requirement (for example because it is part of an optional extra), the package will not be installed even if it’s specific version is specified in the constraint file. Also constraints are exclusively used to perform resolution when the “installation” process resolve packages, and they are immediately forgotten once this particular “install” command completes. This allows to manually upgrade any of the packages that were “pinned” by constraints as long as it is within “requirements” specified by other packages.

Why it is useful

It is useful to get reproducible installs of applications (not libraries) without limiting security upgrades (and non-security upgrades as well).

It allows for fully reproducible, yet secure “from the scratch” Python application installs (web apps, CLI apps - generally apps that are supposed to provide user-facing features rather than libraries for other apps) without pinning specific version of dependencies in “hard” way. Fully reproducible install means that no matter if you install it today or few years from now, the application should install correctly - no matter if direect or transitive dependencies released new versions.

Typically applications might pin their dependencies to specific version and this is how you typically approach “applications” (as opposed to libraries that typically have “open” dependencies) . If you want to have truly “reproducible” install, you need to pin all your dependencies this way (including transitive ones), because otherwise transitive dependencies might break your “from the scratch” install - impacting the “first contact” with your application.

However there is a drawback of that - because when you pin dependencies, user cannot - independently - upgrade any of the dependencies that are pinned - and if those dependencies release even a small security fix, the main applicaiton must be upgraded to take into account. This is a limitation of pinning. It means that user who wants to upgrade security fix must wait for main application to release new version. In the world where supply chain attacks are a thing, and where security becomes more and more important, giving the user option to upgrade independently dependencies after the fact of installing an application is crucial.

Constraints nicely allow to make “reproducible installs” while keeping the possibility of “security updates” for any dependencies.

Another consequences of using constraints is that it also allows the user of application to perform non-security updates for the dependencies, which is important in cases like Airflow, where Airlfow is not only application to run, but also is a platform which provides library for Python developers (in Airflow DAG Authors develop workflows as Python code and they often want to be able to upgrade libraries installed by Airflow).

Lack of support for constraints is the reason why Airlfow discourages usage of poetry (even though we woudl love to be able to get their users to use poetry).

Example

Apache Airlfow is heavily depending on constraints - they developed a mechanism to automatically upgrade their constraint files based on result of automated tests and the only “recommended” way of installing airflow is via pip with constraints. Lack of constraint support is the only reason why poetry is discouraged: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

The constraints of airflow are maintained automatically, tagged together with each version of Airflow - main version here: https://github.com/apache/airflow/tree/constraints-main.

Yes, Airlfow is a bit special case with almost 700 dependencies and ~100 extras which is a bit extreme, but many otther applications could benefit from that approach. Constraint mechanism in Airlfow is used for more than 3 years and it helped Airflow maintainers in numerous cases where 3rd-party dependencies would have broken Airlfow “clean install” for already relased historical packages, while allowing their users to upgrade many dependencies as needed. More elaboration of why it is needed and what problems it solved for Airflow are explained in this talk from PyWaw # 98 “Managing Python dependencies at scale” https://www.youtube.com/watch?v=mlOkkTuucSk

Alternatives

As @dimbleby mentioned in https://github.com/python-poetry/poetry/issues/7047#issuecomment-1317833270 the https://github.com/python-poetry/poetry/issues/3225 closing reason was that it is the same as lock file.

While lock files are “almost” constraints, there is one difference, Lock files are development feature for someone who develops the application and once the package is uploaded to PyPI, the lock file remains only in the source code of the application. On the other hand, constraints are user facing feature. Users should be able to install an application from pypi (or another compliant repository) and apply constraints (for example taken from a published .lock file or file following requirements.txt format) as a single installation command similar to what Airlfow installation does with pip:

pip install "apache-airflow[celery]==2.4.3" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt"

The above is just an example, there might be other conventions used by other applications.

The convention should allow for different set constraints per Python MAJOR.MINOR verison (necessarily for big applications, set of dependencies slightly differ between Python 3.7, 3.8, etc. It would also be great to add other options - such as architecture (ARM/x86) but this is not as important as Python versions. This could be done per convention of the location of the file, or in case of poetry.lock it might be defined using poetry.lock features.

It should be possible to use lock files (and ideally also format compatible with requirements.txt) as “constraints” while installing a package by the user from PyPI, without having sources, pyproject.toml nor without having to copy poetry.lock manually to the local folder. In this sense, poetry.lock is not far from constraints, what is lacking is support for single-line installation, where constraints are specified as a remote URL to pull automatically and use durig installation by the end user from PyPI or another registry. In fact using and publishing poetry.lock as the “constraint” file to be used could be one of the main use-cases for poetry-managed applications.

Ideal properties of the constraints feature

• should be possible to run poetry install NNN --constraint http://..... to apply constraints remotely
• it should support both poetry.lock and traditional “requirements.txt” format to ease interoperabiliyt with pip
• ideally there shoudl be a tool that could allow conversion between the lock and requirement.txt

If there is a consensus among poetry maintainers that this is a worthy feature, I am happy to help in both design and implementation of this. I have vast experience in managing dependencies in Airlfow with using constraints (I am the original author of the approach Airlfow uses and I evolved and maintained it over last 3 years or so).

• I have searched the issues of this repo and believe that this is not a duplicate.

Actually there was a duplicate https://github.com/python-poetry/poetry/issues/3225 but it has been closed, but in the dicusson in https://github.com/python-poetry/poetry/issues/7047 I decided to open it againt.

• I have searched the FAQ and general documentation and believe that my question is not already covered.

[potiuk]

This thread is pure gold for my talk 😃. Thanks @r-richmond for such detailed description and question 😃. I would love to hear if there is an intereste in supporting case like this - because my talk (and I hope maybe future PEP?) is very much about this case.

[potiuk]

I like the direciton of poetry import --constraints to convert a constraints file to something poetry would understand. But I am a bit baffled why we are talking about updaitng pyproject.toml.

I would love to understand some in-s/out-s, and maybe I will try to explain where I am coming from in my words.

Maybe poetry is not the best tool for the job and maybe it’s completely off the radar for poetry what I am trying to achieve.

I believe most of poetry's behaviour, .lock file etc. is about building and extending a package. I.e. - developing a package and its dependencies, it’s about the project that has own python code, main package, and it has a number of dependencies. It is being developed and updated by a developer and the developers might continue to update, freeze (lock) the dependencies and share the .lock file with other developers. And eventually produces a package that can (if needs be) distributed via PyPI - for example as a wheel file.

That’s where local pyproject.toml file (describing your package) is. I believe pyproject.toml is all about describing dependencies of your own package you develop.

For me --constraint feature of pip (and the way we use and recommend - or even require it - in airflow) is purely a user feature. User who wants to install (released by us) airflow 2.4.3 + n other packages and make sure they are not conflicting with each other, and when they want to upgrade airflow to 2.5.0 + n the same packages, they just want to change the version and run a magical command and voila - it is installed with the right set of dependencies.

I am not trying to improve live of someone who has their own python code and develops it and build their own packages, but someone who consistently (now and few months from now - in the CI environment etc.) wants to pull and install a number of dependencies that consists of the “virtual environment” and wants to have an easy path to upgrade those in the future by purely specifying "I want to know have airlfow 2.5.0 installed - plus those other packages as well, go figure which are the best versions.

This is the problem I am after - purely user feature.

I understand (and please correct me if I am wrong) @r-richmond you are one of the airflow users who uses (or maybe mis-uses - I do not know poetry that well - maybe this is not at all intended use of it) poetry to manage such “user installation”.

For me such a user is purely a consumer of packages, not a producer of those (and IMHO pyproject.toml is all about producing packages). None of the relevant PEPs- PEP 517 and PEP 518 nor PEP 621 nor even PEP 660. mention the “user” case - they are all about building packages.

I am talking about purely installing packages from PyPI - without having to build those packages locally - using the wheel files (let’s assume we have all the binary wheels needed). My “improvement” proposal is that such packages distributed in PyPI as wheel files could also have an optional meta-data (in PyPI) describing the “known good set of constraints for repeatable installation”. Imagine “main” application package. And whenever user wants to get a consistent installation with those “known good dependencies” he might want to run (for example) pip install airflow[extras]==2.3.4 --use-known-constraints package1 package2 and the golden set of constraints of airflow 2.3.4 for that Python version will be used to resolve dependencies not only airlfow but also for package1 and package1. I imagined similar usage for poetry.

So I am not sure why do we need to involve pyproject.toml in this case ? Is it just to get list of packages to install? Are we thinking of using it for something else (for me pyproject.toml is a development tool not something that users would like to keep just to tell “those are dependencies I want to install”.

Or maybe I am completely wrong about this? Maybe poetry simply should not be used in the case I am descrinbed? Maybe it does not have an ambition to be used for that case and we should simply use pip install as we do now? Or maybe there is another usage of poetry for that case that I do not understand.

I would really love to know answer to that question, because I feel we are talking about somewhat different use-cases.