Fully automate updates in pinned requirements file

Description of issue or feature request:

TUF has a requirements-pinned.txt, which pins all direct and transitive dependencies for all supported Python versions. The file is created from the direct, unpinned dependencies listed in requirements.txt, using some shell commands (pip-compile for each Python version, etc.) plus manual dressing (combining files, adding per-version environment markers, etc.): https://github.com/theupdateframework/tuf/blob/f7695dace85444041489b83d5a66cd39c761bbd6/requirements.txt#L26-L42

If a new version becomes available for any of the dependencies listed in requirements-pinned.txt it is usually detected by Dependabot, which automatically bumps the version in a PR and triggers tests, allowing us to immediately detect any breaking updates in direct and transitive dependencies.

However, Dependabot does not patch requirements-pinned.txt upon removal or addition of any direct or transitive dependency, or if any environment markers would need a change, e.g. because Python version support is removed or added. Thus re-running above routine on a regular basis is required, albeit easily missed.

Current behavior: Semi-automatic update of pinned requirements

Expected behavior: Fully-automatic update of pinned requirements.

Available options:

• Schedule regular run of above routine E.g. in a GitHub cron-style Action. Note that this also requires automation of the manual part, i.e. adding environment makers. Or alternatively, we could add one file per supported Python version, which is what the authors of pip-compile recommend (see jazzband/pip-tools#651).
• Use state-of-the-art tooling pipenv? poetry?? Do they solve this issue? cc @trishankatdatadog who seems to know about “hypermodern” Python 😃

[lukpueh]

FYI: I recently did this for in-toto: https://github.com/in-toto/in-toto/pull/438/commits/4ce69eaa45fbc569baa80cef76c5bcd64f2bc685

I think we can do the same here and go back to just using pip-compile without any postediting. Regardless, we should consider pip-compileing automatically on a regular basis to account for changes (addition/removal) in transitive dependencies.

[trishankatdatadog]

• Use state-of-the-art tooling pipenv? poetry?? Do they solve this issue? cc @trishankatdatadog who seems to know about “hypermodern” Python 😃

They have their own set of problems: for example…