requirements from pypi-deps-db prioritized over specified provider

When using mach-nix to install a Python package and specifying its provider as “nixpkgs”, the dependency resolution logic appears to prioritize computed information from pypi-deps-db even if the Nix expression in nixpkgs for the package has changed the requirements for the package to something else.

For example, the aws-sam-cli expression in nixpkgs changes its requirements.txt to replace docker~=4.2.0 with docker>=4.2.0, which allows it to build with the current version of the docker package in nixpkgs, which is 4.4.4.

So the shell.nix file is able to combine the requirements aws-sam-cli==1.23.0 and docker==4.4.4, as long as I also set the provider for aws-sam-cli to “nixpkgs”:

{ pkgs ? import <nixpkgs> { } }:

with pkgs; let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix";
    ref = "refs/tags/3.2.0";
  }) {
    # currently aws-sam-cli is a standalone `buildPythonApplication`
    # in nixpkgs; the commit below includes changes which put
    # aws-sam-cli into python3Packages as a `buildPythonPackage`,
    # which makes it visible to mach-nix.  See
    # https://github.com/NixOS/nixpkgs/pull/121716
    pkgs = import (fetchFromGitHub {
      owner = "kini";
      repo = "nixpkgs";
      rev = "bc4048a95f6177b67aecd7622e7fcce650cfed61";
      sha256 = "1mn1hbjx173708nmyvnddjlshh50d9rfhbh5czc1c7fg6z5k8i0h";
    }) { };

    # pypiDataRev = "bb1a875a427e41bf331a7ec8214d5ea9eabc0b16";
    # pypiDataSha256 = "0fjqq49wqy9phavzgiis4d5cnax2zqx472zz0iij757q15ql2z8b";
  };

in mkShell rec {
  buildInputs = [(mach-nix.mkPython {
    requirements = ''
      aws-sam-cli==1.23.0
      docker==4.4.4
    '';
    providers.aws-sam-cli = "nixpkgs";
  })];
}

(See NixOS/nixpkgs#121716 for information about the pkgs = line.)

But if I uncomment the pypiDataRev and pypiDataSha256 lines in the above shell.nix, I get the following error:

building '/nix/store/7d1s8f7ir8ilv2mngl5wwcaxcpaqn80s-mach_nix_file.drv'...

Some requirements could not be resolved.
Top level requirements:
  aws-sam-cli==1.23.0  docker==4.4.4
Providers:
  {'_default': 'wheel,sdist,nixpkgs',
   'aws-sam-cli': 'nixpkgs',
   'gdal': 'nixpkgs',
   'pip': 'nixpkgs,sdist',
   'setupmeta': 'wheel',
   'setuptools': 'nixpkgs',
   'wheel': 'nixpkgs,sdist'}
Mach-nix version: 3.2.0
Python: 3.8.9
Cause: None
The requirements which caused the error:
  docker==4.4.4
  docker==4.2.*,>=4.2.0 - parent: aws-sam-cli:1.23.0

builder for '/nix/store/7d1s8f7ir8ilv2mngl5wwcaxcpaqn80s-mach_nix_file.drv' failed with exit code 1
error: build of '/nix/store/7d1s8f7ir8ilv2mngl5wwcaxcpaqn80s-mach_nix_file.drv' failed
(use '--show-trace' to show detailed location information)

mach-nix 3.2.0 was released before aws-sam-cli 1.23.0, so I guess the default pypiDataRev doesn’t contain information about aws-sam-cli 1.23.0. But the pypiDataRev I’m explicitly providing was snapshotted from PyPI after aws-sam-cli 1.23.0 was released, so I’m guessing that mach-nix is incorrectly prioritizing the requirements information from pypi-deps-db (docker~=4.2.0) rather than the real requirements of the package being built (docker>=4.2.0).

[DavHau]

The behavor you’re experiencing is intentional. If mach-nix has data about a package in its DB, it will always build its dependency tree from this data instead of following nixpkgs. Mach-nix will only fall back to following nixpkgs when it doesn’t have the data. There are some reasons not to blindly trust the nixpkgs dependency tree. See this for example. Dependency trees in nixpkgs are often incomplete whenever tests are disabled, which they are in our case by default.

If we would want to take the nixpgs tree instead, the question arises, how would we even do that? How could mach-nix discover that docker~=4.2.0 has been re-defined to docker>=4.2.0? It could happen through some patch, or some script in patchPhase. There is no fixed standard about this. Mach-nix can only see that the nixpkgs candidate depends on docker version 4.4.0. Should we assume this means ==4.4.0 ? What if you would like to use docker docker~=4.3.0. Should that be allowed? I guess you can see where I’m heading. I think in general the data from the dependency DB is the most reliable and in the majority of cases it’s a good decision to follow this data.

If you have any idea on how to manage this better, I’m happy to discuss about it.

I have been thinking about adding a feature that allows to to inject or force specific edges into the dep tree. Something like:

forceDependencies = {
  aws-sam-cli = "docker==4.4.4"
};

[DavHau]

If it doesn’t have any data about a aws-sam-cli, but it finds aws-sam-cli in nixpkgs, it considers it a leaf in the dependency tree and totally disragards the dependencies of that package. It just inherits this package as is from nixpkgs. This package, of course, comes with dependencies, but the resolver itself is not aware of these and wouldn’t raise an error even if you would define some other version for any of aws-sam-cli’s dependencies in your requirements.

In that case, you would end up with a package collision since you have two different versions in your closure. The one defined in your requirements, and the one, indirectly inherited from nixpkgs.

BUT: There are some sneaky hacks in the mach-nix generated nix expression, which prevent these collisions. For all packages which have been taken from nixpkgs, it will recusively replace their dependencies, if and only if another package with the same normalized name has been defined by mach-nix.

In the end this is a best effort approach, wich works in most of the cases. But I think using the DB is better.