Hemmed in by unnecessary dependency removal in 3.3+

I’m updating a dev environment and tried moving up to 3.3 from 3.2, but the unnecessary dependency removal feature is removing a required dependency of a private package added via overridesPre. My best-guess is that the dependencies of packages in overridesPre aren’t making it into the list of dependencies to preserve?

First, I’ll describe what we’ve been doing in 3.2.

We:

1. have a runCommand that generates a filtered copy of our requirements with incompatible stuff like the --extra-index-url line removed.
2. have a 2nd runCommand that finds the version of our private package specified in the filtered requirements.txt, fetches the private index, and looks up the url for this version.
3. build the package by feeding this url to mach-nix.buildPythonPackage
4. and include it in overridesPre

When we update to 3.3, we start seeing this early on in the build (followed by failures on missing dependencies later):

trace: 
 automatically detected pname: 'private_package'
trace: 
 automatically detected version: '1.2.23'
trace: 
 automatically detected requirements of private_package:1.2.23 22.05pre-git:
Flask
Flask-Cors
Flask-SQLAlchemy
SQLAlchemy
SQLAlchemy-Utils
mistletoe


trace: removing dependency python3.9-aiohttp-3.8.0 from black
... [ several more deps removed from black ] ...
trace: Updated inherited nixpkgs dep flask-script from 2.0.6 to 2.0.5
trace: removing dependency python3.9-nose-1.3.7 from flask-assets
trace: removing dependency python3.9-greenlet-1.1.2 from sqlalchemy
trace: Updated inherited nixpkgs dep flask from 2.0.1 to 1.0.4
trace: Updated inherited nixpkgs dep flask-cors from 3.0.10 to 3.0.10
trace: removing dependency python3.9-Flask-Cors-3.0.10 from private_package
trace: Updated inherited nixpkgs dep flask-sqlalchemy from 2.5.1 to 2.3.2
trace: Updated inherited nixpkgs dep mistletoe from 0.7.2 to 0.8.1
trace: removing dependency python3.9-mistletoe-0.8.1 from private_package
trace: Updated inherited nixpkgs dep sqlalchemy from 1.4.17 to 1.3.15
trace: Updated inherited nixpkgs dep sqlalchemy-utils from 0.37.6 to 0.37.6

I tried moving the package into packagesExtra, but it resolves the dependencies separately and ends up with conflicting versions of common dependencies:

error: collision between /nix/store/33789z8sgagf7kyag5nkw93n4rgfi7l6-python3.8-flask-1.1.2/bin/flask' and /nix/store/aicnwssf9v1alvxyp7qsg0h0fq66iqjb-python3.8-flask-1.0.4/bin/flask’

[bjornfor]

I started seeing errors like this (the segfault appears 2x):

I’ve noticed too, and did some bisecting, but haven’t found the time/energy to submit a (detailed) issue about it. (Apparently a setuptools upgrade in nixpkgs is causing this.)

[bjornfor]

There’s an issue about that segfault now: https://github.com/DavHau/mach-nix/issues/467