[BUG] 61.0.0 breaks "setup.py install" by missing dependencies in install_requires

setuptools version

61.0.0

Python version

3.10

OS

ubuntu-latest in GitHub Actions

Additional environment information

No response

Description

In our pywbem package, we test the different documented methods to install the package. One of them is still “setup.py install”. I do understand that it is deprecated since setuptools v58.3.0, but that still means it should be expected to work.

The issue is that the “setup.py install” command successfully installs the pywbem package, but upon import it turns out that dependent packages were not installed, e.g. “six”.

The GitHub Actions test run showing that is: https://github.com/pywbem/pywbem/runs/5687824533?check_suite_focus=true

“pip install” on the package works with setuptools 61.0.0 and does install the dependent packages.

The requirements.txt file of the pywbem project does specify the dependent packages and the setup.py script loads the dependencies from requirements.txt and specifies them in the “install_requires” parameter of setup().

The “setup.py install” approach worked before setuptools 61.0.0, e.g. with 60.10.0, see this test run.

Expected behavior

61.0.0 should still support “setup.py install” as before.

How to Reproduce

1. Create a new virtual env on Python 3.10
2. pip install setuptools==61.0.0; pip uninstall six
3. git clone https://github.com/pywbem/pywbem.git; cd pywbem
4. ./setup.py install # should succeed
5. pip list # should display:

   Package    Version
   ---------- ----------
   pip        22.0.4
   pywbem     1.5.0.dev1
   setuptools 61.0.0
   wheel      0.37.1
   

   and the issue is that the dependent packages are not installed.
6. pip install . # should succeed
7. pip list # should display the correctly installed dependencies:

   Package            Version
   ------------------ ----------
   certifi            2021.10.8
   charset-normalizer 2.0.12
   idna               3.3
   nocasedict         1.0.2
   nocaselist         1.0.4
   pip                22.0.4
   ply                3.11
   pywbem             1.5.0.dev1
   PyYAML             6.0
   requests           2.27.1
   setuptools         61.0.0
   six                1.16.0
   urllib3            1.26.9
   wheel              0.37.1
   yamlloader         1.1.0
   

Output

See above

[abravalheri]

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.

Hi @AndydeCleyre, I noticed that the fixtures seem to be creating distributions with no python modules.

When I compare METADATA_TEST_CASES for setup.cfg and setup.py I can see that setup.cfg has packages = find:, but setup.py lacks the equivalent packages=find_packages(). Is that on purpose?

---

Note that, prior to setuptools v61, these different configurations are not equivalent:

a) The example with setup.cfg would include sample_lib/__init__.py in the distribution b) The example with setup.py would be empty.

For the majority of use cases, the situation in (b) is accidental and unintended (maybe the same accident happened in your test case?).

With the improvements in v61, setuptools will try to auto-detect which files to include, but it will fail if the root directory has a confusing/equivocal layout.

The example generated by the fixtures seems to have extra folders dists and packages.

setuptools>=61 does not know exactly how to deal with these folders and therefore, halts the build process asking for the user to improve the configuration or use a src-layout.

This is the breaking change motivating the major bump from v60 to v61 and is described in the CHANGELOG.

The layouts recognized by setuptools for auto-discovery are described in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery.

Layouts that differ from the ones documented are asked to explicitly list packages and/or py_modules in their configuration.

These include “intentionally empty” packages and “meta-packages”[^1]. For those scenarios the users are asked to set packages=[].

[^1]: That only exists to specify dependencies.

[abravalheri]

Uummm… It is a possibility but I don’t think setuptools is that smart 😅