Execute python code before parsing setup.cfg

I’m currently trying to make my python packages PEP517/518 compatible. For this, I’m replacing the setup.py with a setup.cfg file and a project.toml file. This was quite straightforward to do for most of my packages, but for one package, I was not able to do this translation.

Package description

The package in question contains a (parent) python package that includes several packages that can also be installed as stand-alone python packages. For this, I created the following project structure:

    ├── main_package
    │   ├── python_module1
    │   ├── python_module3
    │   └── stand_alone_subpackage
    │       ├── stand_alone_subpackage
    │       │   └── subpackage_module
    │       │       └── script.py
    │       ├── pyproject.toml
    │       ├── requirements.txt
    │       ├── setup.py
    │       └── setup.cfg
    └── setup.py

I use the following setup.py file to install the subpackage in the parent package while getting rid of the redundant stand_alone_subpackage namespace.

from setuptools import setup, find_namespace_packages
import re

requirements = [
    "gym",
    "matplotlib",
    "numpy",
    "torch",
    "joblib",
    "tensorboard",
    "mpi4py",
    "psutil",
    "tqdm",
]

# Retrieve package list
PACKAGES = find_namespace_packages(include=["machine_learning_control*"])

# Add extra virtual shortened package for each of namespace_pkgs
namespace_pkgs = ["simzoo"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

# Run python setup
setup(
    name="main_package",
    version="0.0.0",
    description=(
        "A parent python package."
    ),
    install_requires=requirements,
    packages=PACKAGES,
    package_dir=PACKAGE_DIR,
)

Why this works can be found inside a StackOverflow question I opened some months ago. The main thing it does is adds the virtual shortened modules to the PACKAGES and PACKAGES_DIR variables. This way the PythonClassinside thestandalone_subpackage` can be imported using the following import command:

from main_package.stand_alone_supbpackage import PythonClass
``

instead of using the actual (longer) module namespace:

```python
from main_package.stand_alone_subpackage.stand_alone_supbpackage import PythonClass

Translated setup.py file

I translated the setup.py to the following setup.cfg file:

[metadata]
name = main_package
version = 0.0.0

[options]
packages = find:
include_package_data = True
install_requires =
    gym
    matplotlib
python_requires = >=3.5

I, however, did not yet find a way to inject the extra virtual (shortened) modules before the packages = find: command is parsed. One possible way would be to read the PACKAGES and PACKAGE_DIR variables from a file. This would be similar to using the file: and attr: arguments for the package version parameter. While doing so, this file has to be created before the setup.cfg file is parsed. I, however, did not find a way to do this in setuptools 50.3.2. My question is, therefore, does anybody know a way of achieving the more complicated setup procedure I am trying to achieve?

Update

I just found out that you can combine both the setup.cfg and setup.py code. This allows me to define most of the setup information inside the setup.cfg file while keeping the packages and package_dir arguments inside the setup.py file. This allows me to inject the virtual shortened modules inside the setup.py file. I currently use the following setup.py file:

from setuptools import setup, find_namespace_packages
import re

with open("README.md") as f:
    readme = f.read()

# Add extra virtual shortened package for each of namespace_pkgs
PACKAGES = find_namespace_packages(include=["machine_learning_control*"])
namespace_pkgs = ["simzoo"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

# Run python setup
setup(
    packages=PACKAGES, package_dir=PACKAGE_DIR,
)

If anybody knows if it is possible to put everything inside the setup.cfg file please let me know. Thanks a lot in advance!

[webknjaz]

Sounds about right

[nschloe]

As webknjaz said, a setup.cfg file doesn’t allow for much logic at all. If you’re getting more complex than normal (and that’s what your subpackage setup is), you can always keep a stripped-down setup.py around. It’s not unusual either, and necessary for example when you build packages with C++ sources in them. Check out pygalmesh (a package of mine): All metadata in setup.cfg, and the rest of the logic in setup.py. Perhaps this will help.