[BUG] ensure_local_distutils seems to fail inside PyInstaller .exe (for new setuptools versions >= 60)

setuptools version

60.5.0 (>= 60.0.0 seems affected)

Python version

3.7.7 on win32 (3.7 - 3.10 seem affected)

OS

Windows 10

Additional environment information

No response

Description

I described this issue in detail on StackOverflow: https://stackoverflow.com/q/71027006/2111778 Below a copy:

---

I am trying to convert some Python code into an .exe with PyInstaller. My code uses distutils, which has already caused me some head scratching in the past as it seems to duplicate setuptools functionality. It also requires e.g. an unused import of setuptools to work properly which seems very unpythonic to me.

My first attempt to create an exe failed with the error message Module not found: 'setuptools' because my code only does import distutils explicitly (this works fine, but not inside the exe build). But knowing about the “unused import trick” I changed that to essentially import setuptools; import distutils, which basically instructs PyInstaller to include the setuptools module as well.

My script runs fine but after I turn it into an exe I get a traceback inside of the suspicious _distutils_hack submodule of setuptools. And yes it just prints a file name with no context.

Traceback (most recent call last):
  <18 lines omitted>
  File "PyInstaller\loader\pyimod03_importers.py", line 495, in exec_module
  File "_distutils_hack\override.py", line 71, in <module>
  File "_distutils_hack\__init__.py", line 71, in do_override
  File "_distutils_hack\__init__.py", line 59, in ensure_local_distutils
AssertionError: C:\Users\<omitted>\AppData\Local\Temp\_MEI294562\distutils\core.pyc 

I am using

• Python 3.7
• pyinstaller==4.8 (Jan 2022) for Windows
• distutils==3.7.7 (built-in)
• setuptools==60.5.0 (Jan 2022)

Apparently, setuptools is listed on PyPI and thus upgradable, but distutils is not listed on PyPI and thus not upgradable (the version is bundled with Python).

A workaround I found was downgrading to any version before 60

pip install --upgrade setuptools==59.8.0

Expected behavior

See above

How to Reproduce

See above

Output

See above

[xjcl]

I was able to do some additional debugging by replacing

import _frozen_importlib as _bootstrap

with

from . import _bootstrap
__bootstrap._setup(sys, _imp)

inside of importlib/__init__.py

---

Findings:

In Python, sys.meta_path is a list of importer objects (BuiltinImporter, FrozenImporter, SourceFileLoader, etc.) which are walked in order to find a module.

• _distutils_hack defines a DistutilsMetaFinder and appends it to the front of this list to ensure setuptools._distutils is loaded in place of distutils
• PyInstaller defines its own pyimod03_importers.FrozenImporter which is appended near the back of this list. This is what ends up being used, I assume the earlier importer get skipped because the source code files are not available to the exe. This is why a bytecode (.pyc) file appears in the assert.

So a potential fix would extend DistutilsMetaFinder to also be able to find the relevant .pyc file.

[xjcl]

On second thought my PowerShell script might overcomplicate things, plus the Python installer fails with exit code 0x666 if there is already a newer version on the system. So here a short manual guide assuming a pre-existing Python install.

1. Install via pip

setuptools==60.0.0 pyinstaller==4.8

1. Create file repro.py with content

import setuptools; import distutils

1. Run (the pyinstaller.exe should be in the Scripts directory of your Python install)

pyinstaller.exe --onefile repro.py

This should produce a file .\dist\repro.exe relative to the current working directory.

1. Run the file, this produces the AssertionError inside ensure_local_distutils on the command line

.\dist\repro.exe

  File "_distutils_hack\__init__.py", line 59, in ensure_local_distutils
AssertionError: C:\Users\<omitted>\AppData\Local\Temp\_MEI294562\distutils\core.pyc 

---

2a) I just found this. Note that the file

import setuptools

produces a different traceback:

  File "_distutils_hack\__init__.py", line 92, in create_module                                      
  File "importlib\__init__.py", line 126, in import_module                                        
ModuleNotFoundError: No module named 'setuptools._distutils' 

2b) Note that the file

import distutils; import setuptools

produces the same error as 2a but with an additional warning:

_distutils_hack\__init__.py:23: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.