64 bit pip installs 32 bit DLLs (PyQt5 and PySide2)

Environment

• pip version: 20.1.1
• Python version: 3.8
• OS: Windows 10

Installed Python for all users in custom location (%ProgramFiles%\Python38)

Description

Cannot run PyQt5 or PySide2 applications written in Python, either those I make myself or those also installed via pip.

Expected behavior

That 64 bit pip installs 64 bit DLLs so that Python scripts do not fail to load them. The code “from PyQt5.QtWidgets import QApplication” (or with PySide2 instead) should return without error.

How to Reproduce

Install packages PyQt5 and PySide2 with command “pip3 install pyqt5 pyside2” which I did from a console with admin privileges, though I don’t think that matters.

Output

C:\Users\Jesse Maurais>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5.QtWidgets import QApplication
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing QtWidgets: %1 is not a valid Win32 application.
>>>

The “not a valid Win32 application” error is one I’ve seen before so I went to the site-packages folder and found the Qt DLLs that are installed with PyQt5 and PySide2. Running dumpbin on them to determine their architecture, found

C:\Program Files\Python38\Lib\site-packages\PyQt5\Qt\bin>dumpbin /headers Qt5Widgets.dll
Microsoft (R) COFF/PE Dumper Version 14.26.28806.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file Qt5Widgets.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        5DB2AC3B time date stamp Fri Oct 25 02:03:07 2019
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
            2102 characteristics
                   Executable
                   32 bit word machine
                   DLL

Where the machine (x86) is the relevant information. Similarly for PySide2, found

C:\Program Files\Python38\Lib\site-packages\PySide2>dumpbin /headers Qt5Widgets.dll
Microsoft (R) COFF/PE Dumper Version 14.26.28806.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file Qt5Widgets.dll

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        5EB90FEE time date stamp Mon May 11 02:42:22 2020
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
            2102 characteristics
                   Executable
                   32 bit word machine
                   DLL

Then I double checked that the Python and pip executables were matching. They were not.

C:\Program Files\Python38>dumpbin /headers python.exe
Microsoft (R) COFF/PE Dumper Version 14.26.28806.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file python.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
            8664 machine (x64)
               6 number of sections
        5F15BF71 time date stamp Mon Jul 20 09:59:45 2020
               0 file pointer to symbol table
               0 number of symbols
              F0 size of optional header
              22 characteristics
                   Executable
                   Application can handle large (>2GB) addresses

Here we have machine (x64) and obviously a 64 bit program cannot load a 32 bit DLL (or vice versa). In fact no Qt app written in Python with these packages will even launch, exactly as I’ve observed. If pip cannot install 64 bit DLLs for Qt then it should not install these packages at all.

[JesseMaurais]

I believe that I’ve discovered the problem.

in …/Lib/distutils/util.py we have the code for the platform

def get_platform():
    if os.name == 'nt':
        TARGET_TO_PLAT = {
            'x86' : 'win32',
            'x64' : 'win-amd64',
            'arm' : 'win-arm32',
        }
        return TARGET_TO_PLAT.get(os.environ.get('VSCMD_ARG_TGT_ARCH')) or get_host_platform()
    else:
        return get_host_platform()

Which checks the environment variable VSCMD_ARG_TGT_ARCH for which a Google search did not land me in any official documentation, but I did find this page: https://bugs.python.org/issue38989

Besides the fact the MSVC’s target platform isn’t really related to the architecture for dependencies being installed, I’m not sure VSCMD_ARG_TGT_ARCH is an appropriate variable to look at in the first place. It doesn’t seem to be at all documented, and (from looking at the implemenation) it seems to be meant as how parse_cmd.bat communicates with vcvars.bat, dotnet.bat, winsdk.bat, etc (these are all sub-programms that make up vcvarsall.bat)

I think %PLATFORM% (which generally ends up set equal to %VSCMD_ARG_TGT_ARCH% once vsdevcmd\ext\vcvars.bat has done its thing) is might be a better variable that cross-compiling should actually look at when it wants to know what architecture MSVC will compile to.

And I think Kevin Puetz (puetzk) is correct. Using this variable to determine the platform dependencies for Python packages is not reliable; it can grab packages that do not match the architecture against which Python was built and so scripts using such packages will not run in the interpreter for which they were installed.

I ran afoul of this because I installed the packages in my Visual Studio Developers Command Prompt and because my Visual Studio installation was x86. This may have been obvious had I not cropped out the header whenever pasting text output from the prompt.

**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.6.3
** Copyright (c) 2020 Microsoft Corporation
**********************************************************************

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.util
>>> distutils.util.get_platform()
'win32'
>>>

And as for the environment of that same console

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>set VS
VS160COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\
VSCMD_ARG_app_plat=Desktop
VSCMD_ARG_HOST_ARCH=x86
VSCMD_ARG_TGT_ARCH=x86
VSCMD_VER=16.6.3
VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\
VSSDK150INSTALL=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VSSDK
VSSDKINSTALL=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VSSDK

Whereas if I run from the vanilla console

Microsoft Windows [Version 10.0.19041.388]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Users\Jesse Maurais>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.util
>>> distutils.util.get_platform()
'win-amd64'
>>>

And that is the correct answer.

C:\Users\Jesse Maurais>set VS
Environment variable VS not defined

Maybe this would also have been more obvious had I not been running the dev console as admin, since it lands you in C:\Windows\System32 by default, whereas with a default user it lands in you the VS folder under Program Files (x86) for some reason. Also I had probably been using the vanilla console when I ran some of your tests that gave the correct answer and the dev console when I installed the packages. The fault there was mine.

My work around for now is to install using vanilla cmd.exe and the dev console for my work. However, I suspect that the use of VSCMD_ARG_TGT_ARCH for this purpose is incorrect and that more people are not running into this issue only because they are not using the dev console.

[puetzk]

https://github.com/pypa/packaging/issues/327#issuecomment-826952546 is fixed by https://github.com/pypa/packaging/pull/396, but it needs a release and an update to the vendored pin before pip works again in VsDevCmd prompts.