pip install generates file paths > 260 which break on Windows

Description

NB: This is a cross-bug report with https://github.com/pypa/pipenv/issues/5160. As per @matteius advice I am putting this on pip backlog.

pip can generate file paths during build which are more than 260 characters which breaks on Windows as described in this issue by https://github.com/netleibi/fastchunking/issues/6#issuecomment-877677653 by @netleibi

There are no suitable OS workarounds and therefore I propose that this is a pip bug.

Expected behavior

1. It should at least warn that long file paths are there and that users should apply a workaround to trim build path prefix using TMPDIR=C:\temp && TEMP=C:\temp or similar
2. It should generate shorter paths.

pip version

22.1.2

Python version

3.9.7

OS

Windows 10

How to Reproduce

pip install fastchunking

Output

$ pip install fastchunking
Collecting fastchunking
  Downloading fastchunking-0.0.3.tar.gz (24 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pybindgen
  Using cached PyBindGen-0.22.1-py2.py3-none-any.whl (152 kB)
Using legacy 'setup.py install' for fastchunking, since package 'wheel' is not installed.
Installing collected packages: pybindgen, fastchunking
  Running setup.py install for fastchunking: started
  Running setup.py install for fastchunking: finished with status 'error'
  error: subprocess-exited-with-error

  Running setup.py install for fastchunking did not run successfully.
  exit code: 1

  [50 lines of output]
  Generating file C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp
  running install
  running build
  running build_py
  creating build\lib.win-amd64-3.10
  creating build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\benchmark.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\test.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\__init__.py -> build\lib.win-amd64-3.10\fastchunking
  creating build\lib.win-amd64-3.10\lib
  copying lib\rabinkarpgen.py -> build\lib.win-amd64-3.10\lib
  copying lib\rabinkarp_gen.py -> build\lib.win-amd64-3.10\lib
  copying lib\__init__.py -> build\lib.win-amd64-3.10\lib
  running build_ext
  building 'fastchunking._rabinkarprh' extension
  creating build\temp.win-amd64-3.10
  creating build\temp.win-amd64-3.10\Release
  creating build\temp.win-amd64-3.10\Release\Users
  creating build\temp.win-amd64-3.10\Release\Users\matte
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build
  creating build\temp.win-amd64-3.10\Release\lib
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /TpC:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp /Fobuild\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj
  rabinkarprh.cpp
  lib\rabinkarp.h(116): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  lib\rabinkarp.h(171): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
  lib\rabinkarp.h(214): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(62): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(55): note: while compiling class template member function 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)'
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(34): note: see reference to function template instantiation 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)' being compiled
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(81): note: see reference to class template instantiation 'CharacterHash<uint32,unsigned char>' being compiled
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(64): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(67): warning C4293: '<<': shift count negative or too big, undefined behavior
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /Tplib/rabinkarp.cpp /Fobuild\temp.win-amd64-3.10\Release\lib/rabinkarp.obj
  rabinkarp.cpp
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\libs /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64 /EXPORT:PyInit__rabinkarprh build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj build\temp.win-amd64-3.10\Release\lib/rabinkarp.obj /OUT:build\lib.win-amd64-3.10\fastchunking\_rabinkarprh.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.lib
  LINK : fatal error LNK1104: cannot open file 'build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.exp'
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1104
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Encountered error while trying to install package.

fastchunking

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Hmm, so on Linux we could check PATH_MAX but on Windows we can’t tell (see this SO question which suggests that the setting is per-process and can’t be queried via documented APIs).

Given that I’m a strong -1 on producing a warning on paths over 260 characters for Windows systems with long paths enabled, this feels like it’s too hard to get right for too little benefit.

I’m willing to be convinced otherwise by a working PR (that doesn’t report false positives) though.

[pfmoore]

Yes, I think this is the point where it likely needs to be handled by setuptools, as they will know the process failed and can potentially report on why. Although of course they may also simply say that it’s the linker’s issue to give a better error message, as well…

As a strawman example, a linker on any platform that failed on paths over 100 characters long, for no better reason than they hard coded path buffers to that length, would behave in basically the same way as we’re seeing here. At some point, we have to assume that the tools we’re invoking work properly 🤷

The problem is that the further up the chain you go, the worse the risk of mistakenly guessing that it’s a “path too long” issue when it isn’t.