Poetry debugging with PyCharm not possible?

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• [x ] I have searched the documentation and believe that my question is not covered.

Issue

Hi! I am working on a Poetry enhancement to install Anaconda packages with Poetry. The package resolution already works fine. I just transformed the Anaconda packages into dummy Python Wheels reflecting the conda package dependencies. These dummy wheels I deliver via a private PyPIserver. And Poetry can work with them.

volker@runner:~/workspace/PYTHON4/test_poetry$ poetry add numpy
Using version ^1.20.3 for numpy

Updating dependencies
Resolving dependencies... (0.1s)PackageInfo: Invalid constraint (_libgcc_mutex) found in libgcc-ng-9.3.0 dependencies, skipping
PackageInfo: Invalid constraint (_openmp_mutex (>=4.5)) found in libgcc-ng-9.3.0 dependencies, skipping
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 9 installs, 1 update, 0 removals

  • Installing intel-openmp (2021.4.0)
  • Installing libgcc-ng (9.3.0)
  • Installing mkl (2021.4.0)
  • Installing blas (1.0)
  • Installing libstdcxx-ng (9.3.0)
  • Installing mkl-service (2.4.0)
  • Installing mkl-fft (1.3.1)
  • Installing mkl-random (1.2.2)
  • Installing numpy-base (1.20.3)
  • Updating numpy (1.22.1 -> 1.20.3)

As you can see the dummy Wheels as well as the resolution of the deps works quite well.

But now I have to build a CondaInstaller(BaseInstaller) that does the actual symlinking of the conda-packages into the Poetry-venv. Or if this is not so easy I could utilize Conda to do the same thing. This should be only a couple of lines.

But to enhance the Poetry Code I must be able to debug it from within PyCharm.

Running poetry from PyCharm works without problems. I configured the run-config to utilize ~/.local/bin/poetry with cmd line parameters “add numpy”. As Interpreter I can choose the venv poetry is installed in or the system python. The working directory is set to the target venv “test_poetry” where the numpy package should be added. Works like charm.

But when I activate the debugging I get the following stacktrace. This is independet of interpreter settings etc.:

  EnvCommandError

  Command ['/home/volker/workspace/venvs/test-poetry-8iaVxEo2-py3.9/bin/python', '-W', 'ignore', '-'] errored with the following return code 2, and output: 
  unknown option --port
  usage: /home/volker/workspace/venvs/test-poetry-8iaVxEo2-py3.9/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
  Try `python -h' for more information.
  input was : import sys
  
  if hasattr(sys, "real_prefix"):
      print(sys.real_prefix)
  elif hasattr(sys, "base_prefix"):
      print(sys.base_prefix)
  else:
      print(sys.prefix)
  

  at ~/workspace/PYTHON4/poetry/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│ 
      1185│         return decode(output)
      1186│ 
      1187│     def execute(self, bin, *args, **kwargs):

Process finished with exit code 1

This behavior is quite easily to reproduce. Just try to debug a typical installation of Poetry with PyCharm. This looks to me as if Poetry switches the interpreter and this messes up with the pydevd debug framework. I can actually debug poetry for a while, but in

    def create_venv(
        self, io, name=None, executable=None, force=False
    ):  # type: (IO, Optional[str], Optional[str], bool) -> Env
        if self._env is not None and not force:
            return self._env

        cwd = self._poetry.file.parent
        **env = self.get(reload=True)**

the exception is raised.

Any help appreciated Volker

[volkerjaenisch]

Workaround found!

Unchecking Utilizing Settings -> Build, Execution, Deployment -> Python Debugger -> “Attach to subprocess automatically while debugging”

Solves the the problem.

This also disables the debugging of subprocesses, so this is not a general solution but only a workaround.

A clean solution would be for Poetry to check if it is run from PyCharm, debugger. Poetry already checks for some execution environments Anaconda, Mingw, etc, why not also checking for PyCharm debugging? Then Poetry could include the pydevd proxy of PyCharm (extending the cmd) to enable debugging of the subprocesses.

Cheers,

Volker

[neersighted]

Also, the issue is mostly because of Poetry aggressively using subprocesses to introspect the states of remote Python environments (we run python -c with strings a lot), or to install packages (spawning child pips) – it’s more Poetry stressing the limits of PyCharm than anything else.