improve reliability of `poetry shell` configuring $PATH properly

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 10.14
• Poetry version: 0.11.5

Issue

From https://github.com/sdispater/poetry/issues/198#issuecomment-430742299.

Preface: I’m not sure if this can be realistically solved by Poetry or if you’re too much at the mercy of each person’s shell configuration. But here goes.

Certain shell configurations interact poorly with poetry shell; see https://github.com/sdispater/poetry/issues/198#issuecomment-424016403 and https://github.com/sdispater/poetry/issues/172 for examples. Resolutions to the issue tend to be ad-hoc workarounds that involve spelunking in your shell configuration, see https://github.com/sdispater/poetry/issues/198#issuecomment-430742299 and https://github.com/sdispater/poetry/issues/172#issuecomment-401554154 respectively.

The root of the issue seems to be that Poetry attaches the virtualenv to the beginning of $PATH, then spawns the shell, which may override that configuration by attaching more things to the beginning of $PATH. If possible, modifying $PATH after the shell initializes ought to give a better result that will work by default on more systems. I don’t recall seeing this issue with pipenv shell, but admittedly I didn’t dig as deep while I was using it as my primary tool.

[betafcc]

I was curious about the inconsistency of this command so I followed the bread crumbs, the OP points the right reason, but here is my walkthrough, which may be useful:

TL;DR

don’t use subprocess.call('/bin/bash') AFTER setting the venv variables, it will in fact inherit the environ but the child will run the rc files after, which may or may not overshadow relevant variables, depending on the user configuration

instead, do:

1. create the shell in a child process
2. send source command via stdin
3. bring up child’s interaction to user

This can be done via Popen, but a reliable and portable way is using pexpect.spawn.sendline followed by pexpect.spawn.interact

Explanation

1. Chasing the source lines

1 - poetry shell calls poetry.console.main which calls Application().run(), which uses cleo’s BaseApplication and wtv and finally gets to poetry.console.commands.shell.ShellCommand

2 - ShellCommand.handle runs self.env.execute(Shell.get().path)

Shell.get().path is meant to return the path of the current-running shell binary, probably no problem there

In [1]: from poetry.utils.shell import Shell
In [2]: Shell.get().path
Out[2]: '/bin/bash'  # also works in zsh and even in sh

3 - self.env.execute is the next step

self.env is created via Env.create_venv(self.poetry.file.parent, o, self.poetry.package.name) which creates VirtualEnv which finally contains the execute method that is run:

    def execute(self, bin, *args, **kwargs):
        with self.temp_environ():
            os.environ["PATH"] = self._updated_path()
            os.environ["VIRTUAL_ENV"] = str(self._path)


            self.unset_env("PYTHONHOME")
            self.unset_env("__PYVENV_LAUNCHER__")


            return super(VirtualEnv, self).execute(bin, *args, **kwargs)

You can see it sets enviroment variables and calls super().execute (which is Env.execute defined in the same file)

and there is the problem

    def execute(self, bin, *args, **kwargs):
        bin = self._bin(bin)


        return subprocess.call([bin] + list(args), **kwargs)

2. Whats wrong?

The problem is that the environment variables are set before the shell process is created, which do inherit these variables, but can overshadow in its .rc files execution startup-routine. This will be the case in every shell, maybe you’re lucky and your rc files does not overshadows the relevant variables, or maybe you can work around in your rc files to guard a $POETRY_SHELL conditional but that will always be unreliable

Here’s how it works:

$ which python
/home/betafcc/.pyenv/shims/python  # .bashrc activated pyenv shim

$ source "$(dirname $(poetry run which python))/activate" # chases the original venv/bin/activate and sources it
(poet-test-py3.7) $ which python
/home/betafcc/.cache/pypoetry/virtualenvs/poet-test-py3.7/bin/python  # virtualenv overshadows pyenv shim

Now notices what happens if I spawn a new bash:

(poet-test-py3.7) $ bash
(poet-test-py3.7) $

As any child process do, all environment variables are inherited, even my PROMPT (via $PS1) is still marked as in the virtualenv the result will be exactly the same if I had called bash via python’s subprocess.call:

(poet-test-py3.7) $ exit
(poet-test-py3.7) $ # returns to outer bash
(poet-test-py3.7) $ python -c 'import subprocess; subprocess.call("/bin/bash")'
(poet-test-py3.7) $ # now I'm in python-called bash shell

In both cases, the rc files will be executed as they normally do every time a bash|zsh|wtv process is initiaded, therefore, overshadowing the parent-inherited environment variables:

(poet-test-py3.7) $ which python
/home/betafcc/.pyenv/shims/python

Note that my prompt still inherited the parent $PS1, as I don’t have any PS1 setting in my .bashrc, but I do have pyenv’s shims activation, which overshadows the venv binaries

3. Solution

In short, the core problem can be reproduced in a fresh shell:

$ source /home/betafcc/.cache/pypoetry/virtualenvs/poet-test-py3.7/bin/activate; \
> bash;
(poet-test-py3.7) $ # does activate the venv
(poet-test-py3.7) $ which python
/home/betafcc/.pyenv/shims/python # but its overshadowed by startup rc sourcing

Ie, 1 - “run the source command”, 2 - “start interactive shell”,

What should be done is sending the command after the startup routines:

$ # fresh shell
$ bash -i <<< 'source /home/betafcc/.cache/pypoetry/virtualenvs/poet-test-py3.7/bin/activate;
> exec </dev/tty;'
$ source /home/betafcc/.cache/pypoetry/virtualenvs/poet-test-py3.7/bin/activate; exec </dev/tty
(poet-test-py3.7) $
(poet-test-py3.7) $ which python
/home/betafcc/.cache/pypoetry/virtualenvs/poet-test-py3.7/bin/python

Note the resulted lines, it wasn’t me who typed the line $ source /home/... it came from stdin, then the control came back to me via exec </dev/tty

Turns out this is the same thing pipenv shell does

    def fork_compat(self, venv, cwd, args):
        from .vendor import pexpect


        # Grab current terminal dimensions to replace the hardcoded default
        # dimensions of pexpect.
        dims = get_terminal_size()
        with temp_environ():
            c = pexpect.spawn(self.cmd, ["-i"], dimensions=(dims.lines, dims.columns))
        c.sendline(_get_activate_script(self.cmd, venv))
        if args:
            c.sendline(" ".join(args))


        # Handler for terminal resizing events
        # Must be defined here to have the shell process in its context, since
        # we can't pass it as an argument
        def sigwinch_passthrough(sig, data):
            dims = get_terminal_size()
            c.setwinsize(dims.lines, dims.columns)


        signal.signal(signal.SIGWINCH, sigwinch_passthrough)


        # Interact with the new shell.
        c.interact(escape_character=None)
        c.close()
        sys.exit(c.exitstatus)

the relevant lines are:

...
            c = pexpect.spawn(self.cmd, ["-i"], dimensions=(dims.lines, dims.columns))
        c.sendline(_get_activate_script(self.cmd, venv))
...
        # Interact with the new shell.
        c.interact(escape_character=None)

Ie, 1 - Create a interactive shell; 2 - queue the source command to be executed via stdin; 3 - bring subprocess interaction to user

[betafcc]

Using pipenv aproach, this probably would do as an almost drop in replacement for the subprocess.call:

import sys
import subprocess

def spawn_venv(cmd, venv_path, args=[], call_kwargs={}):
    if sys.platform == "win32":
        return subprocess.call([bin] + list(args), **call_kwargs)

    # pexpect.spawn doesn't work on windows
    # https://pexpect.readthedocs.io/en/stable/overview.html#windows
    from pexpect import spawn

    shell = spawn(cmd, list(args))

    # tell child process to source venv
    shell.sendline(get_activate_script(cmd, venv_path))

    # give child process interaction to user
    return shell.interact(escape_character=None)


# https://github.com/pypa/pipenv/blob/8707fe52571422ff5aa2905a2063fdf5ce14840b/pipenv/shells.py#L32-L54
def get_activate_script(cmd, venv):
    """Returns the string to activate a virtualenv.
    This is POSIX-only at the moment since the compat (pexpect-based) shell
    does not work elsewhere anyway.
    """
    # Suffix and source command for other shells.
    # Support for fish shell.
    if "fish" in cmd:
        suffix = ".fish"
        command = "source"
    # Support for csh shell.
    elif "csh" in cmd:
        suffix = ".csh"
        command = "source"
    else:
        suffix = ""
        command = "."
    # Escape any spaces located within the virtualenv path to allow
    # for proper activation.
    venv_location = str(venv).replace(" ", r"\ ")
    # The leading space can make history cleaner in some shells.
    return " {2} {0}/bin/activate{1}".format(venv_location, suffix, command)

Then just change subprocess.call([bin] + list(args), **kwargs) to spawn_venv(bin, self._path, args, kwargs)

This is not good enough for a PR because:

1. you may want to set the dimensions argument in the spawn call and add a event handler for terminal resizing, pipenv uses a backport of shutil.get_terminal_size for that

2. pipenv provides a cli flag that end’s up using subprocess.call normally

3. I don’t know which **kwargs can be used, would need to normalize between subprocess.call and pexpect.spawn

4. Maybe it breaks the simmetry between Env.execute and Env.run