poetry install fails on KeyError even for defined environment variables

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian Linux 10 Buster
• Poetry version: 1.1.2
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pepastach/85b133f035fbb805cacd4d0fd75562b1

Issue

We run poetry install --remove-untracked and the process randomly fails on various “missing” environment variables. This has been reported in #3144 for the PATH variable (which happens quite a lot, more than other variables), but I have seen it for many other variables (often GitLab or Docker related). In all cases they are defined.

It fails really randomly. Different dependencies being installed, different environment variables. Sometimes it does not fail (but most often it does). Sometimes it fails on multiple environment variables. Like KeyError 'PATH', KeyError 'CI_JOB_STAGE' or KeyError 'DOCKER_ENV_CI_DISPOSABLE_ENVIRONMENT' in the same run. It seems to me like some kind of race condition in the parallel install. I can’t prove this hypothesis until we can disable parallel installation. I think it’s scheduled for an upcoming release.

Here is an example output where it fails on the PATH variable.

Creating virtualenv bp-xxx-3RqKXHa2-py3.7 in /builds/xxx/.cache/poetry/pypoetry/virtualenvs
Installing dependencies from lock file
Package operations: 106 installs, 0 updates, 0 removals
  • Installing attrs (20.2.0)
  • Installing six (1.15.0)
  • Installing cattrs (1.0.0)
...
Stack trace:
  8  /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:199 in _execute_operation
      197│ 
      198│             try:
    → 199│                 result = self._do_execute_operation(operation)
      200│             except EnvCommandError as e:
      201│                 if e.e.returncode == -2:
  7  /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:273 in _do_execute_operation
      271│             return 0
      272│ 
    → 273│         result = getattr(self, "_execute_{}".format(method))(operation)
      274│ 
      275│         if result != 0:
  6  /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:408 in _execute_install
      406│ 
      407│     def _execute_install(self, operation):  # type: (Install) -> None
    → 408│         return self._install(operation)
      409│ 
      410│     def _execute_update(self, operation):  # type: (Update) -> None
  5  /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:446 in _install
      444│             args.insert(2, "-U")
      445│ 
    → 446│         return self.run_pip(*args)
      447│ 
      448│     def _update(self, operation):
  4  /usr/local/lib/python3.7/site-packages/poetry/installation/executor.py:297 in run_pip
      295│     def run_pip(self, *args, **kwargs):  # type: (...) -> int
      296│         try:
    → 297│             self._env.run_pip(*args, **kwargs)
      298│         except EnvCommandError as e:
      299│             output = decode(e.e.output)
  3  /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:916 in run_pip
       914│         pip = self.get_pip_command()
       915│         cmd = pip + list(args)
    →  916│         return self._run(cmd, **kwargs)
       917│ 
       918│     def _run(self, cmd, **kwargs):
  2  /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:1183 in _run
      1181│     def _run(self, cmd, **kwargs):
      1182│         with self.temp_environ():
    → 1183│             os.environ["PATH"] = self._updated_path()
      1184│             os.environ["VIRTUAL_ENV"] = str(self._path)
      1185│ 
  1  /usr/local/lib/python3.7/site-packages/poetry/utils/env.py:1215 in _updated_path
      1213│ 
      1214│     def _updated_path(self):
    → 1215│         return os.pathsep.join([str(self._bin_dir), os.environ["PATH"]])
      1216│ 
      1217│ 
  KeyError
  'PATH'
  at /usr/local/lib/python3.7/os.py:681 in __getitem__
       677│         try:
       678│             value = self._data[self.encodekey(key)]
       679│         except KeyError:
       680│             # raise KeyError with the original key value
    →  681│             raise KeyError(key) from None
       682│         return self.decodevalue(value)
       683│ 
       684│     def __setitem__(self, key, value):
       685│         key = self.encodekey(key)
  • Installing pluggy (0.13.1)
  • Installing py (1.9.0)
  • Installing wcwidth (0.2.5)
  • Installing yarl (1.6.0)

[gsemet]

I see this issue from time to time on our gitlab ci (kubernetes runner). I works pretty much all the time after a retry (so same environment, same container, same git commit)

[cglacet]

That’s strange I got a similar error on gitlab CI (even though I don’t get the same locally, even using the gitlab-runner):

 • Installing multidict (5.0.0)
  Stack trace:
  12  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:202 in _execute_operation
      result = self._do_execute_operation(operation)
  11  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:276 in _do_execute_operation
      result = getattr(self, "_execute_{}".format(method))(operation)
  10  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:411 in _execute_install
      return self._install(operation)
   9  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:437 in _install
      archive = self._download(operation)
   8  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:580 in _download
      return self._download_link(operation, link)
   7  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:589 in _download_link
      archive = self._download_archive(operation, link)
   6  /usr/local/lib/python3.8/site-packages/poetry/installation/executor.py:615 in _download_archive
      response = self._authenticator.request(
   5  /usr/local/lib/python3.8/site-packages/poetry/installation/authenticator.py:66 in request
      settings = session.merge_environment_settings(
   4  /usr/local/lib/python3.8/site-packages/requests/sessions.py:711 in merge_environment_settings
      env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
   3  /usr/local/lib/python3.8/site-packages/requests/utils.py:775 in get_environ_proxies
      return getproxies()
   2  /usr/local/lib/python3.8/urllib/request.py:2502 in getproxies_environment
      for name, value in os.environ.items():
   1  /usr/local/lib/python3.8/_collections_abc.py:744 in __iter__
      yield (key, self._mapping[key])
  KeyError
  'VIRTUAL_ENV'
  at /usr/local/lib/python3.8/os.py:675 in __getitem__
       671│         try:
       672│             value = self._data[self.encodekey(key)]
       673│         except KeyError:
       674│             # raise KeyError with the original key value
    →  675│             raise KeyError(key) from None
       676│         return self.decodevalue(value)
       677│ 
       678│     def __setitem__(self, key, value):
       679│         key = self.encodekey(key)

On the other hand, locally I have the environment variable defined:

❯ echo $VIRTUAL_ENV
/Users/cglacet/.pyenv/versions/3.7.7/envs/env-name

The suggested fix did work for me too:

poetry config experimental.new-installer false