Exception when adding apache-beam[gcp] to empty project

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 12.0.1
• Poetry version: 1.1.12, 1.2.0a2 (tried both)
• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

I’m receiving an exception when trying to add apache-beam[gcp] to a new and empty project. It is possible that the issue resides with the library itself and not Poetry. If that’s the case I’ll make sure to open a ticket for them instead 😃

The issue can easily be replicated with Poetry by creating a new empty folder, and running:

poetry init
poetry add apache-beam[gcp]

This is the full exception I received:

 • Installing apache-beam (2.34.0): Failed

  CalledProcessError

  Command '['/Users/guyzyl/sightsource/test/.venv/bin/python', '/usr/local/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/guyzyl/sightsource/test/.venv', '--no-deps', 'file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip']' returned non-zero exit status 1.

  at /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/Users/guyzyl/sightsource/test/.venv/bin/python', '/usr/local/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-21.0.1-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/guyzyl/sightsource/test/.venv', '--no-deps', 'file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip'] errored with the following return code 1, and output:
  Processing /Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh
         cwd: /private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-req-build-vua9jsg4
    Complete output (26 lines):
    Traceback (most recent call last):
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 280, in <module>
        main()
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py", line 114, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
        return self._get_build_requires(
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
        self.run_setup()
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 267, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 158, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 97, in <module>
        _PIP_VERSION = get_distribution('pip').version
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 471, in get_distribution
        dist = get_provider(dist)
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 347, in get_provider
        return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 891, in require
        needed = self.resolve(parse_requirements(requirements))
      File "/private/var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/pip-build-env-xae9op1i/overlay/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'pip' distribution was not found and is required by the application
    ----------------------------------------
  WARNING: Discarding file:///Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip. Command errored out with exit status 1: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh Check the logs for full command output.
  ERROR: Command errored out with exit status 1: /Users/guyzyl/sightsource/test/.venv/bin/python /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpj7fug9l2_in_process.py get_requires_for_build_wheel /var/folders/ww/qcz5f6hs42v4h9t9gdm4pcz00000gn/T/tmpyf7pddkh Check the logs for full command output.


  at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1300 in _run
      1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )
      1299│         except CalledProcessError as e:
    → 1300│             raise EnvCommandError(e, input=input_)
      1301│
      1302│         return decode(output)
      1303│
      1304│     def execute(self, bin: str, *args: str, **kwargs: Any) -> Optional[int]:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install file:/Users/guyzyl/Library/Caches/pypoetry/artifacts/a6/78/23/9338a4b21c37427c5b18dfbd31d03b7b5240a24aa9d82bc6182708fb9d/apache-beam-2.34.0.zip

  at /usr/local/lib/python3.9/site-packages/poetry/utils/pip.py:60 in pip_install
       56│                     *env.get_pip_command(),
       57│                     *args,
       58│                     env={**os.environ, "PYTHONPATH": str(env.purelib)},
       59│                 )
    →  60│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       61│
       62│
       63│ def pip_editable_install(directory: Path, environment: Env) -> Union[int, str]:
       64│     return pip_install(

[oleksandr-yatsuk]

In my case, the issue was related to the experimental installer

poetry config experimental.new-installer false

fixed the issue

[dimbleby]

apache-beam is making a wholly redundant check that a recent version of pip is available in its environment. But

• this need not be true (as in this example, where the pip that is being run is in a different virtual environment altogether)
• the “recent” version being checked for was last updated in 2016, we can reasonably bet that no-one is using an older pip than that.

Suggest that anyone who cares about this issue raise an MR at apache-beam removing this block of code

https://github.com/apache/beam/blob/7ad4864b0cb19b6c8405265f84fff24bf5b2c8b3/sdks/python/setup.py#L93-L101