Poetry build does not install build dependencies (affects both install and build)

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: OS X 10.15.5 (also tested in docker on ubuntu 16.04)
• Poetry version: 1.1.0b2
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/mdgoldberg/96a9955c8c823f2127dbc471a954b898

Issue

I created a repo with a minimal reproduction: https://github.com/mdgoldberg/poetry-cython-example

The problem here is that Cython isn’t installed prior to building/installing the root package, even though Cython is specified in build-system.requires in the pyproject.toml. Here’s how I am reproducing the issues, for both poetry install and poetry build:

$ git clone [e-mail]:mdgoldberg/poetry-cython-example.git
$ cd poetry-cython-example
$ poetry install -vvv
Creating virtualenv poetry-cython-example in /Users/mattgoldberg/poetry-cython-example/.venv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Installing the current project: poetry-cython-example (0.1.0)
  - Building package poetry-cython-example in editable mode
  - Executing build script: build.py
Traceback (most recent call last):
  File "/Users/mattgoldberg/poetry-cython-example/build.py", line 6, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
  - Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
  - Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
  - Building wheel
Traceback (most recent call last):
  File "build.py", line 6, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

  CalledProcessError

  Command '['/Users/mattgoldberg/poetry-cython-example/.venv/bin/python', 'build.py']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.6.10/lib/python3.6/subprocess.py:311 in check_call
       307│     if retcode:
       308│         cmd = kwargs.get("args")
       309│         if cmd is None:
       310│             cmd = popenargs[0]
    →  311│         raise CalledProcessError(retcode, cmd)
       312│     return 0
       313│
       314│
       315│ def check_output(*popenargs, timeout=None, **kwargs):
$ # now install Cython and they work
$ pip install Cython
Collecting Cython
  Using cached Cython-0.29.21-cp36-cp36m-macosx_10_9_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.21
$ poetry install -vvv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Installing the current project: poetry-cython-example (0.1.0)
  - Building package poetry-cython-example in editable mode
  - Executing build script: build.py
  - Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
  - Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
  - Building wheel
  - Built poetry_cython_example-0.1.0-cp36-cp36m-macosx_10_15_x86_64.whl

Note that for the fix, all that matters is whether Cython is installed in the “current” Python environment. If I install Cython globally but then activate the virtual env (which doesn’t have Cython), then poetry install and poetry build still fail. Conversely, if I install Cython in the virtualenv and try to poetry install or poetry build from the directory but outside the virtualenv, it also fails.

Please, let me know if I’m misusing build.py or any other features, I know some of them may not be stabilized. For now, I’m just installing Cython before running poetry build or poetry install` in my Dockerfile, but it really seems like I shouldn’t need to, per PEP 518. Let me know if I’m misunderstanding, and thank you for developing Poetry!

[espdev]

… for now I will just pip install Cython before building.

@mdgoldberg,

Unfortunately, this does not work if you want to publish and distribute your package. A user will need to install build dependencies for your package manually. It is not a good way.

For example, I need to publish a library with C++ pybind11 based extension modules. The library is distributed in prebuilt wheels for many platforms. I can add pybind11 to dev dependencies for development and building wheels. But also I want to provide sdist with source code, so that a user can install it from the source. And I don’t want to require pybind11 dependency always. Only for installing from the sources. pybind11 is not required for prebuilt wheels. I need to build dependencies here, dev dependencies do not work in this case.

[matteosantama]

At least what’s working for me is adding cython to tool.poetry.dev-dependencies. That shouldn’t be necessary, i don’t think, but it’s kinda true at least.

I would like to add that this workaround no longer works in 1.2.0a2. Hopefully the 1.2.x series will address this issue properly!