`poetry build -f wheel` broken for C extensions

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Issue

I’m trying to build a library that includes a C extension.

• poetry build works and creates both .whl and .tar.gz
• poetry build -f sdist works and creates .tar.gz
• poetry build -f wheel fails

Error output

… ~/awaitwhat> poetry build -vvv --f wheel
Using virtualenv: /Users/xxx/Library/Caches/pypoetry/virtualenvs/awaitwhat-QZ4jZTXn-py3.7
Building awaitwhat (1.0)
 - Building wheel
 - Adding: /Users/xxx/awaitwhat/awaitwhat/__init__.py
 - Adding: /Users/xxx/awaitwhat/awaitwhat/what.c

[EnvCommandError]
Command ['/Users/xxx/Library/Caches/pypoetry/virtualenvs/awaitwhat-QZ4jZTXn-py3.7/bin/python', '/Users/xxx/awaitwhat/setup.py', 'build', '-b', '/Users/xxx/awaitwhat/build'] errored with the following return code 2, and output:
/Library/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/xxx/awaitwhat/setup.py': [Errno 2] No such file or directory


Traceback (most recent call last):
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 132, in run
    status_code = command.handle(parsed_args, io)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 119, in handle
    status_code = self._do_handle(args, io)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 167, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/Users/xxx/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/Users/xxx/.poetry/lib/poetry/console/commands/build.py", line 30, in handle
    builder.build(fmt)
  File "/Users/xxx/.poetry/lib/poetry/masonry/builder.py", line 21, in build
    return builder.build()
  File "/Users/xxx/.poetry/lib/poetry/masonry/builders/wheel.py", line 81, in build
    self._build(zip_file)
  File "/Users/xxx/.poetry/lib/poetry/masonry/builders/wheel.py", line 104, in _build
    'python', str(setup), 'build', '-b', str(self._path / 'build')
  File "/Users/xxx/.poetry/lib/poetry/utils/env.py", line 875, in run
    return super(VirtualEnv, self).run(bin, *args, **kwargs)
  File "/Users/xxx/.poetry/lib/poetry/utils/env.py", line 708, in run
    raise EnvCommandError(e, input=input_)

Environment

… ~/awaitwhat> uname
Darwin
… ~/awaitwhat> poetry --version
Poetry version 1.0.0b1

Source code

https://github.com/dimaqq/awaitwhat/tree/poetry/issues/1332 (I’ve tagged my library poetry/issues/1332 at the point where issue can be reproduced)

[jhrmnn]

A hack that fixes the PEP 517 issue is to have this in pyproject.toml:

[build-system]
backend-path = ["."]
build-backend = "patched_poetry_masonry_api"

and then this in patched_poetry_masonry_api.py:

import sys
import os
import tarfile
from contextlib import contextmanager
from pathlib import Path

from clikit.io import NullIO

from poetry.factory import Factory
from poetry.utils._compat import unicode
from poetry.utils.env import SystemEnv
from poetry.utils.helpers import temporary_directory

from poetry.masonry.builders.sdist import SdistBuilder
from poetry.masonry.builders.wheel import WheelBuilder
from poetry.masonry.api import *


@contextmanager
def _unpacked_tarball(path):
    tf = tarfile.open(str(path))

    with temporary_directory() as tmpdir:
        tf.extractall(tmpdir)
        files = os.listdir(tmpdir)

        assert len(files) == 1, files

        yield Path(tmpdir) / files[0]


def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
    """Builds a wheel, places it in wheel_directory"""

    poetry = Factory().create_poetry(Path('.'))
    env = SystemEnv(Path(sys.prefix))

    with temporary_directory() as tmpdir_sdist:
        sdist_file = SdistBuilder(poetry, env, NullIO()).build(
            Path(tmpdir_sdist)
        )
        with _unpacked_tarball(sdist_file) as tmpdir:
            return unicode(
                WheelBuilder.make_in(
                    Factory().create_poetry(tmpdir),
                    env,
                    NullIO(),
                    Path(wheel_directory),
                    original=poetry,
                )
            )

The latter is just a patch of poetry.masonry.api.build_wheel() that does what CompleteBuilder does, that is building the wheel from a temporary source distribution.

[jhrmnn]

If I understand the issue correctly, the problem is that for packages with a build option, WheelBuilder actually calls python setup.py build to build the C extensions. This works with poetry build, because that uses CompleteBuilder, which creates a source distribution first (which generates setup.py), and then creates the wheel from that distribution (so at that point setup.py already exists—the generated one). But with poetry build -f wheel, setup.py is not generated.

So I don’t think this is a bug, it’s just that this scenario (building a wheel from the source tree without building a source distribution first, for packages with C extensionos) has never been considered at all by the current code.

This also causes the PEP 517 build too fail, because Pip apparently attempts to create the wheel directly from the source tree, even though PEP 517 says:

To ensure that wheels from different sources are built the same way, frontends may call build_sdist first, and then call build_wheel in the unpacked sdist. But if the backend indicates that it is missing some requirements for creating an sdist (see below), the frontend will fall back to calling build_wheel in the source directory.

@sdispater What would be the course of action here? Could we add a safe check that ensures that the wheel builder always uses a generated source distribution rather than the source tree if the package contains the build option?