build meta wheel does not respect metadata_directory per PEP-517

From https://www.python.org/dev/peps/pep-0517/#build-wheel:

If the build frontend has previously called prepare_metadata_for_build_wheel and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created .dist-info directory as the metadata_directory argument.

https://github.com/pypa/setuptools/blob/master/setuptools/build_meta.py#L208 though ignores the passed in metadata_directory argument and goes to regenerate this information in the root of the project.

    def build_wheel(self, wheel_directory, config_settings=None,
                    metadata_directory=None):
        return self._build_with_temp_dir(['bdist_wheel'], '.whl',
                                         wheel_directory, config_settings)

[uranusjr]

But then what happens if a backend return different things from build_wheel and prepare_metadata_for_build_wheel? (Which setuptools is guity of doing right now.) It is certainly possible to remove the requirement for backend to make use of the metadata_directory argument, but the backend should still somehow return consistent things from the two hooks. Otherwise the frontend would be stuck and can’t proceed. The discussion needs to be moved to discuss.python.org if you want to do that so frontend authors can join the conversation.

[wimglenn]

setuptools-ext is a thin wrapper around setuptools.build_meta allowing to specify some more metadata fields such as Requires-External, which aren’t otherwise supported by setuptools. Passing around the metadata_directory (ref) would have been really useful if it worked!

If setuptools were to diligently follow PEP 517 (i.e. does not just ignore the metadata_directory), we could simply call prepare_metadata_for_build_wheel before dropping in some additional files, or modify existing ones, under .dist-info subdir, and then call build_wheel passing in the metadata dir.

However since that didn’t work, instead setuptools-ext has to call build_wheel, and then completely rewrite the result, which is a lot more complicated (considering that modifying the METADATA file also invalidates the RECORD file!)

This “tweaking” of a build backend by wrapping the hooks is apparently an expected usage, because it’s documented as a way for an in-tree build backend to augment the build dependencies (by @abravalheri here). It’s unfortunate that one can’t reliably use a similar technique for prepare_metadata_for_build_wheel + build_wheel.