Building from source not working?

Describe the bug

I ran the following commands in https://scikit-learn.org/stable/developers/advanced_installation.html to build from source. I setup a conda environment using mini forge, and tried the make clean and then pip install ... sequence.

However, if I run make, then everything works and scikit-learn builds successfully.

Steps/Code to Reproduce

conda create -n sklearn-dev -c conda-forge python numpy scipy cython \
    joblib threadpoolctl pytest compilers llvm-openmp
conda activate sklearn-dev
make clean
pip install --verbose --no-build-isolation --editable .

then try

make

Expected Results

I presume no error should be thrown?

Actual Results

I get the following error.

(sklearn-dev) adam2392@Adams-MBP-2 scikit-learn % make clean
rm -f tags
python setup.py clean
Partial import of sklearn during the build process.
running clean
Will remove generated .c files
rm -rf dist
(sklearn-dev) adam2392@Adams-MBP-2 scikit-learn % pip install --verbose --no-build-isolation --editable .
Using pip 21.1.2 from /Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-ephem-wheel-cache-9wc424h4
Created temporary directory: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg
Initialized build tracking at /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg
Created build tracker: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg
Entered build tracker: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg
Created temporary directory: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-install-b_zwexn3
Obtaining file:///Users/adam2392/Documents/scikit-learn
  Added file:///Users/adam2392/Documents/scikit-learn to build tracker '/private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg'
    Created temporary directory: /private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-modern-metadata-0tvvc2yq
    Running command /Users/adam2392/miniforge3/envs/sklearn-dev/bin/python3.9 /Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/tmpsoy0n1z_
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 280, in <module>
        main()
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/setuptools/build_meta.py", line 266, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/setuptools/build_meta.py", line 158, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 54, in <module>
        from sklearn.externals._packaging.version import parse as parse_version  # noqa
    ModuleNotFoundError: No module named 'sklearn.externals._packaging'
    Preparing wheel metadata ... error
WARNING: Discarding file:///Users/adam2392/Documents/scikit-learn. Command errored out with exit status 1: /Users/adam2392/miniforge3/envs/sklearn-dev/bin/python3.9 /Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/tmpsoy0n1z_ Check the logs for full command output.
ERROR: Command errored out with exit status 1: /Users/adam2392/miniforge3/envs/sklearn-dev/bin/python3.9 /Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/tmpsoy0n1z_ Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 204, in wrapper
    return func(self, options, args)
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 444, in make_requirement_from_install_req
    raise self._build_failures[ireq.link]
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 334, in __init__
    super().__init__(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 345, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 622, in prepare_editable_requirement
    dist = _get_prepared_distribution(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 60, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 36, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 563, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 548, in _generate_metadata
    return generate_metadata(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 31, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 184, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
    self._subprocess_runner(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 274, in runner
    call_subprocess(
  File "/Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 244, in call_subprocess
    raise InstallationSubprocessError(proc.returncode, command_desc)
pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: /Users/adam2392/miniforge3/envs/sklearn-dev/bin/python3.9 /Users/adam2392/miniforge3/envs/sklearn-dev/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/tmpsoy0n1z_ Check the logs for full command output.
Removed file:///Users/adam2392/Documents/scikit-learn from build tracker '/private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg'
Removed build tracker: '/private/var/folders/6_/sl83qtkd68x3_mvfys07_6qm0000gn/T/pip-req-tracker-nuvhs9dg'

Versions

After I get things working with just make, I get:

(sklearn-dev) adam2392@Adams-MBP-2 scikit-learn % python -c "import sklearn; sklearn.show_versions()"

System:
    python: 3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:10:52)  [Clang 11.1.0 ]
executable: /Users/adam2392/miniforge3/envs/sklearn-dev/bin/python
   machine: macOS-11.5.2-arm64-arm-64bit

Python dependencies:
          pip: 21.1.2
   setuptools: 49.6.0.post20210108
      sklearn: 1.1.dev0
        numpy: 1.20.3
        scipy: 1.6.3
       Cython: 0.29.23
       pandas: 1.2.5
   matplotlib: 3.4.2
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True

[ogrisel]

Sorry for that, I thought our CI pipeline would have caught this kind of problems…

In the meantime the following should work:

pip install -v --no-build-isolation -e "git://github.com/scikit-learn/scikit-learn.git@1.0#egg=scikit-learn"

[adrinjalali]

Seems like the issue is fixed for the OP, and that it’s an environment issue in most cases. Happy to reopen if we can reproduce the issue.