Error: package directory 'nni_node' does not exist when pip installing from GitHub

Describe the issue:

Hi there, just starting to use nni and was interested in trying some features in the master branch, but couldn’t get the install to work. The error message is error: package directory 'nni_node' does not exist.

Environment:

• NNI version: https://github.com/microsoft/nni/commit/cbb63c5bec618354a25583c0861f45d4a01d9812
• Training service (local|remote|pai|aml|etc): local
• Client OS: LInux
• Server OS (for remote mode only):
• Python version: 3.9
• PyTorch/TensorFlow version: 1.11
• Is conda/virtualenv/venv used?: conda/mamba
• Is running in Docker?: No

Configuration:

• Experiment config (remember to remove secrets!):
• Search space:

Log message:

• nnimanager.log:
• dispatcher.log:
• nnictl stdout and stderr:

How to reproduce it?:

Run pip install git+https://github.com/microsoft/nni.git. This produces the error:

Collecting git+https://github.com/microsoft/nni.git
  Cloning https://github.com/microsoft/nni.git to /tmp/pip-req-build-2m66e716
  Running command git clone --filter=blob:none --quiet https://github.com/microsoft/nni.git /tmp/pip-req-build-2m66e716
  Resolved https://github.com/microsoft/nni.git to commit cbb63c5bec618354a25583c0861f45d4a01d9812
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running egg_info
      creating /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info
      writing /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/entry_points.txt
      writing requirements to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/requires.txt
      writing top-level names to /tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/top_level.txt
      writing manifest file '/tmp/pip-pip-egg-info-g9bbwyzp/nni.egg-info/SOURCES.txt'
      package init file 'nni/algorithms/compression/tensorflow/__init__.py' not found (or not a regular file)
      package init file 'nni/algorithms/compression/v2/__init__.py' not found (or not a regular file)
      package init file 'nni/compression/pytorch/quantization/__init__.py' not found (or not a regular file)
      error: package directory 'nni_node' does not exist
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

For reference, I’m using pip=22.0.4, setuptools=62.1.0, wheel=0.37.1.

[matluster]

@liuzhe-lz Any comments on this?

P.S. When I looked at pyright the other day, I noticed an interesting way, that is to publish the ts package on npm, and maintain the python wrapper in another repo. We can also try that. That might solve the problem that we need a customized python setup.py develop.

[weiji14]

To my understanding build is a frontend, but we are missing the backend.

Came across this backend library called Hatch (https://hatch.pypa.io/1.5). The Jupyter folks seem to have some success using it recently to build their Javascript assets (see https://blog.jupyter.org/packaging-for-jupyter-in-2022-c7be64c38926?gi=dcb04ade667a and https://github.com/jupyterlab/hatch-jupyter-builder). Haven’t tried it personally, but just thought it might be of interest.