[ray] Better UX for finding local modules

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• Ray installed from (source or binary): pip
• Ray version: latest
• Python version: 3.5
• Exact command to reproduce: see linked repo

1. clone this repo
2. cd to project folder
3. jupyter-notebook <cloned-project>/jupyter/
4. just run the cells

Describe the problem

In the linked repo I provide a M.W.E. of how I envision sacred and tune would work together. Inside there is a local python module named project which contains four submodules model, utils, sacred and tune.

The model submodule has code for a simple twin embedding model in keras (simply to provide some “realism” to the MWE. This code functions and can be tested in the directory jupyter.

The sacred submodule, attempts to decouple sacred specific code as much as possible (e.g. ingredients, observers, etc). If you import the ex from this module it still runs on its own. (again, see the jupyter directory)

The tune submodule is where I envision how tune would wrap the sacred ex for hyperparameter optimization.

Most likely, as I get more familiar with tune (or for users who are) the amount of code here would expand. Nonetheless there are two simple functions, one which wraps sacred, and another which registers the wrapper and expects a hyper-parameter configuration to be passed.

When calling this function ray / tune fails as:

(pid=40079) ModuleNotFoundError: No module named 'project'

which seems to be because of this python module structure.

Notably, if all this code is put at the top level of a module, this works.

I would like to keep this structure though, and I do not see why tune should or would be opinionated about that.

Source code / logs

repo at https://gitlab.com/SumNeuron/extune

[SumNeuron]

@richardliaw @robertnishihara 's suggestion to use a setup.py file and pip install -e is a sufficient work around…

However I personally dislike both approaches. In general python style guides encourage all import statements at the top of the file.

Having to setup and manage the module shouldn’t be necessary either.

Please clone the repo and try the jupyter notebook as well as the “script.py” file. I ask this not just because it may help in figuring out exactly why these workarounds are necessary, but because I added a fair amount of documentation throughout the functions. So it might serve as as good starting point should ray / tune want to add an example or official docs on sacred integration

I think the true solution requires the ray module to become relative / local module aware.

[kschuerholt]

Sure. Basically, you make your local module a package. Instead of importing it locally from python path, you install it with pip. that way, it becomes available, even if the package isn’t in your path. To that end, create a file called ‘setup.py’ in the package you want to install. There’s a lot of info on that online, you’ll find all the details you need. It needs to contain the import of setup,

from setuptools import setup

setup(
    name="name_of_your_package",
    version="0.1.2.3",
    author="Author Name",
    author_email="[e-mail]",
    packages=["module_1", "module_2"],
    description="An awesome package that does something",
)

AFAIK that the modules in packages need to be python modules, so they have to be in separate folders with corresponding names and have to have __init__.py files. If you have that, just navigate to the directory with the setup.py and run pip install -e ., which will install your new package. When you import your package, it imports the actual code that is in the directory, so you don’t need to reinstall when you change your code. Read up on packages e.g. here: https://betterscientificsoftware.github.io/python-for-hpc/tutorials/python-pypi-packaging/