How to submit a task which is implemented in a separate module?

Evironment

• python version: 3.7.6
• ray version: 0.8.1
• OS of driver: ubuntu 18.04
• OS of worker: ubuntu 18.04

Hi, I met a problem recently when I was using ray. I want to let my remote function run in the remote node. This function was not written in the main script file but in a separate file. However, I get a exception said: This function was not imported properly.

This is what my script is written:

## module-test.py

import ext_module
import ray

@ray.remote
def other_module():
    return ext_module.cac_sqrt3()

if __name__ == '__main__':
    ray.init(address='auto', redis_password='5241590000000000')
    res = other_module.remote()
    print("result is:", ray.get(res))

## ext_module.py

def cac_sqrt3():
    return 1.732

I build a cluster of three nodes (one head node and two other nodes), to make sure the task is not run in the local machine when I connect the cluster, I set num-cpus=0

ray start --address='xxxxxxx' --redis-password='5241590000000000' --num-cpus=0

This is what the error log like:

2020-02-12 12:12:23,627	WARNING worker.py:1063 -- Failed to unpickle the remote function '__main__.other_module' with function ID bb6e6179f5b2ef0eb388e72b4492bf508bc9ed39. Traceback:
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/ray_dev/lib/python3.7/site-packages/ray/function_manager.py", line 402, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/xxx/miniconda3/envs/ray_dev/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle.py", line 1136, in subimport
    __import__(name)
ModuleNotFoundError: No module named 'ext_module'


Traceback (most recent call last):
  File "module-test.py", line 20, in <module>
    print("result is:", ray.get(res))
  File "/home/xxx/anaconda3/envs/ray_dev2/lib/python3.7/site-packages/ray/worker.py", line 1492, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError: ray::__main__.other_module() (pid=16330, ip=xxxxxx)
  File "python/ray/_raylet.pyx", line 640, in ray._raylet.execute_task
Exception: This function was not imported properly.

What should I do to make it run normally?

[MartinBubel]

I encountered this issue as well. I found a workaround which might be far away from a best-practice style but might be helpful for anyone looking for a simple fix:
For me, moving the sys.path.append and the respective import ... to the @ray.remote function

@ray.remote
def some_function():
    sys.path.append('...')
    import ...
    ...

fixed the issue. That workaround for sure is inefficient and kind of ugly but may do the job for anyone simply trying to get some code running on a cluster.

[architkulkarni]

Runtime Environments should be the best-practice solution for this issue–please let us know if there’s any gap in the functionality! https://docs.ray.io/en/latest/handling-dependencies.html#runtime-environments. (See the working_dir and py_modules fields)