Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Passing ObjectID as a function argument in local_mode is broken

See original GitHub issue

System information

OS Platform and Distribution: Ubuntu 18.04
Ray installed from (source or binary): binary
Ray version: 0.8.0.dev6
Python version: 3.7
Exact command to reproduce: see below

Describe the problem

The argument passing behavior with local_mode=True vs False seems to be different. When I run the code snippet below:

ray.init(local_mode=True)  # Replace with False to get a working example

@ray.remote
def remote_function(x):
    obj = x['a']
    return ray.get(obj)


a = ray.put(42)
d = {'a': a}
result = remote_function.remote(d)
print(ray.get(result))

With local_mode=False I get output 42, as expected. With local_mode=True I get the following error:

Traceback (most recent call last):
  File "/home/alex/all/projects/doom-neurobot/playground/ray_local_mode_bug.py", line 13, in <module>
    print(ray.get(result))
  File "/home/alex/miniconda3/envs/doom-rl/lib/python3.7/site-packages/ray/worker.py", line 2194, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(KeyError): /home/alex/miniconda3/envs/doom-rl/bin/python /home/alex/all/projects/doom-neurobot/playground/ray_local_mode_bug.py (pid=2449, ip=10.136.109.38)
  File "/home/alex/miniconda3/envs/doom-rl/lib/python3.7/site-packages/ray/local_mode_manager.py", line 55, in execute
    results = function(*copy.deepcopy(args))
  File "/home/alex/all/projects/doom-neurobot/playground/ray_local_mode_bug.py", line 7, in remote_function
    return ray.get(obj)
  File "/home/alex/miniconda3/envs/doom-rl/lib/python3.7/site-packages/ray/local_mode_manager.py", line 105, in get_objects
    raise KeyError("Value for {} not found".format(object_id))
KeyError: 'Value for LocalModeObjectID(89f92e430883458c8107c10ed53eb35b26099831) not found'

It looks like the LocalObjectID instance inside d loses it’s field value when it gets deep copied during the “remote” function call (currently it’s local_mode_manager.py:55). It’s hard to tell why exactly that happens, looks like a bug.