[Core] [Error Message] Possible unhandled error from worker

When you have a remote function that handles an exception, a Possible unhandled error message is still printed on the driver side. For example,

import time
import ray

ray.init()

@ray.remote
def foo():
    raise Exception("oops")

@ray.remote
def main():
    i = 0
    while True:
        i += 1
        time.sleep(1)
        print(i)
        try:
            ray.get(foo.remote())
        except:
            pass

ray.get(main.remote())

Shows the following message on the driver

2020-11-13 18:03:16,514	ERROR worker.py:1036 -- Possible unhandled error from worker: ray::foo() (pid=68380, ip=192.168.2.228)
  File "python/ray/_raylet.pyx", line 482, in ray._raylet.execute_task
  File "<stdin>", line 3, in foo
Exception: oops

even though the exception is actually being properly handled in the remote function main. This is very confusing for the user because it implies that an error actually occurs even though it doesn’t. At the very least we should emphasize that it is a Possible unhandled error and that it can be safe to ignore this message if it is being handled.

See https://github.com/ray-project/ray/issues/5240 and https://github.com/ray-project/ray/pull/11849#issuecomment-727111327 for more examples.

[amogkam]

@valiantljk the message is being shown even though no error actually occurs. Since you are using ParallelIterators, it is safe to ignore this.

It looks like you are using ParallelIterators inside a Ray task, correct? The reason why you are seeing this message is because the error is being handled in the remote worker, but not in the driver, so the message is erroneously being printed on the driver side. This is a bug on our end, and we will fix it shortly!

[oliverhu]

👌 @rkooo567