Worker print output vanishes if not flushed (or if worker unexpectedly dies)

import sys
import ray

@ray.remote
class A:
    def __init__(self):
        print("print 1")
        sys.exit(1)

@ray.remote
class B:
    def __init__(self):
        pass
    def f(self):
        print("print 2")
        sys.exit(1)

@ray.remote
def f():
    print("print 3")
    sys.exit(1)


ray.init()
A.remote()
try:
    ray.get(B.remote().f.remote())
except:
    pass
try:
    ray.get(f.remote())
except:
    pass

This won’t print anything at all, and used to print all of 1, 2, 3.

[javierabosch2]

I believe I am having these same issues occurring in ray 1.1.0

After a worker errors out, and I attempt to run the job again, I cannot neither print to the console or log warnings using log.warning

[robertnishihara]

Ok, there are two issues here.

1. For the workload in the first comment, the prints actually do happen if the script stays alive, but if the driver exits before the prints get streamed from the log monitor to the driver then they don’t get printed.

2. For the Tune workload in https://github.com/ray-project/ray/issues/4082#issuecomment-464623602, the driver was blocked in ray.wait which didn’t release the GIL and so the logs weren’t getting printed. Fixed in #4190.