Console output got relipated n times if there are n ray nodes on the same machine

What is the problem?

Ray version: 0.8.7 Python version: 3.6.7

Problem:

If multiple ray nodes (say n ray nodes) are started on the same machine, then one ray nodes’ console output will got replicated n times on driver.

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

Step 1: start two ray nodes on the same machine

ray start --head
ray start --address='the address printed' --redis-password='the password printed'

Step 2: run the following script

import ray
def print_hello():
    print("hello")
    return 0

remote_print = ray.remote(print_hello)
ray.get(remote_print.remote())

Then “hello” will be printed twice on the driver.

If we cannot run your script, we cannot fix your issue.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[yangw1234]

@rkooo567 https://github.com/ray-project/ray/blob/master/python/ray/node.py#L233, seems non-head ray process just read the “temp_dir” set to head node, and each ray node cannot be assigned to different log dir.

I worked around this issue by manually killing the redundant log_monitor processes. However, I think it would be nice to support it in ray. E.g. separating each ray nodes’ log dir to “/tmp/ray/session-xx/0/logs”, “/tmp/ray/session-xxx/1/logs”, .etc

[wuisawesome]

I’ll close this issue for now since you will open a new issue with the repro