[Bug] ERROR gcs_utils.py:137 – Failed to send request to gcs

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

On ray start on worker I expected the head and worker to be connected, but instead I see this error only on ray 1.9 Worker can talk to the head in ray version 1.8.

$ ray start --address='100.96.24.172:6379' --redis-password='5241590000000000'
Local node IP: 100.96.191.45
2022-02-05 16:39:15,662 ERROR gcs_utils.py:137 -- Failed to send request to gcs, reconnecting. Error <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1644079155.661725492","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3134,"referenced_errors":[{"created":"@1644079155.661724203","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>
2022-02-05 16:39:16,665 ERROR gcs_utils.py:137 -- Failed to send request to gcs, reconnecting. Error <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1644079156.664823970","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3134,"referenced_errors":[{"created":"@1644079156.664822821","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>

Versions / Dependencies

• Ray 1.9
• Python 3.7.10
• OS Linux

Reproduction script

Install the above versions on 2 nodes.

Start the head node using ray.init()

Start the worker node using `ray start --address=… --redis-password=…

Anything else

https://discuss.ray.io/t/error-gcs-utils-py-137-failed-to-send-request-to-gcs/4936

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[mwtian]

@Nithanaroy I sent a message to you on discuss.ray.io to setup a debugging session. What I want to investigate together:

1. See if the issue exists in Ray 1.10.0
2. After starting Ray 1.9 head node, use Redis-cli to try to read the value of key GcsServerAddress from the head node address, e.g. 100.96.24.172:6379. Verify if it is set correctly.
3. curl the GCS address from the worker node.

Please let us know if having a debugging session would help!

[mwtian]

Closing this issue since there is a workaround. But feel free to reopen if you think this use case needs to be supported!