[Bug] Jobs CLI with `ray.init()` fails on master with `AssertionError: Module: http does not have ClientBuilder.`

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Dashboard, Others

What happened + What you expected to happen

| your_working_directory ("./")
| ├── script.py

# script.py
import ray
ray.init()

Then run

ray start --head
export RAY_ADDRESS=http://127.0.0.1:8265
ray job submit --working-dir="./" -- python script.py

as in the docs.

We get the following output:

ray start --head
export RAY_ADDRESS=http://127.0.0.1:8265
ray job submit --working-dir="./" -- python script.py
Local node IP: 127.0.0.1
2022-01-13 18:39:01,675 INFO services.py:1382 -- View the Ray dashboard at http://127.0.0.1:8265

--------------------
Ray runtime started.
--------------------

Next steps
  To connect to this Ray runtime from another node, run
    ray start --address='127.0.0.1:6379' --redis-password='5241590000000000'
  
  Alternatively, use the following Python code:
    import ray
    ray.init(address='auto', _redis_password='5241590000000000')
  
  To connect to this Ray runtime from outside of the cluster, for example to
  connect to a remote cluster from your laptop directly, use the following
  Python code:
    import ray
    ray.init(address='ray://<head_node_ip_address>:10001')
  
  If connection fails, check your firewall settings and network configuration.
  
  To terminate the Ray runtime, run
    ray stop
Job submission server address: http://127.0.0.1:8265
2022-01-13 18:39:03,348 INFO sdk.py:189 -- Uploading package gcs://_ray_pkg_867dc877fe33e555.zip.
2022-01-13 18:39:03,349 INFO packaging.py:362 -- Creating a file package for local directory './'.

-------------------------------------------------------
Job 'raysubmit_jx9F57bhMiqZP9UN' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_jx9F57bhMiqZP9UN
  Query the status of the job:
    ray job status raysubmit_jx9F57bhMiqZP9UN
  Request the job to be stopped:
    ray job stop raysubmit_jx9F57bhMiqZP9UN

Tailing logs until the job exits (disable with --no-wait):
Traceback (most recent call last):
  File "script.py", line 2, in <module>
    ray.init()
  File "/Users/archit/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/archit/ray/python/ray/worker.py", line 767, in init
    builder = ray.client(address, _deprecation_warn_enabled=False)
  File "/Users/archit/ray/python/ray/client_builder.py", line 347, in client
    builder = _get_builder_from_address(address)
  File "/Users/archit/ray/python/ray/client_builder.py", line 318, in _get_builder_from_address
    assert "ClientBuilder" in dir(module), (f"Module: {module_string} does "
AssertionError: Module: http does not have ClientBuilder.

---------------------------------------
Job 'raysubmit_jx9F57bhMiqZP9UN' failed
---------------------------------------

Traceback (most recent call last):
  File "/Users/archit/anaconda3/envs/ray-py38/bin/ray", line 11, in <module>
    load_entry_point('ray', 'console_scripts', 'ray')()
  File "/Users/archit/ray/python/ray/scripts/scripts.py", line 1952, in main
    return cli()
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/archit/ray/python/ray/autoscaler/_private/cli_logger.py", line 808, in wrapper
    return f(*args, **kwargs)
  File "/Users/archit/ray/python/ray/dashboard/modules/job/cli.py", line 177, in job_submit
    asyncio.get_event_loop().run_until_complete(
  File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/archit/ray/python/ray/dashboard/modules/job/cli.py", line 68, in _tail_logs
    _log_job_status(client, job_id)
  File "/Users/archit/ray/python/ray/dashboard/modules/job/cli.py", line 56, in _log_job_status
    cli_logger.print(f"Status message: {status.message}")
  File "/Users/archit/ray/python/ray/autoscaler/_private/cli_logger.py", line 589, in print
    self._print(_format_msg(msg, *args, **kwargs), _level_str=_level_str)
  File "/Users/archit/ray/python/ray/autoscaler/_private/cli_logger.py", line 238, in _format_msg
    return numbering_str + msg.format(*args, **kwargs) + tags_str
KeyError: 'module_string'

Versions / Dependencies

Ray master branch, macOS (non-M1), Python 3.8

Reproduction script

Listed above

Anything else

This happens nearly every time, but I haven’t noticed a pattern for when it succeeds and I unfortunately don’t remember what, if anything, I had tweaked anything to make it succeed. I’ll keep trying.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[grechaw]

A related issue is that ray job CLI does not work with anyscale urls. I wish it did… should I make a Ray project issue for this or a Anyscale one?

[starpit]

btw, we ran into this because we were setting, in a ray worker process, the RAY_ADDRESS env var to an http: address when running in-cluster. by removing that env var from the kube pod, this particular error went away.