[Bug] Specifying runtime_env leads to failure on Apple ARM64

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

Having recently obtained an Apple ARM laptop, I’m in the process of bringing up my development environment on it. I’ve followed https://docs.ray.io/en/master/installation.html#m1-mac-apple-silicon-support to set up, and verified that ray start --head and ray microbenchmark work, but my pipeline does not.

I’ve narrowed it down to the runtime_env argument to ray.init(), which leads to this message:

% python foo.py
Traceback (most recent call last):
  File "/Users/josephlong/devel/test/ray_bug/foo.py", line 14, in <module>
    res = ray.get(ref)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/worker.py", line 1627, in get
    raise value
ray.exceptions.RuntimeEnvSetupError: The runtime environment for this task or actor failed to be installed. Corresponding error logs should have been streamed to the driver's STDOUT.
(raylet) [2021-11-21 12:20:27,923 E 56546 3176166] agent_manager.cc:134: Not all required Ray dependencies for the runtime_env feature were found. To install the required dependencies, please run `pip install 'ray[default]'`.

Versions / Dependencies

ray 1.8 Python 3.9 macOS 12 Monterey

Reproduction script

import ray

ray_init_kwargs = {'runtime_env': {'env_vars': {'RAY_USER_SETUP_FUNCTION': 'logging.basicConfig'}}}
# ray_init_kwargs = {}  # comment above and uncomment this and it'll work

@ray.remote
def bar(a):
    print(a)
    return a

if __name__ == "__main__":
    ray.init(**ray_init_kwargs)
    ref = bar.remote(1)
    res = ray.get(ref)
    print(res)

Anything else

I had a look at the code that produces that error and was unable to figure out which dependency it needed. It may be that this is just a dependency issue…

[joseph-long]

Thanks, that’s the diagnostic I was missing! Not an ARM64 thing at all, it turns out. It’s just https://github.com/ray-project/ray/issues/19940 again.

>>> import ray.dashboard.optional_deps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/dashboard/optional_deps.py", line 8, in <module>
    import aiohttp.signals
ModuleNotFoundError: No module named 'aiohttp.signals'

Fixed with pip install 'aiohttp<3.8'

(Would be great if that ImportError output were part of the Corresponding error logs should have been streamed to the driver's STDOUT. 😃)

[joseph-long]

% RAY_BACKEND_LOG_LEVEL=debug python foo.py
[2021-11-22 17:11:38,014 I 94310 3885808] logging.cc:190: Set ray log level from environment variable RAY_BACKEND_LOG_LEVEL to -1
Traceback (most recent call last):
  File "/Users/josephlong/devel/test/ray_bug/foo.py", line 14, in <module>
    res = ray.get(ref)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/josephlong/mambaforge/envs/py39/lib/python3.9/site-packages/ray/worker.py", line 1627, in get
    raise value
ray.exceptions.RuntimeEnvSetupError: The runtime environment for this task or actor failed to be installed. Corresponding error logs should have been streamed to the driver's STDOUT.
(raylet) [2021-11-22 17:11:38,150 E 94316 3885842] agent_manager.cc:134: Not all required Ray dependencies for the runtime_env feature were found. To install the required dependencies, please run `pip install 'ray[default]'`.