[Bug] runtime_env doesn't honor env_vars first

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Clusters

What happened + What you expected to happen

I set both env_vars and pip in runtime_env, one of the pip install need to access the user-defined env during runtime, but looks like the env is not honored and pip install failed, with errors:

File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/client/server/proxier.py", line 234, in _create_runtime_env
    "Failed to create runtime_env for Ray client "
RuntimeError: Failed to create runtime_env for Ray client server: Failed to install conda environment /tmp/ray/session_2022-02-23_13-49-17_742804_6/runtime_resources/conda/bda0b0d51deac4d51a7063a37ecbbc2a53c9e840:
raise error
RuntimeError: java home not found, try setting JAVA_HOME

Versions / Dependencies

ray 1.9.0 python 3.7

Reproduction script

import ray,os
runtime_env = {
        "env_vars": {"JAVA_HOME": "/opt/tiger/jdk/jdk1.8"},
       "pip": ["pydoop==2.0.0",],
 }

ray.init("ray://head:port",runtime_env=runtime_env)
print(ray.cluster_resources())
#@ray.remote
#def f():
#  return os.environ.get("JAVA_HOME")

#print(ray.get(f.remote())) # this line works as expected, i.e., JAVA_HOME was set correctly

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[grechaw]

Is this an issue with pydoop installation? I verified that JAVA_HOME gets set correctly for both Ray 1.9 and 1.10… but what’s possible here is that it’s not set at the time that pip installs pydoop

[architkulkarni]

Closed by #22730