[RFC] Change the behavior of Runtime Environments inheritance

Hi all: Runtime Environments is already GA in Ray 1.6.0. The latest doc is here. And now, we already supported a inheritance behavior as follows (copied from the doc):

• The runtime_env[“env_vars”] field will be merged with the runtime_env[“env_vars”] field of the parent. This allows for environment variables set in the parent’s runtime environment to be automatically propagated to the child, even if new environment variables are set in the child’s runtime environment.
• Every other field in the runtime_env will be overridden by the child, not merged. For example, if runtime_env[“py_modules”] is specified, it will replace the runtime_env[“py_modules”] field of the parent.

For example, if the parent runtime env is:

{
   "py_modules": "s3://ab.zip",
    "pip": ["requests", "pendulum==2.1.2"],
    "env_vars":{
        "a":"b",
    }
}

and the child runtime env is:

{
   "py_modules": "s3://cd.zip",
    "env_vars":{
        "c":"d",
    }
}

The final merged runtime env for child is:

{
   "py_modules": "s3://cd.zip",
    "pip": ["requests", "pendulum==2.1.2"],
    "env_vars":{
        "a":"b",
        "c":"d",
    }
}

We think this runtime env merging logic is so complex and confusing to users because users can’t know the final runtime env before the jobs are run.

Another case is that If you set a special conda env in parent actor and set a special pip env in child actor, you will get a ValueError exception because conda and pip are conflicting.

So, we want to do a refactor and change the behavior of Runtime Environments inheritance. Here is the new behavior:

• If there is no runtime env option when we create actor, inherit the parent runtime env.
• Otherwise, use the optional runtime env directly and don’t do the merging.

We think the behavior above could cover more than 90% scenes. If you really want to do a specific merging from parent runtime env, you can use a new API named ray.get_current_runtime_env() to get the parent runtime env and modify this dict by yourself. Like: Actor.options(runtime_env=ray.get_current_runtime_env().update({"X": "Y"}))

Refer to some discuss in issue https://github.com/ray-project/ray/issues/21494.

[edoakes]

I think this behavior is reasonable and simplifies the mental model, while still allowing libraries and advanced users to achieve today’s functionality.

My only concern is that this is going to be a breaking change for some users. If we do this, it needs to be well documented and ideally we would raise a useful warning message in the cases where behaviors have changed (I’m not sure if that’s possible to detect though).

[rkooo567]

Since it is already GA, I added core-api-change-approval. cc @ericl @edoakes please review it!