[runtime env] pip-based runtime env refactor

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Work Items

1. Introduce Virtualenv, detail:
   • Create vituralenv for jobs or workers
   • check ray version
   • use pip check to find python dependencies conflict
   • download dependencies
   • PR:
     • https://github.com/ray-project/ray/pull/21801/
     • https://github.com/ray-project/ray/pull/22309
2. Use coroutine in runtime env agent, detail: #21950
3. Introduce DownloadManager, detail：
   • limit download peak.
   • singletons, manager all download tasks
4. Use coroutine to setup py_modules and working_dir, so that they can be executed in parallel with virtualenv creation.
5. Support change pip version
6. Install conda and pip in container
7. Optimize pip install $HTTP_URL, detail:
   • we download package from $HTTP_URL
   • and pip install /path/to/this/package
8. Support cached Virtualenv

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[edoakes]

@SongGuyang @Catch-Bull actually, to make this even less intrusive, what if we put these options nested inside of the runtime_env field? That would prevent us from needing to update schemas all over the place to support it. For example:

@ray.remote(runtime_env=RuntimeEnv(..., config=RuntimeEnvConfig(setup_timeout_s=10))
def f(): pass

@ray.remote(runtime_env={..., "config": {"setup_timeout_s": 10}})
def f(): pass

I would be happy to go forward with this proposal, it is the least intrusive I can think of (I don’t think most users will ever think about these advanced options).

[architkulkarni]

I agree that we should have pip_check, pip_version and setup_runtime_env_timeout_seconds per runtime-env rather than cluster env, for the reason you mentioned. The timeout_seconds especially makes sense as a runtime_env option because certain environments are much slower to install than others.

Since the timeout shouldn’t be part of the hash like you said, I think the second interface you proposed makes more sense, but I don’t have a strong opinion about it.