use PythonVirtualenvOperator with a prebuilt env

Description Instead of passing in the requirements and relying Airflow to build the env, in some cases it would be more straightforward and desirable to just make Airflow use a prebuilt env.

This could be done with PythonVirtualenvOperator with a param like env_path.

Use case / motivation

virtualenv_task = PythonVirtualenvOperator(
    task_id="virtualenv_python",
    python_callable=callable_virtualenv,
    env_path='./color-env', # the path to prebuilt env
    # requirements=["colorama==0.4.0"], # replaces this
    system_site_packages=False,
    dag=dag,
)

Are you willing to submit a PR?

Perhaps

Related Issues

[gaoyibin0001]

@ManikandanUV We are doing it the following way for now:

env = vars_dict.get("conda_env", None)
path_to_python = f"/home/username/.conda{'/envs/'+env if env is not None else ''}/bin/python"

parse_files = BashOperator(
            task_id='parse-files',
            bash_command=f"{path_to_python} {abs_path_code}/my_repo/parse.py {files_to_parse}",
            env={"PATH": os.environ["PATH"],
                 "DB_CONN": db_conn}
        )

We have an environment variable containing the conda-env name which is used to get the full path to the Python executable. Then, using a BashOperator, we can use the same environment again for different Tasks.

Additionally, we run an update to the environment if requirements changed (note that we are using poetry as package manager):

update_repo = BashOperator(
    task_id=f"update-repo-{folder}",
    bash_command=f"cd {abs_path_code}/{folder}; "
           "git checkout master; git stash; git stash drop; git pull"
    )
install_dependencies = BashOperator(
    task_id=f"install-dependencies-{folder}",
          bash_command=f"cd {abs_path_code}/{folder}; conda activate {env_name}; poetry install "
    )
update_repo >> install_dependencies
`
``

may use "conda run -n env_name python xxx.py

[potiuk]

Just one comment - this fine, if you can make sure all your - distributed - venvs are present on all the workers (which might be tricky if you want to update those) - and you have to somehow link the “task” definition (expecting certain venv with certain requirement versions) with the “deployment” (i.e. worker definition). Any kind of “upgrade” to such an env might be tricky. The “local” installation pattern had the advantage, that you always got the requirements in the version you described in the task definition (via requirements specification).

I think a better solution would be to use caching mechanism to the task and modify the PythonVirtualenv to use it. However this might be tricky to get right when you have multiple tasks of the same type running in the same runner in Celery deployment.