Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add virtual env creation in Celery workers

See original GitHub issue

Description

Add the option for the celery worker to create a new virtual env, install some packages, and run airflow run command inside it (based on executor_config params). Really nice to have - have reusable virtual env that can be shared between tasks with the same param (based on user configuration).

Use case / motivation

Once getting to a point when you want to create cluster for different types of python tasks and you’ve multiple teams working on the same cluster, you need to start splitting into different python packages the business login code to allow better versioning control and avoid the need of restarting the workers when deploying new util code. I think it would be amazing if we can allow creating new virtual envs as part of Airflow and control the package versions.

I know that PythonVirtualenvOperator exists, but:

Creating env related thing feels like an executor job to me, the coder should not use specific operators for it.
The big downside to it is that if I want to use ShortCircuitOperator or BranchPythonOperator or any kind of new python based operator, I’ve to create a new operator that will inherit from PythonVirtualenvOperator and duplicate the desired functionality.

Are you willing to submit a PR?

Yes, would love to.

Related Issues

Not that I can find.

Issue Analytics

State:
Created 3 years ago
Comments:20 (19 by maintainers)

Top GitHub Comments

1reaction

zacharya19commented, Jan 8, 2021

@potiuk Didn’t get the response email, weird, but anyway sent my Wiki ID now.

1reaction

zacharya19commented, Jan 2, 2021

This seems to be more reliable approach (managing deps on deployment level) than managing dependencies via application (Airflow) which main task is scheduling and executing tasks.

As I said, the issue here is that the user have to create some kind of complex deployment scripts, because running airflow inside venv requires restart when updating the packages. Anyway, I had a few extra ideas to make this feature more reliable, I sent a request to get edit access to the wiki and will write a more detailed proposal when I do.

Top Results From Across the Web

First steps with Celery using a virtualenv - Stack Overflow

Tried to cleanly install celery in virtualenv (without installing it in ... .python/bin/activate pip install celery celery worker --help.

Running celeryd as a daemon — Celery 2.5.5 documentation

Path to the celeryd program. Default is celeryd . You can point this to an virtualenv, or even use manage.py for django. CELERYBEAT_USER....

Celery With Supervisor Virtualenv - Micropyramid

First, you need to install supervisor in your virtualenv and generate a configuration file. It's a simplified version of the Celery supervisor ...

Celery with Django in production - DEV Community ‍ ‍

1. Create the dedicated user and group · 2. Create the celery configuration file · 3. Create the systemd file · 4. Restart...

Celery Tutorial Using Python - Javatpoint

First, we will create a Django project named learn_celery in the celery_django folder. But first we will create the virtual environment. All dependencies...