Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to use `configured` with the celery k8s executor

See original GitHub issue

When I try to use configured on the celery k8s executor definition and launch a run, I get the following error:

This is because we are trying to grab the config straight from run config before processing the entire run config:

execution_run_config = run_config["execution"][CELERY_K8S_CONFIG_KEY].get("config", {})
res = process_config(execution_config_schema, execution_run_config)

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:5 (4 by maintainers)

Top GitHub Comments

1reaction

alangenfeldcommented, Mar 4, 2021

is this a good first task to do?

No, this one is pretty difficult

1reaction

schrockncommented, Feb 17, 2021

The refactor I mentioned is complete. Feel free to work on or assign to someone!

Top Results From Across the Web

dagster - AWS credentials not found for celery-k8s deployment

The deployment configuration applies to the user code servers. Meanwhile the celery executor runs your pipeline code in separate kubernetes ...

Kubernetes Executor — Airflow Documentation - Apache Airflow

The Kubernetes executor runs each task instance in its own pod on a Kubernetes cluster. KubernetesExecutor runs as a process in the Airflow...

The Kubernetes executor for GitLab Runner

Notice: If a non-existing key will be used then job will fail on Pod creation stage. Secret volumes. Secret volume configuration instructs Kubernetes...

Use the KubernetesPodOperator | Cloud Composer

Secret variables configuration: Passes a Kubernetes Secret object to the pod. Pod affinity configuration: Limits the available nodes to schedule pods on. Full ......

Source code for dagster_celery_k8s.launcher - Dagster Docs

A run worker Kubernetes Job, which traverses the dagster run execution plan ... "Using configured with the celery-k8s executor is not supported at...