Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

CeleryKubernetesExecutor is broken in 2.1.0

See original GitHub issue

Tested with both chart 1.1.0rc1 (i.e. main branch r.n.) and 1.0.0

in Airflow 2.1.0, scheduler does not exit immediately (this was an issue < 2.1.0), but all tasks fail like this:

2021-06-08 15:30:17,167] {scheduler_job.py:1241} ERROR - Executor reports task instance <TaskInstance: sqoop_acquisition.terminate_job_flow 2021-06-08 13:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-06-08 15:30:17,170] {scheduler_job.py:1241} ERROR - Executor reports task instance <TaskInstance: gsheets.state_mapping.to_s3 2021-06-08 14:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-06-08 15:30:17,171] {scheduler_job.py:1241} ERROR - Executor reports task instance <TaskInstance: gsheets.app_event_taxonomy.to_s3 2021-06-08 14:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-06-08 15:30:17,172] {scheduler_job.py:1241} ERROR - Executor reports task instance <TaskInstance: gsheets.strain_flavors.to_s3 2021-06-08 14:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-06-08 15:30:19,053] {scheduler_job.py:1205} INFO - Executor reports execution of reporting_8hr.dev.cannalytics.feature_duration.sql execution_date=2021-06-08 07:00:00+00:00 exited with status failed for try_number 1
[2021-06-08 15:30:19,125] {scheduler_job.py:1241} ERROR - Executor reports task instance <TaskInstance: reporting_8hr.dev.cannalytics.feature_duration.sql 2021-06-08 07:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
[2021-06-08 15:30:23,842] {dagrun.py:429} ERROR - Marking run <DagRun gsheets @ 2021-06-08 14:00:00+00:00: scheduled__2021-06-08T14:00:00+00:00, externally triggered: False> failed

@kaxil @jedcunningham do you see this when you run CKE? Any suggestions?

Issue Analytics

State:
Created 2 years ago
Comments:7 (7 by maintainers)

Top GitHub Comments

2reactions

kaxilcommented, Jun 29, 2021

I was able to reproduce it and https://github.com/apache/airflow/pull/16700 should fix it.

Workaround until the PR is merged and release is to set PYTHONOPTIMIZE=1 as Environment Variable for the worker.

0reactions

damon09273commented, Jun 29, 2021

I was able to reproduce it and #16700 should fix it.

Workaround until the PR is merged and release is to set PYTHONOPTIMIZE=1 as Environment Variable for the worker.

Got it, thank you for providing information.

Top Results From Across the Web

Configuration Reference — Airflow Documentation

New in version 2.1.0. Specifies the method or methods allowed when accessing the resource. Type. string. Default. '' Environment Variable.

[airflow] branch v2-1-test updated: Add changelog.

This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a commit to branch v2-1-test in repository ...

apache-airflow Changelog - pyup.io

Implement ``send_callback`` method for ``CeleryKubernetesExecutor`` and ... Document fix for broken elasticsearch logs with 2.3.0+ upgrade (23821)

Configuring Apache Airflow on Kubernetes for use with Elyra

Pipelines in Elyra can be run locally in JupyterLab, or remotely on Kubeflow Pipelines or Apache Airflow to take advantage of shared resources...

apache-airflow-providers-airbyte: Versions - Openbase

... Fixed broken json_client (#17529); Handle and log exceptions raised during task ... must be a dict" bug (#16648); Fix CeleryKubernetesExecutor (#16700) ...